2024-03-16T19:18:27.249 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-16T19:18:27.249 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --name leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi --verbose --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} --archive /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602108 -- /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602108/orig.config.yaml 2024-03-16T19:18:27.301 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T19:18:27.384 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602108 branch: wip-lusov-qdb-squid description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} email: null first_in_suite: false job_id: '7602108' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb-squid ceph: cephfs: session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - but it is still running - objects unfound and apparently lost - MDS_SLOW_METADATA_IO - MDS_TRIM sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b kclient: syntax: v2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb-squid sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b owner: scheduled_leonidus@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 6918 sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b sleep_before_teardown: 0 subset: 111/10000 suite: fs suite_branch: wip-lusov-qdb-squid suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRsXkXxfZSNA6a7epG+n39p3eYTeaH0kgIkfHYLXj4ijGrI5/lmYi+jnEcuid+o40QXqd1woHEZLdI87ygM77Y= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBADm3bpJk5YJJHDFYSEn665kH7RZaDY4QpUnVqq1EXzhLtmsb4pxAn/p5mB2nA/tywJfgLoA78z8nMrDdcGtL8w= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - thrashosds: null - workunit: clients: all: - fs/misc/trivial_sync.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-15_11:44:52 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-16T19:18:27.384 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa; will attempt to use it 2024-03-16T19:18:27.385 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks 2024-03-16T19:18:27.385 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-16T19:18:27.386 INFO:teuthology.task.internal:Checking packages... 2024-03-16T19:18:27.409 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b' 2024-03-16T19:18:27.409 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-16T19:18:27.409 INFO:teuthology.packaging:ref: None 2024-03-16T19:18:27.409 INFO:teuthology.packaging:tag: None 2024-03-16T19:18:27.409 INFO:teuthology.packaging:branch: wip-lusov-qdb-squid 2024-03-16T19:18:27.410 INFO:teuthology.packaging:sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:18:27.410 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb-squid 2024-03-16T19:18:27.734 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1681.gaba3c1d2 2024-03-16T19:18:27.735 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-16T19:18:27.741 INFO:teuthology.task.internal:no buildpackages task found 2024-03-16T19:18:27.742 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-16T19:18:27.749 INFO:teuthology.task.internal:Saving configuration 2024-03-16T19:18:27.768 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-16T19:18:27.774 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T19:18:27.801 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602108', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 19:13:21.311989', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRsXkXxfZSNA6a7epG+n39p3eYTeaH0kgIkfHYLXj4ijGrI5/lmYi+jnEcuid+o40QXqd1woHEZLdI87ygM77Y='} 2024-03-16T19:18:27.819 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602108', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 19:13:21.310249', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBADm3bpJk5YJJHDFYSEn665kH7RZaDY4QpUnVqq1EXzhLtmsb4pxAn/p5mB2nA/tywJfgLoA78z8nMrDdcGtL8w='} 2024-03-16T19:18:27.820 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-16T19:18:27.825 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-16T19:18:27.826 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-16T19:18:27.826 INFO:teuthology.run_tasks:Running task console_log... 2024-03-16T19:18:27.898 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6ff14ea0d0>, signals=[15]) 2024-03-16T19:18:27.898 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-16T19:18:27.905 INFO:teuthology.task.internal:Opening connections... 2024-03-16T19:18:27.905 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-03-16T19:18:27.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:18:27.998 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-03-16T19:18:27.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:18:28.086 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-16T19:18:28.092 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-03-16T19:18:28.110 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-03-16T19:18:28.110 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-03-16T19:18:28.167 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-03-16T19:18:28.167 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-03-16T19:18:28.167 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-03-16T19:18:28.167 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-03-16T19:18:28.167 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-03-16T19:18:28.167 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-03-16T19:18:28.167 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T19:18:28.167 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-03-16T19:18:28.168 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-03-16T19:18:28.168 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T19:18:28.168 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-03-16T19:18:28.168 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T19:18:28.168 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T19:18:28.168 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T19:18:28.169 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-03-16T19:18:28.195 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-03-16T19:18:28.212 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-03-16T19:18:28.213 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-03-16T19:18:28.269 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-03-16T19:18:28.269 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T19:18:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T19:18:28.271 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-03-16T19:18:28.295 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-16T19:18:28.304 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-16T19:18:28.310 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-16T19:18:28.310 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-03-16T19:18:28.312 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-03-16T19:18:28.328 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-16T19:18:28.334 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-16T19:18:28.334 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-03-16T19:18:28.370 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-03-16T19:18:28.455 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-16T19:18:28.505 INFO:teuthology.run_tasks:Running task kernel... 2024-03-16T19:18:28.520 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-16T19:18:28.520 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-16T19:18:28.520 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-16T19:18:28.520 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T19:18:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:18:28.537 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-16T19:18:28.595 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-361.el9.x86_64 2024-03-16T19:18:28.595 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.14.0-361.el9.x86_64 2024-03-16T19:18:28.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2024-03-16T19:18:31.688 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 4.9 MB/s | 6.7 MB 00:01 2024-03-16T19:18:33.070 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-16T19:18:37.103 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-16T19:18:39.220 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-16T19:18:43.396 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-16T19:18:45.638 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 15 kB 00:01 2024-03-16T19:18:46.902 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-16T19:18:51.864 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-16T19:18:54.079 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T19:18:54.079 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T19:18:54.171 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-16T19:18:54.180 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-16T19:18:54.180 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-03-16T19:18:54.180 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-16T19:18:54.180 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-16T19:18:54.180 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T19:18:54.180 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-16T19:18:54.180 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T19:18:54.180 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T19:18:54.180 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T19:18:54.181 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:18:54.181 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-16T19:18:54.181 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-16T19:18:54.181 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2024-03-16T19:18:54.181 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:18:54.181 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 100 M 2024-03-16T19:18:54.182 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 123 M 2024-03-16T19:18:54.182 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-16T19:18:55.590 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 5.2 MB/s | 6.0 MB 00:01 2024-03-16T19:18:55.840 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 15 MB/s | 21 MB 00:01 2024-03-16T19:18:56.549 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 19 MB/s | 39 MB 00:02 2024-03-16T19:18:57.532 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 17 MB/s | 33 MB 00:01 2024-03-16T19:18:57.535 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-16T19:18:57.535 INFO:teuthology.orchestra.run.smithi172.stdout:Total 30 MB/s | 100 MB 00:03 2024-03-16T19:18:58.421 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-16T19:18:58.773 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-16T19:18:58.774 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-16T19:19:00.128 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-16T19:19:00.129 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-16T19:19:02.835 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-16T19:19:03.478 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T19:19:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T19:19:06.444 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T19:19:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T19:19:11.360 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T19:19:11.480 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T19:19:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T19:19:45.542 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T19:19:45.543 INFO:teuthology.orchestra.run.smithi172.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T19:19:45.543 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:19:45.628 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T19:19:46.165 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T19:19:46.165 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T19:19:46.165 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T19:19:46.165 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T19:19:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T19:19:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:19:46.605 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-16T19:19:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T19:19:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T19:19:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T19:19:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T19:19:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:19:46.606 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-16T19:19:47.113 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T19:19:47.296 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T19:19:47.297 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T19:19:47.297 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T19:19:47.297 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T19:19:47.297 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T19:19:47.297 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-16T19:19:47.297 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T19:19:47.297 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T19:19:47.421 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T19:19:47.421 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T19:19:47.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2024-03-16T19:19:47.455 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2024-03-16T19:19:47.455 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2024-03-16T19:19:47.455 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.06 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 68.el9 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Group : Unspecified 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 8264254 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : builder@centos.org 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : CentOS 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2024-03-16T19:19:47.456 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2024-03-16T19:19:47.457 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:19:47.457 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T19:19:47.457 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T19:19:47.457 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T19:19:47.457 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2024-03-16T19:19:47.457 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:19:47.457 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T19:19:47.458 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T19:19:47.458 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 5.14.0-427.el9.x86_64 2024-03-16T19:19:47.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T19:19:48.057 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-03-16T19:19:49.428 INFO:teuthology.orchestra.run.smithi172.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T19:19:49.466 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-03-16T19:19:49.468 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-16T19:19:49.483 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.RT3Tg5syk2 2024-03-16T19:19:49.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RT3Tg5syk2 2024-03-16T19:19:49.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.RT3Tg5syk2 2024-03-16T19:19:49.685 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.RT3Tg5syk2 is 6KB 2024-03-16T19:19:49.733 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.RT3Tg5syk2 2024-03-16T19:19:49.747 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T19:19:49.812 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T19:19:49.812 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T19:19:49.812 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T19:19:49.813 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T19:19:49.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T19:19:49.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-03-16T19:20:19.971 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-16T19:20:19.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-16T19:20:19.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:20:54.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-16T19:21:03.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-16T19:21:03.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:21:21.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-16T19:21:33.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-16T19:21:33.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:21:37.350 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-16T19:21:37.856 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-16T19:21:37.856 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-16T19:21:37.857 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-16T19:21:37.871 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-427.el9.x86_64 2024-03-16T19:21:37.871 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T19:21:37.871 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T19:21:37.871 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T19:21:38.873 DEBUG:teuthology.parallel:result is None 2024-03-16T19:21:38.873 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-16T19:21:38.880 INFO:teuthology.task.internal:Creating test directory... 2024-03-16T19:21:38.881 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T19:21:38.883 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T19:21:38.902 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-16T19:21:38.908 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-16T19:21:38.965 INFO:teuthology.task.internal:Creating archive directory... 2024-03-16T19:21:38.965 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T19:21:38.968 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T19:21:39.055 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-16T19:21:39.062 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-16T19:21:39.062 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T19:21:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:21:39.079 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T19:21:39.081 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T19:21:39.166 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T19:21:39.189 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T19:21:39.207 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T19:21:39.227 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T19:21:39.229 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-16T19:21:39.236 INFO:teuthology.task.internal:Configuring sudo... 2024-03-16T19:21:39.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T19:21:39.239 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T19:21:39.268 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-16T19:21:39.277 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-16T19:21:39.277 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T19:21:39.306 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T19:21:39.324 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T19:21:39.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T19:21:39.451 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T19:21:39.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T19:21:39.576 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:21:39.576 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T19:21:39.649 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T19:21:39.692 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T19:21:39.750 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T19:21:39.834 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T19:21:39.879 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:21:39.879 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T19:21:39.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-03-16T19:21:39.950 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-03-16T19:21:39.994 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T19:21:40.030 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T19:21:40.509 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-16T19:21:40.516 INFO:teuthology.task.internal:Starting timer... 2024-03-16T19:21:40.516 INFO:teuthology.run_tasks:Running task pcp... 2024-03-16T19:21:40.526 INFO:teuthology.run_tasks:Running task selinux... 2024-03-16T19:21:40.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-03-16T19:21:40.585 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-03-16T19:21:40.586 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-03-16T19:21:40.665 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-03-16T19:21:40.667 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-16T19:21:40.667 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-03-16T19:21:40.682 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-03-16T19:21:40.682 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-03-16T19:21:40.717 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-03-16T19:21:40.718 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi172.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2024-03-16T19:21:40.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-16T19:21:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:21:40.751 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-16T19:21:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:21:40.786 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-16T19:21:40.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-03-16T19:21:40.827 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-03-16T19:21:40.861 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-16T19:21:40.871 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-16T19:21:40.888 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-16T19:21:40.889 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi172.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-16T19:26:32.259 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi172.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-03-16T19:26:32.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-16T19:26:32.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:26:32.352 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-16T19:26:32.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-16T19:26:32.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-16T19:26:32.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T19:26:32.516 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-03-16T19:26:32.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-16T19:26:32.541 INFO:teuthology.run_tasks:Running task clock... 2024-03-16T19:26:32.552 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-16T19:26:32.552 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T19:26:32.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T19:26:32.555 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T19:26:32.555 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T19:26:32.584 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T19:26:32.601 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T19:26:32.611 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T19:26:32.627 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T19:26:32.630 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-03-16T19:26:32.644 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-03-16T19:26:32.657 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-03-16T19:26:32.661 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T19:26:32.670 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-03-16T19:26:32.677 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T19:26:32.687 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T19:26:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T19:26:32.730 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-16T19:26:32.733 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T19:26:32.733 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-16T19:26:32.733 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T19:26:32.733 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T19:26:32.733 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T19:26:32.733 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T19:26:32.747 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-16T19:26:32.750 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T19:26:32.751 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-16T19:26:32.751 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T19:26:32.751 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T19:26:32.751 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T19:26:32.751 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T19:26:32.751 INFO:teuthology.run_tasks:Running task install... 2024-03-16T19:26:32.759 DEBUG:teuthology.task.install:project ceph 2024-03-16T19:26:32.759 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}} 2024-03-16T19:26:32.759 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'} 2024-03-16T19:26:32.759 INFO:teuthology.task.install:Using flavor: default 2024-03-16T19:26:32.765 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-16T19:26:32.765 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-16T19:26:32.766 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'tag': None, 'wait_for_package': False} 2024-03-16T19:26:32.766 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:26:32.768 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'tag': None, 'wait_for_package': False} 2024-03-16T19:26:32.768 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:26:32.917 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/ 2024-03-16T19:26:32.918 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1681.gaba3c1d2 2024-03-16T19:26:33.026 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T19:26:33.026 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:26:33.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T19:26:33.055 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T19:26:33.055 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T19:26:33.087 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/ 2024-03-16T19:26:33.088 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1681.gaba3c1d2 2024-03-16T19:26:33.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T19:26:33.196 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T19:26:33.197 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:26:33.197 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T19:26:33.201 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T19:26:33.224 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T19:26:33.224 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T19:26:33.268 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2024-03-16T19:26:33.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2024-03-16T19:26:33.291 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T19:26:33.371 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T19:26:33.437 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-03-16T19:26:33.438 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-03-16T19:26:33.518 INFO:teuthology.orchestra.run.smithi172.stdout:74 files removed 2024-03-16T19:26:33.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T19:26:33.672 INFO:teuthology.orchestra.run.smithi174.stdout:74 files removed 2024-03-16T19:26:33.699 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T19:26:34.497 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 163 kB/s | 83 kB 00:00 2024-03-16T19:26:34.526 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 126 kB/s | 83 kB 00:00 2024-03-16T19:26:34.951 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-16T19:26:35.047 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-03-16T19:26:35.510 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-03-16T19:26:35.606 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-03-16T19:26:36.221 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-16T19:26:36.350 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-03-16T19:26:37.671 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-16T19:26:37.742 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-16T19:26:38.537 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-16T19:26:40.779 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-16T19:26:41.433 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-03-16T19:26:43.483 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-16T19:26:45.047 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-16T19:26:46.864 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-16T19:26:47.583 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 34 MB/s | 21 MB 00:00 2024-03-16T19:26:49.750 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.1 MB 00:02 2024-03-16T19:26:51.259 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2024-03-16T19:26:52.582 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-16T19:26:53.055 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-16T19:26:55.080 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T19:26:55.080 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T19:26:55.095 INFO:teuthology.orchestra.run.smithi172.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T19:26:55.096 INFO:teuthology.orchestra.run.smithi172.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T19:26:55.098 INFO:teuthology.orchestra.run.smithi172.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T19:26:55.099 INFO:teuthology.orchestra.run.smithi172.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T19:26:55.101 INFO:teuthology.orchestra.run.smithi172.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T19:26:55.104 INFO:teuthology.orchestra.run.smithi172.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T19:26:55.105 INFO:teuthology.orchestra.run.smithi172.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T19:26:55.186 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-16T19:26:55.215 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-16T19:26:55.215 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-16T19:26:55.215 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-16T19:26:55.215 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-16T19:26:55.215 INFO:teuthology.orchestra.run.smithi172.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T19:26:55.215 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 6.6 k 2024-03-16T19:26:55.216 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.3 M 2024-03-16T19:26:55.216 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 887 k 2024-03-16T19:26:55.216 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 145 k 2024-03-16T19:26:55.216 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 1.5 M 2024-03-16T19:26:55.216 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 138 k 2024-03-16T19:26:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 1.8 M 2024-03-16T19:26:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 7.4 M 2024-03-16T19:26:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 49 k 2024-03-16T19:26:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 9.8 M 2024-03-16T19:26:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 47 M 2024-03-16T19:26:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 264 k 2024-03-16T19:26:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 748 k 2024-03-16T19:26:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 218 k 2024-03-16T19:26:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-shell x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 27 k 2024-03-16T19:26:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-top noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 18 k 2024-03-16T19:26:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T19:26:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T19:26:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T19:26:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 31 k 2024-03-16T19:26:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 744 k 2024-03-16T19:26:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 127 k 2024-03-16T19:26:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T19:26:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T19:26:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T19:26:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T19:26:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T19:26:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T19:26:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 161 k 2024-03-16T19:26:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 321 k 2024-03-16T19:26:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 298 k 2024-03-16T19:26:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 99 k 2024-03-16T19:26:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 85 k 2024-03-16T19:26:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T19:26:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 171 k 2024-03-16T19:26:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T19:26:55.222 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2024-03-16T19:26:55.222 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T19:26:55.222 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T19:26:55.222 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T19:26:55.222 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T19:26:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T19:26:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T19:26:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T19:26:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T19:26:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T19:26:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T19:26:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T19:26:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T19:26:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T19:26:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.4 M 2024-03-16T19:26:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T19:26:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T19:26:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T19:26:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T19:26:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T19:26:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T19:26:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T19:26:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T19:26:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T19:26:55.226 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T19:26:55.226 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-16T19:26:55.226 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T19:26:55.226 INFO:teuthology.orchestra.run.smithi172.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T19:26:55.226 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T19:26:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 20 M 2024-03-16T19:26:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 31 k 2024-03-16T19:26:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 2.3 M 2024-03-16T19:26:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 248 k 2024-03-16T19:26:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 4.7 M 2024-03-16T19:26:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 17 M 2024-03-16T19:26:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 15 k 2024-03-16T19:26:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 25 k 2024-03-16T19:26:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T19:26:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T19:26:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T19:26:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T19:26:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T19:26:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T19:26:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T19:26:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T19:26:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T19:26:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 163 k 2024-03-16T19:26:55.230 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T19:26:55.230 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T19:26:55.230 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T19:26:55.230 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T19:26:55.230 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T19:26:55.231 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 498 k 2024-03-16T19:26:55.231 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T19:26:55.231 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.1 M 2024-03-16T19:26:55.231 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T19:26:55.232 INFO:teuthology.orchestra.run.smithi172.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T19:26:55.232 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T19:26:55.232 INFO:teuthology.orchestra.run.smithi172.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T19:26:55.232 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T19:26:55.232 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T19:26:55.233 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T19:26:55.233 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T19:26:55.233 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T19:26:55.233 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T19:26:55.233 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T19:26:55.233 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T19:26:55.234 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T19:26:55.234 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T19:26:55.234 INFO:teuthology.orchestra.run.smithi172.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T19:26:55.234 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T19:26:55.234 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T19:26:55.235 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T19:26:55.235 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 45 k 2024-03-16T19:26:55.235 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 131 k 2024-03-16T19:26:55.235 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T19:26:55.235 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T19:26:55.236 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T19:26:55.236 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T19:26:55.236 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T19:26:55.236 INFO:teuthology.orchestra.run.smithi172.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T19:26:55.236 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T19:26:55.236 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T19:26:55.237 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T19:26:55.237 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T19:26:55.237 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T19:26:55.237 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T19:26:55.237 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T19:26:55.237 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T19:26:55.237 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T19:26:55.237 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T19:26:55.238 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T19:26:55.239 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T19:26:55.240 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout:Installing weak dependencies: 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout:Install 150 Packages 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 24 Packages 2024-03-16T19:26:55.241 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:26:55.242 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 224 M 2024-03-16T19:26:55.242 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-16T19:26:56.050 INFO:teuthology.orchestra.run.smithi172.stdout:(1/174): ceph-19.0.0-1681.gaba3c1d2.el9.x86_64. 41 kB/s | 6.6 kB 00:00 2024-03-16T19:26:56.367 INFO:teuthology.orchestra.run.smithi172.stdout:(2/174): ceph-fuse-19.0.0-1681.gaba3c1d2.el9.x8 2.7 MB/s | 887 kB 00:00 2024-03-16T19:26:56.433 INFO:teuthology.orchestra.run.smithi172.stdout:(3/174): ceph-immutable-object-cache-19.0.0-168 2.1 MB/s | 145 kB 00:00 2024-03-16T19:26:56.726 INFO:teuthology.orchestra.run.smithi172.stdout:(4/174): ceph-base-19.0.0-1681.gaba3c1d2.el9.x8 6.4 MB/s | 5.3 MB 00:00 2024-03-16T19:26:56.776 INFO:teuthology.orchestra.run.smithi172.stdout:(5/174): ceph-mds-19.0.0-1681.gaba3c1d2.el9.x86 6.7 MB/s | 2.3 MB 00:00 2024-03-16T19:26:56.926 INFO:teuthology.orchestra.run.smithi172.stdout:(6/174): ceph-mgr-19.0.0-1681.gaba3c1d2.el9.x86 7.4 MB/s | 1.5 MB 00:00 2024-03-16T19:26:57.560 INFO:teuthology.orchestra.run.smithi172.stdout:(7/174): ceph-mon-19.0.0-1681.gaba3c1d2.el9.x86 6.0 MB/s | 4.7 MB 00:00 2024-03-16T19:26:58.118 INFO:teuthology.orchestra.run.smithi172.stdout:(8/174): ceph-common-19.0.0-1681.gaba3c1d2.el9. 9.2 MB/s | 20 MB 00:02 2024-03-16T19:26:58.143 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 6.1 kB/s | 1.7 kB 00:00 2024-03-16T19:26:58.168 INFO:teuthology.orchestra.run.smithi172.stdout:(9/174): ceph-selinux-19.0.0-1681.gaba3c1d2.el9 506 kB/s | 25 kB 00:00 2024-03-16T19:26:59.568 INFO:teuthology.orchestra.run.smithi172.stdout:(10/174): ceph-radosgw-19.0.0-1681.gaba3c1d2.el 4.9 MB/s | 9.8 MB 00:02 2024-03-16T19:26:59.727 INFO:teuthology.orchestra.run.smithi172.stdout:(11/174): ceph-osd-19.0.0-1681.gaba3c1d2.el9.x8 6.0 MB/s | 17 MB 00:02 2024-03-16T19:26:59.752 INFO:teuthology.orchestra.run.smithi172.stdout:(12/174): cephfs-mirror-19.0.0-1681.gaba3c1d2.e 1.2 MB/s | 218 kB 00:00 2024-03-16T19:26:59.777 INFO:teuthology.orchestra.run.smithi172.stdout:(13/174): cephfs-shell-19.0.0-1681.gaba3c1d2.el 540 kB/s | 27 kB 00:00 2024-03-16T19:26:59.803 INFO:teuthology.orchestra.run.smithi172.stdout:(14/174): libcephfs-devel-19.0.0-1681.gaba3c1d2 624 kB/s | 31 kB 00:00 2024-03-16T19:26:59.861 INFO:teuthology.orchestra.run.smithi172.stdout:(15/174): libcephsqlite-19.0.0-1681.gaba3c1d2.e 2.7 MB/s | 163 kB 00:00 2024-03-16T19:26:59.903 INFO:teuthology.orchestra.run.smithi172.stdout:(16/174): libcephfs2-19.0.0-1681.gaba3c1d2.el9. 5.8 MB/s | 744 kB 00:00 2024-03-16T19:26:59.937 INFO:teuthology.orchestra.run.smithi172.stdout:(17/174): librados-devel-19.0.0-1681.gaba3c1d2. 1.6 MB/s | 127 kB 00:00 2024-03-16T19:27:00.046 INFO:teuthology.orchestra.run.smithi172.stdout:(18/174): libradosstriper1-19.0.0-1681.gaba3c1d 3.4 MB/s | 498 kB 00:00 2024-03-16T19:27:00.129 INFO:teuthology.orchestra.run.smithi172.stdout:(19/174): python3-ceph-argparse-19.0.0-1681.gab 541 kB/s | 45 kB 00:00 2024-03-16T19:27:00.213 INFO:teuthology.orchestra.run.smithi172.stdout:(20/174): python3-ceph-common-19.0.0-1681.gaba3 1.5 MB/s | 131 kB 00:00 2024-03-16T19:27:00.297 INFO:teuthology.orchestra.run.smithi172.stdout:(21/174): python3-cephfs-19.0.0-1681.gaba3c1d2. 1.9 MB/s | 161 kB 00:00 2024-03-16T19:27:00.388 INFO:teuthology.orchestra.run.smithi172.stdout:(22/174): python3-rados-19.0.0-1681.gaba3c1d2.e 3.4 MB/s | 321 kB 00:00 2024-03-16T19:27:00.480 INFO:teuthology.orchestra.run.smithi172.stdout:(23/174): python3-rbd-19.0.0-1681.gaba3c1d2.el9 3.2 MB/s | 298 kB 00:00 2024-03-16T19:27:00.564 INFO:teuthology.orchestra.run.smithi172.stdout:(24/174): python3-rgw-19.0.0-1681.gaba3c1d2.el9 1.2 MB/s | 99 kB 00:00 2024-03-16T19:27:00.574 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T19:27:00.575 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T19:27:00.590 INFO:teuthology.orchestra.run.smithi174.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T19:27:00.591 INFO:teuthology.orchestra.run.smithi174.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T19:27:00.592 INFO:teuthology.orchestra.run.smithi174.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T19:27:00.593 INFO:teuthology.orchestra.run.smithi174.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T19:27:00.595 INFO:teuthology.orchestra.run.smithi174.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T19:27:00.598 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T19:27:00.599 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T19:27:00.647 INFO:teuthology.orchestra.run.smithi172.stdout:(25/174): rbd-fuse-19.0.0-1681.gaba3c1d2.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-16T19:27:00.677 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-16T19:27:00.694 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-16T19:27:00.694 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-16T19:27:00.694 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-16T19:27:00.694 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 6.6 k 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.3 M 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 887 k 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 145 k 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 1.5 M 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 138 k 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 1.8 M 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 7.4 M 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 49 k 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 9.8 M 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 47 M 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 264 k 2024-03-16T19:27:00.695 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 748 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 218 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 27 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 18 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 31 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 744 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 127 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T19:27:00.696 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 161 k 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 321 k 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 298 k 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 99 k 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 85 k 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 171 k 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T19:27:00.697 INFO:teuthology.orchestra.run.smithi174.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.4 M 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T19:27:00.698 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 20 M 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 31 k 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 2.3 M 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 248 k 2024-03-16T19:27:00.699 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 4.7 M 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 17 M 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 15 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 25 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 163 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T19:27:00.700 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 498 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.1 M 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T19:27:00.701 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 45 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 131 k 2024-03-16T19:27:00.702 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T19:27:00.703 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T19:27:00.704 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T19:27:00.705 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T19:27:00.706 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-16T19:27:00.707 INFO:teuthology.orchestra.run.smithi174.stdout:Install 150 Packages 2024-03-16T19:27:00.708 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 24 Packages 2024-03-16T19:27:00.708 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:27:00.708 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 224 M 2024-03-16T19:27:00.708 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-16T19:27:00.848 INFO:teuthology.orchestra.run.smithi172.stdout:(26/174): librgw2-19.0.0-1681.gaba3c1d2.el9.x86 5.6 MB/s | 5.1 MB 00:00 2024-03-16T19:27:00.940 INFO:teuthology.orchestra.run.smithi172.stdout:(27/174): rbd-nbd-19.0.0-1681.gaba3c1d2.el9.x86 1.8 MB/s | 171 kB 00:00 2024-03-16T19:27:01.057 INFO:teuthology.orchestra.run.smithi172.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1681.g 269 kB/s | 31 kB 00:00 2024-03-16T19:27:01.174 INFO:teuthology.orchestra.run.smithi172.stdout:(29/174): rbd-mirror-19.0.0-1681.gaba3c1d2.el9. 5.9 MB/s | 3.1 MB 00:00 2024-03-16T19:27:01.199 INFO:teuthology.orchestra.run.smithi172.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1681.gaba3c1d 974 kB/s | 138 kB 00:00 2024-03-16T19:27:01.503 INFO:teuthology.orchestra.run.smithi174.stdout:(1/174): ceph-19.0.0-1681.gaba3c1d2.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-16T19:27:01.708 INFO:teuthology.orchestra.run.smithi172.stdout:(31/174): ceph-test-19.0.0-1681.gaba3c1d2.el9.x 13 MB/s | 47 MB 00:03 2024-03-16T19:27:01.758 INFO:teuthology.orchestra.run.smithi172.stdout:(32/174): ceph-mgr-dashboard-19.0.0-1681.gaba3c 3.1 MB/s | 1.8 MB 00:00 2024-03-16T19:27:01.783 INFO:teuthology.orchestra.run.smithi172.stdout:(33/174): ceph-mgr-modules-core-19.0.0-1681.gab 3.2 MB/s | 248 kB 00:00 2024-03-16T19:27:01.816 INFO:teuthology.orchestra.run.smithi172.stdout:(34/174): ceph-mgr-rook-19.0.0-1681.gaba3c1d2.e 844 kB/s | 49 kB 00:00 2024-03-16T19:27:01.842 INFO:teuthology.orchestra.run.smithi172.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1681.ga 261 kB/s | 15 kB 00:00 2024-03-16T19:27:01.925 INFO:teuthology.orchestra.run.smithi172.stdout:(36/174): ceph-volume-19.0.0-1681.gaba3c1d2.el9 2.4 MB/s | 264 kB 00:00 2024-03-16T19:27:01.976 INFO:teuthology.orchestra.run.smithi172.stdout:(37/174): cephadm-19.0.0-1681.gaba3c1d2.el9.noa 5.5 MB/s | 748 kB 00:00 2024-03-16T19:27:01.993 INFO:teuthology.orchestra.run.smithi172.stdout:(38/174): cephfs-top-19.0.0-1681.gaba3c1d2.el9. 272 kB/s | 18 kB 00:00 2024-03-16T19:27:02.170 INFO:teuthology.orchestra.run.smithi174.stdout:(2/174): ceph-fuse-19.0.0-1681.gaba3c1d2.el9.x8 1.3 MB/s | 887 kB 00:00 2024-03-16T19:27:02.218 INFO:teuthology.orchestra.run.smithi172.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 7.3 MB/s | 7.4 MB 00:01 2024-03-16T19:27:03.120 INFO:teuthology.orchestra.run.smithi174.stdout:(3/174): ceph-immutable-object-cache-19.0.0-168 153 kB/s | 145 kB 00:00 2024-03-16T19:27:03.336 INFO:teuthology.orchestra.run.smithi172.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 487 kB/s | 662 kB 00:01 2024-03-16T19:27:03.385 INFO:teuthology.orchestra.run.smithi172.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 91 kB/s | 106 kB 00:01 2024-03-16T19:27:03.511 INFO:teuthology.orchestra.run.smithi172.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 1.7 MB/s | 307 kB 00:00 2024-03-16T19:27:03.536 INFO:teuthology.orchestra.run.smithi172.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 210 kB/s | 32 kB 00:00 2024-03-16T19:27:04.020 INFO:teuthology.orchestra.run.smithi172.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 17 kB 00:00 2024-03-16T19:27:04.129 INFO:teuthology.orchestra.run.smithi174.stdout:(4/174): ceph-mds-19.0.0-1681.gaba3c1d2.el9.x86 2.3 MB/s | 2.3 MB 00:01 2024-03-16T19:27:04.362 INFO:teuthology.orchestra.run.smithi172.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 29 kB/s | 10 kB 00:00 2024-03-16T19:27:04.529 INFO:teuthology.orchestra.run.smithi174.stdout:(5/174): ceph-mgr-19.0.0-1681.gaba3c1d2.el9.x86 3.7 MB/s | 1.5 MB 00:00 2024-03-16T19:27:04.578 INFO:teuthology.orchestra.run.smithi172.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.8 MB/s | 3.0 MB 00:01 2024-03-16T19:27:04.604 INFO:teuthology.orchestra.run.smithi172.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 659 kB/s | 159 kB 00:00 2024-03-16T19:27:04.637 INFO:teuthology.orchestra.run.smithi172.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 759 kB/s | 44 kB 00:00 2024-03-16T19:27:04.688 INFO:teuthology.orchestra.run.smithi172.stdout:(49/174): bison-3.7.4-5.el9.x86_64.rpm 342 kB/s | 920 kB 00:02 2024-03-16T19:27:04.838 INFO:teuthology.orchestra.run.smithi172.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 661 kB 00:00 2024-03-16T19:27:04.980 INFO:teuthology.orchestra.run.smithi172.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 1.7 MB/s | 578 kB 00:00 2024-03-16T19:27:05.005 INFO:teuthology.orchestra.run.smithi172.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 146 kB/s | 46 kB 00:00 2024-03-16T19:27:05.055 INFO:teuthology.orchestra.run.smithi172.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 282 kB 00:00 2024-03-16T19:27:05.089 INFO:teuthology.orchestra.run.smithi172.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-16T19:27:05.147 INFO:teuthology.orchestra.run.smithi172.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-16T19:27:05.164 INFO:teuthology.orchestra.run.smithi172.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 314 kB/s | 34 kB 00:00 2024-03-16T19:27:05.354 INFO:teuthology.orchestra.run.smithi174.stdout:(6/174): ceph-mon-19.0.0-1681.gaba3c1d2.el9.x86 5.7 MB/s | 4.7 MB 00:00 2024-03-16T19:27:05.923 INFO:teuthology.orchestra.run.smithi172.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 1.0 MB/s | 811 kB 00:00 2024-03-16T19:27:06.132 INFO:teuthology.orchestra.run.smithi172.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 1.5 MB/s | 1.5 MB 00:00 2024-03-16T19:27:06.307 INFO:teuthology.orchestra.run.smithi172.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 349 kB/s | 134 kB 00:00 2024-03-16T19:27:06.599 INFO:teuthology.orchestra.run.smithi172.stdout:(60/174): openblas-openmp-0.3.21-2.el9.x86_64.r 3.1 MB/s | 4.7 MB 00:01 2024-03-16T19:27:06.754 INFO:teuthology.orchestra.run.smithi174.stdout:(7/174): ceph-osd-19.0.0-1681.gaba3c1d2.el9.x86 12 MB/s | 17 MB 00:01 2024-03-16T19:27:06.841 INFO:teuthology.orchestra.run.smithi172.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 1.9 MB/s | 1.0 MB 00:00 2024-03-16T19:27:06.866 INFO:teuthology.orchestra.run.smithi172.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 292 kB/s | 78 kB 00:00 2024-03-16T19:27:07.179 INFO:teuthology.orchestra.run.smithi174.stdout:(8/174): ceph-base-19.0.0-1681.gaba3c1d2.el9.x8 915 kB/s | 5.3 MB 00:05 2024-03-16T19:27:07.191 INFO:teuthology.orchestra.run.smithi172.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 75 kB/s | 24 kB 00:00 2024-03-16T19:27:07.313 INFO:teuthology.orchestra.run.smithi174.stdout:(9/174): ceph-radosgw-19.0.0-1681.gaba3c1d2.el9 17 MB/s | 9.8 MB 00:00 2024-03-16T19:27:07.329 INFO:teuthology.orchestra.run.smithi174.stdout:(10/174): ceph-selinux-19.0.0-1681.gaba3c1d2.el 168 kB/s | 25 kB 00:00 2024-03-16T19:27:07.408 INFO:teuthology.orchestra.run.smithi172.stdout:(64/174): postgresql-server-13.11-1.el9.x86_64. 4.5 MB/s | 5.7 MB 00:01 2024-03-16T19:27:07.425 INFO:teuthology.orchestra.run.smithi172.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 77 kB/s | 18 kB 00:00 2024-03-16T19:27:07.630 INFO:teuthology.orchestra.run.smithi174.stdout:(11/174): cephfs-mirror-19.0.0-1681.gaba3c1d2.e 727 kB/s | 218 kB 00:00 2024-03-16T19:27:07.692 INFO:teuthology.orchestra.run.smithi172.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 510 kB/s | 144 kB 00:00 2024-03-16T19:27:07.709 INFO:teuthology.orchestra.run.smithi172.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 112 kB/s | 32 kB 00:00 2024-03-16T19:27:07.738 INFO:teuthology.orchestra.run.smithi174.stdout:(12/174): cephfs-shell-19.0.0-1681.gaba3c1d2.el 250 kB/s | 27 kB 00:00 2024-03-16T19:27:07.784 INFO:teuthology.orchestra.run.smithi172.stdout:(68/174): python3-babel-2.9.1-2.el9.noarch.rpm 6.1 MB/s | 5.8 MB 00:00 2024-03-16T19:27:07.847 INFO:teuthology.orchestra.run.smithi174.stdout:(13/174): libcephfs-devel-19.0.0-1681.gaba3c1d2 289 kB/s | 31 kB 00:00 2024-03-16T19:27:08.001 INFO:teuthology.orchestra.run.smithi172.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 747 kB/s | 217 kB 00:00 2024-03-16T19:27:08.111 INFO:teuthology.orchestra.run.smithi172.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 536 kB/s | 174 kB 00:00 2024-03-16T19:27:08.176 INFO:teuthology.orchestra.run.smithi172.stdout:(71/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.9 MB/s | 4.8 MB 00:00 2024-03-16T19:27:08.202 INFO:teuthology.orchestra.run.smithi172.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 344 kB/s | 69 kB 00:00 2024-03-16T19:27:08.227 INFO:teuthology.orchestra.run.smithi172.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 360 kB/s | 41 kB 00:00 2024-03-16T19:27:08.277 INFO:teuthology.orchestra.run.smithi172.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-16T19:27:08.302 INFO:teuthology.orchestra.run.smithi172.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-16T19:27:08.328 INFO:teuthology.orchestra.run.smithi172.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.1 MB/s | 211 kB 00:00 2024-03-16T19:27:08.353 INFO:teuthology.orchestra.run.smithi172.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 633 kB/s | 47 kB 00:00 2024-03-16T19:27:08.378 INFO:teuthology.orchestra.run.smithi172.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 857 kB/s | 43 kB 00:00 2024-03-16T19:27:08.645 INFO:teuthology.orchestra.run.smithi172.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 132 kB/s | 39 kB 00:00 2024-03-16T19:27:08.670 INFO:teuthology.orchestra.run.smithi172.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 141 kB/s | 41 kB 00:00 2024-03-16T19:27:08.879 INFO:teuthology.orchestra.run.smithi172.stdout:(81/174): uuid-1.6.2-55.el9.x86_64.rpm 272 kB/s | 56 kB 00:00 2024-03-16T19:27:08.904 INFO:teuthology.orchestra.run.smithi172.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-03-16T19:27:08.930 INFO:teuthology.orchestra.run.smithi174.stdout:(14/174): libcephfs2-19.0.0-1681.gaba3c1d2.el9. 687 kB/s | 744 kB 00:01 2024-03-16T19:27:09.113 INFO:teuthology.orchestra.run.smithi172.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 269 kB/s | 63 kB 00:00 2024-03-16T19:27:09.138 INFO:teuthology.orchestra.run.smithi172.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 189 kB/s | 44 kB 00:00 2024-03-16T19:27:09.239 INFO:teuthology.orchestra.run.smithi174.stdout:(15/174): libcephsqlite-19.0.0-1681.gaba3c1d2.e 529 kB/s | 163 kB 00:00 2024-03-16T19:27:09.397 INFO:teuthology.orchestra.run.smithi172.stdout:(85/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-03-16T19:27:09.422 INFO:teuthology.orchestra.run.smithi172.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 255 kB/s | 79 kB 00:00 2024-03-16T19:27:09.447 INFO:teuthology.orchestra.run.smithi172.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 785 kB/s | 241 kB 00:00 2024-03-16T19:27:09.464 INFO:teuthology.orchestra.run.smithi174.stdout:(16/174): librados-devel-19.0.0-1681.gaba3c1d2. 563 kB/s | 127 kB 00:00 2024-03-16T19:27:09.514 INFO:teuthology.orchestra.run.smithi172.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 9.9 MB/s | 1.1 MB 00:00 2024-03-16T19:27:09.540 INFO:teuthology.orchestra.run.smithi172.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-16T19:27:09.573 INFO:teuthology.orchestra.run.smithi172.stdout:(90/174): python3-ply-3.11-14.el9.noarch.rpm 827 kB/s | 103 kB 00:00 2024-03-16T19:27:09.598 INFO:teuthology.orchestra.run.smithi172.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-16T19:27:09.907 INFO:teuthology.orchestra.run.smithi172.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 511 kB/s | 187 kB 00:00 2024-03-16T19:27:10.082 INFO:teuthology.orchestra.run.smithi172.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-16T19:27:10.273 INFO:teuthology.orchestra.run.smithi174.stdout:(17/174): libradosstriper1-19.0.0-1681.gaba3c1d 616 kB/s | 498 kB 00:00 2024-03-16T19:27:10.524 INFO:teuthology.orchestra.run.smithi172.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 49 kB/s | 21 kB 00:00 2024-03-16T19:27:10.658 INFO:teuthology.orchestra.run.smithi172.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 166 kB/s | 125 kB 00:00 2024-03-16T19:27:10.783 INFO:teuthology.orchestra.run.smithi172.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 98 kB/s | 25 kB 00:00 2024-03-16T19:27:10.808 INFO:teuthology.orchestra.run.smithi172.stdout:(97/174): libquadmath-11.4.1-3.el9.x86_64.rpm 161 kB/s | 195 kB 00:01 2024-03-16T19:27:11.617 INFO:teuthology.orchestra.run.smithi172.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 307 kB/s | 248 kB 00:00 2024-03-16T19:27:11.667 INFO:teuthology.orchestra.run.smithi172.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 278 kB/s | 245 kB 00:00 2024-03-16T19:27:11.926 INFO:teuthology.orchestra.run.smithi172.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 135 kB/s | 35 kB 00:00 2024-03-16T19:27:12.231 INFO:teuthology.orchestra.run.smithi174.stdout:(18/174): ceph-test-19.0.0-1681.gaba3c1d2.el9.x 9.6 MB/s | 47 MB 00:04 2024-03-16T19:27:12.267 INFO:teuthology.orchestra.run.smithi172.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 130 kB/s | 44 kB 00:00 2024-03-16T19:27:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:(102/174): qatlib-23.11.0-1.el9.x86_64.rpm 320 kB/s | 216 kB 00:00 2024-03-16T19:27:12.340 INFO:teuthology.orchestra.run.smithi174.stdout:(19/174): python3-ceph-argparse-19.0.0-1681.gab 418 kB/s | 45 kB 00:00 2024-03-16T19:27:12.368 INFO:teuthology.orchestra.run.smithi172.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-03-16T19:27:12.501 INFO:teuthology.orchestra.run.smithi172.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-03-16T19:27:12.540 INFO:teuthology.orchestra.run.smithi174.stdout:(20/174): python3-ceph-common-19.0.0-1681.gaba3 657 kB/s | 131 kB 00:00 2024-03-16T19:27:12.632 INFO:teuthology.orchestra.run.smithi174.stdout:(21/174): python3-cephfs-19.0.0-1681.gaba3c1d2. 1.7 MB/s | 161 kB 00:00 2024-03-16T19:27:12.882 INFO:teuthology.orchestra.run.smithi174.stdout:(22/174): ceph-common-19.0.0-1681.gaba3c1d2.el9 1.7 MB/s | 20 MB 00:11 2024-03-16T19:27:12.902 INFO:teuthology.orchestra.run.smithi172.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-03-16T19:27:12.907 INFO:teuthology.orchestra.run.smithi174.stdout:(23/174): python3-rados-19.0.0-1681.gaba3c1d2.e 1.1 MB/s | 321 kB 00:00 2024-03-16T19:27:12.936 INFO:teuthology.orchestra.run.smithi172.stdout:(106/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 140 kB/s | 94 kB 00:00 2024-03-16T19:27:13.003 INFO:teuthology.orchestra.run.smithi172.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-03-16T19:27:13.007 INFO:teuthology.orchestra.run.smithi174.stdout:(24/174): python3-rgw-19.0.0-1681.gaba3c1d2.el9 994 kB/s | 99 kB 00:00 2024-03-16T19:27:13.128 INFO:teuthology.orchestra.run.smithi172.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 101 kB/s | 19 kB 00:00 2024-03-16T19:27:13.132 INFO:teuthology.orchestra.run.smithi174.stdout:(25/174): rbd-fuse-19.0.0-1681.gaba3c1d2.el9.x8 683 kB/s | 85 kB 00:00 2024-03-16T19:27:13.158 INFO:teuthology.orchestra.run.smithi174.stdout:(26/174): python3-rbd-19.0.0-1681.gaba3c1d2.el9 1.1 MB/s | 298 kB 00:00 2024-03-16T19:27:13.228 INFO:teuthology.orchestra.run.smithi172.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-16T19:27:13.378 INFO:teuthology.orchestra.run.smithi172.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-16T19:27:13.383 INFO:teuthology.orchestra.run.smithi174.stdout:(27/174): rbd-nbd-19.0.0-1681.gaba3c1d2.el9.x86 759 kB/s | 171 kB 00:00 2024-03-16T19:27:13.403 INFO:teuthology.orchestra.run.smithi172.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 277 kB/s | 49 kB 00:00 2024-03-16T19:27:13.483 INFO:teuthology.orchestra.run.smithi174.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1681.g 315 kB/s | 31 kB 00:00 2024-03-16T19:27:13.512 INFO:teuthology.orchestra.run.smithi172.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-03-16T19:27:13.692 INFO:teuthology.orchestra.run.smithi174.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1681.gaba3c1d 664 kB/s | 138 kB 00:00 2024-03-16T19:27:13.746 INFO:teuthology.orchestra.run.smithi172.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-03-16T19:27:13.771 INFO:teuthology.orchestra.run.smithi172.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 411 kB/s | 151 kB 00:00 2024-03-16T19:27:13.846 INFO:teuthology.orchestra.run.smithi172.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-03-16T19:27:13.863 INFO:teuthology.orchestra.run.smithi172.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-03-16T19:27:13.913 INFO:teuthology.orchestra.run.smithi172.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 482 kB/s | 32 kB 00:00 2024-03-16T19:27:13.930 INFO:teuthology.orchestra.run.smithi172.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-16T19:27:13.997 INFO:teuthology.orchestra.run.smithi172.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-16T19:27:14.089 INFO:teuthology.orchestra.run.smithi172.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-16T19:27:14.115 INFO:teuthology.orchestra.run.smithi172.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.9 MB/s | 358 kB 00:00 2024-03-16T19:27:14.309 INFO:teuthology.orchestra.run.smithi174.stdout:(30/174): rbd-mirror-19.0.0-1681.gaba3c1d2.el9. 2.6 MB/s | 3.1 MB 00:01 2024-03-16T19:27:14.315 INFO:teuthology.orchestra.run.smithi172.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 157 kB/s | 35 kB 00:00 2024-03-16T19:27:14.340 INFO:teuthology.orchestra.run.smithi172.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 899 kB/s | 203 kB 00:00 2024-03-16T19:27:14.441 INFO:teuthology.orchestra.run.smithi172.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-03-16T19:27:14.466 INFO:teuthology.orchestra.run.smithi172.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-16T19:27:14.516 INFO:teuthology.orchestra.run.smithi172.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-16T19:27:14.541 INFO:teuthology.orchestra.run.smithi172.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-03-16T19:27:14.592 INFO:teuthology.orchestra.run.smithi172.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 308 kB/s | 23 kB 00:00 2024-03-16T19:27:14.617 INFO:teuthology.orchestra.run.smithi172.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-03-16T19:27:14.667 INFO:teuthology.orchestra.run.smithi172.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 262 kB/s | 20 kB 00:00 2024-03-16T19:27:14.851 INFO:teuthology.orchestra.run.smithi172.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 252 kB/s | 46 kB 00:00 2024-03-16T19:27:14.884 INFO:teuthology.orchestra.run.smithi172.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 3.8 MB/s | 1.0 MB 00:00 2024-03-16T19:27:14.927 INFO:teuthology.orchestra.run.smithi172.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-16T19:27:14.952 INFO:teuthology.orchestra.run.smithi172.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 862 kB/s | 58 kB 00:00 2024-03-16T19:27:15.011 INFO:teuthology.orchestra.run.smithi172.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-16T19:27:15.036 INFO:teuthology.orchestra.run.smithi172.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-16T19:27:15.078 INFO:teuthology.orchestra.run.smithi172.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-16T19:27:15.111 INFO:teuthology.orchestra.run.smithi172.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-03-16T19:27:15.145 INFO:teuthology.orchestra.run.smithi172.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 459 kB/s | 31 kB 00:00 2024-03-16T19:27:15.195 INFO:teuthology.orchestra.run.smithi172.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-16T19:27:15.220 INFO:teuthology.orchestra.run.smithi172.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-03-16T19:27:15.271 INFO:teuthology.orchestra.run.smithi172.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-03-16T19:27:15.296 INFO:teuthology.orchestra.run.smithi172.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-03-16T19:27:15.301 INFO:teuthology.orchestra.run.smithi174.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1681.gaba3c 1.1 MB/s | 1.8 MB 00:01 2024-03-16T19:27:15.363 INFO:teuthology.orchestra.run.smithi172.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-16T19:27:15.388 INFO:teuthology.orchestra.run.smithi172.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 975 kB/s | 90 kB 00:00 2024-03-16T19:27:15.417 INFO:teuthology.orchestra.run.smithi174.stdout:(32/174): librgw2-19.0.0-1681.gaba3c1d2.el9.x86 1.0 MB/s | 5.1 MB 00:05 2024-03-16T19:27:15.513 INFO:teuthology.orchestra.run.smithi172.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2024-03-16T19:27:15.526 INFO:teuthology.orchestra.run.smithi174.stdout:(33/174): ceph-mgr-rook-19.0.0-1681.gaba3c1d2.e 455 kB/s | 49 kB 00:00 2024-03-16T19:27:15.539 INFO:teuthology.orchestra.run.smithi172.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-03-16T19:27:15.598 INFO:teuthology.orchestra.run.smithi172.stdout:(148/174): openssl-devel-3.0.7-27.el9.x86_64.rp 857 kB/s | 4.1 MB 00:04 2024-03-16T19:27:15.618 INFO:teuthology.orchestra.run.smithi174.stdout:(34/174): ceph-mgr-modules-core-19.0.0-1681.gab 782 kB/s | 248 kB 00:00 2024-03-16T19:27:15.635 INFO:teuthology.orchestra.run.smithi174.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1681.ga 141 kB/s | 15 kB 00:00 2024-03-16T19:27:15.648 INFO:teuthology.orchestra.run.smithi172.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-16T19:27:15.831 INFO:teuthology.orchestra.run.smithi172.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-03-16T19:27:15.898 INFO:teuthology.orchestra.run.smithi172.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-03-16T19:27:15.952 INFO:teuthology.orchestra.run.smithi174.stdout:(36/174): ceph-volume-19.0.0-1681.gaba3c1d2.el9 791 kB/s | 264 kB 00:00 2024-03-16T19:27:16.061 INFO:teuthology.orchestra.run.smithi174.stdout:(37/174): cephfs-top-19.0.0-1681.gaba3c1d2.el9. 168 kB/s | 18 kB 00:00 2024-03-16T19:27:16.182 INFO:teuthology.orchestra.run.smithi172.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 8.1 MB/s | 2.3 MB 00:00 2024-03-16T19:27:16.265 INFO:teuthology.orchestra.run.smithi172.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-16T19:27:16.315 INFO:teuthology.orchestra.run.smithi172.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 852 kB/s | 42 kB 00:00 2024-03-16T19:27:16.352 INFO:teuthology.orchestra.run.smithi174.stdout:(38/174): cephadm-19.0.0-1681.gaba3c1d2.el9.noa 1.0 MB/s | 748 kB 00:00 2024-03-16T19:27:16.374 INFO:teuthology.orchestra.run.smithi172.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.2 MB/s | 248 kB 00:00 2024-03-16T19:27:16.378 INFO:teuthology.orchestra.run.smithi174.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 662 kB 00:00 2024-03-16T19:27:16.424 INFO:teuthology.orchestra.run.smithi172.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-16T19:27:16.475 INFO:teuthology.orchestra.run.smithi172.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 732 kB/s | 36 kB 00:00 2024-03-16T19:27:16.503 INFO:teuthology.orchestra.run.smithi174.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 6.0 MB/s | 920 kB 00:00 2024-03-16T19:27:16.528 INFO:teuthology.orchestra.run.smithi174.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 704 kB/s | 106 kB 00:00 2024-03-16T19:27:16.542 INFO:teuthology.orchestra.run.smithi172.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-16T19:27:16.570 INFO:teuthology.orchestra.run.smithi174.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-16T19:27:16.587 INFO:teuthology.orchestra.run.smithi174.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 541 kB/s | 32 kB 00:00 2024-03-16T19:27:16.609 INFO:teuthology.orchestra.run.smithi172.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.8 MB/s | 256 kB 00:00 2024-03-16T19:27:16.667 INFO:teuthology.orchestra.run.smithi172.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 395 kB/s | 23 kB 00:00 2024-03-16T19:27:16.712 INFO:teuthology.orchestra.run.smithi174.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 137 kB/s | 17 kB 00:00 2024-03-16T19:27:16.734 INFO:teuthology.orchestra.run.smithi172.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-16T19:27:16.784 INFO:teuthology.orchestra.run.smithi172.stdout:(162/174): librbd1-19.0.0-1681.gaba3c1d2.el9.x8 2.8 MB/s | 3.1 MB 00:01 2024-03-16T19:27:16.804 INFO:teuthology.orchestra.run.smithi174.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-03-16T19:27:16.809 INFO:teuthology.orchestra.run.smithi172.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-16T19:27:16.829 INFO:teuthology.orchestra.run.smithi174.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 87 kB/s | 10 kB 00:00 2024-03-16T19:27:16.846 INFO:teuthology.orchestra.run.smithi174.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-16T19:27:16.877 INFO:teuthology.orchestra.run.smithi172.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 953 kB/s | 64 kB 00:00 2024-03-16T19:27:16.880 INFO:teuthology.orchestra.run.smithi174.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-16T19:27:16.913 INFO:teuthology.orchestra.run.smithi174.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-16T19:27:16.935 INFO:teuthology.orchestra.run.smithi172.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 494 kB/s | 29 kB 00:00 2024-03-16T19:27:16.947 INFO:teuthology.orchestra.run.smithi174.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-16T19:27:16.969 INFO:teuthology.orchestra.run.smithi172.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 736 kB/s | 136 kB 00:00 2024-03-16T19:27:16.972 INFO:teuthology.orchestra.run.smithi174.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 801 kB/s | 46 kB 00:00 2024-03-16T19:27:16.998 INFO:teuthology.orchestra.run.smithi174.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-16T19:27:17.002 INFO:teuthology.orchestra.run.smithi172.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 18 MB/s | 1.2 MB 00:00 2024-03-16T19:27:17.023 INFO:teuthology.orchestra.run.smithi174.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-16T19:27:17.048 INFO:teuthology.orchestra.run.smithi174.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-16T19:27:17.073 INFO:teuthology.orchestra.run.smithi174.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-16T19:27:17.111 INFO:teuthology.orchestra.run.smithi172.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 21 MB/s | 2.3 MB 00:00 2024-03-16T19:27:17.161 INFO:teuthology.orchestra.run.smithi172.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-03-16T19:27:17.212 INFO:teuthology.orchestra.run.smithi172.stdout:(170/174): librados2-19.0.0-1681.gaba3c1d2.el9. 2.1 MB/s | 3.4 MB 00:01 2024-03-16T19:27:17.237 INFO:teuthology.orchestra.run.smithi172.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 413 kB/s | 31 kB 00:00 2024-03-16T19:27:17.262 INFO:teuthology.orchestra.run.smithi172.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 3.0 MB/s | 465 kB 00:00 2024-03-16T19:27:17.290 INFO:teuthology.orchestra.run.smithi174.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 7.0 MB/s | 1.5 MB 00:00 2024-03-16T19:27:17.357 INFO:teuthology.orchestra.run.smithi174.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-03-16T19:27:17.399 INFO:teuthology.orchestra.run.smithi174.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 7.4 MB/s | 811 kB 00:00 2024-03-16T19:27:17.421 INFO:teuthology.orchestra.run.smithi172.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.1 MB/s | 1.1 MB 00:00 2024-03-16T19:27:17.424 INFO:teuthology.orchestra.run.smithi174.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-16T19:27:17.515 INFO:teuthology.orchestra.run.smithi174.stdout:(60/174): ceph-mgr-diskprediction-local-19.0.0- 2.3 MB/s | 7.4 MB 00:03 2024-03-16T19:27:17.665 INFO:teuthology.orchestra.run.smithi174.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 21 MB/s | 5.7 MB 00:00 2024-03-16T19:27:17.699 INFO:teuthology.orchestra.run.smithi174.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-03-16T19:27:17.724 INFO:teuthology.orchestra.run.smithi174.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 378 kB/s | 78 kB 00:00 2024-03-16T19:27:17.821 INFO:teuthology.orchestra.run.smithi172.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 127 kB/s | 74 kB 00:00 2024-03-16T19:27:17.826 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-16T19:27:17.826 INFO:teuthology.orchestra.run.smithi172.stdout:Total 9.9 MB/s | 224 MB 00:22 2024-03-16T19:27:17.933 INFO:teuthology.orchestra.run.smithi174.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-16T19:27:17.958 INFO:teuthology.orchestra.run.smithi174.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 106 kB/s | 24 kB 00:00 2024-03-16T19:27:17.983 INFO:teuthology.orchestra.run.smithi174.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 71 kB/s | 18 kB 00:00 2024-03-16T19:27:18.008 INFO:teuthology.orchestra.run.smithi174.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-16T19:27:18.034 INFO:teuthology.orchestra.run.smithi174.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-16T19:27:18.226 INFO:teuthology.orchestra.run.smithi174.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-16T19:27:18.251 INFO:teuthology.orchestra.run.smithi174.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 901 kB/s | 217 kB 00:00 2024-03-16T19:27:18.276 INFO:teuthology.orchestra.run.smithi174.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 725 kB/s | 174 kB 00:00 2024-03-16T19:27:18.301 INFO:teuthology.orchestra.run.smithi174.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 923 kB/s | 69 kB 00:00 2024-03-16T19:27:18.315 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-16T19:27:18.326 INFO:teuthology.orchestra.run.smithi174.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-16T19:27:18.351 INFO:teuthology.orchestra.run.smithi174.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-16T19:27:18.377 INFO:teuthology.orchestra.run.smithi174.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-16T19:27:18.402 INFO:teuthology.orchestra.run.smithi174.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-16T19:27:18.427 INFO:teuthology.orchestra.run.smithi174.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-03-16T19:27:18.452 INFO:teuthology.orchestra.run.smithi174.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 572 kB/s | 43 kB 00:00 2024-03-16T19:27:18.626 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-16T19:27:18.626 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-16T19:27:18.744 INFO:teuthology.orchestra.run.smithi174.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 133 kB/s | 39 kB 00:00 2024-03-16T19:27:18.770 INFO:teuthology.orchestra.run.smithi174.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 130 kB/s | 41 kB 00:00 2024-03-16T19:27:18.979 INFO:teuthology.orchestra.run.smithi174.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-16T19:27:19.129 INFO:teuthology.orchestra.run.smithi174.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-03-16T19:27:19.153 INFO:teuthology.orchestra.run.smithi174.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 148 kB/s | 56 kB 00:00 2024-03-16T19:27:19.179 INFO:teuthology.orchestra.run.smithi174.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 314 kB/s | 63 kB 00:00 2024-03-16T19:27:19.204 INFO:teuthology.orchestra.run.smithi174.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-16T19:27:19.313 INFO:teuthology.orchestra.run.smithi174.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 496 kB/s | 79 kB 00:00 2024-03-16T19:27:19.371 INFO:teuthology.orchestra.run.smithi174.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 241 kB 00:00 2024-03-16T19:27:19.397 INFO:teuthology.orchestra.run.smithi174.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-16T19:27:19.438 INFO:teuthology.orchestra.run.smithi174.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.1 MB 00:00 2024-03-16T19:27:19.464 INFO:teuthology.orchestra.run.smithi174.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-16T19:27:19.489 INFO:teuthology.orchestra.run.smithi174.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-16T19:27:19.856 INFO:teuthology.orchestra.run.smithi174.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 478 kB/s | 187 kB 00:00 2024-03-16T19:27:19.931 INFO:teuthology.orchestra.run.smithi174.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 442 kB/s | 195 kB 00:00 2024-03-16T19:27:19.956 INFO:teuthology.orchestra.run.smithi174.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-03-16T19:27:19.990 INFO:teuthology.orchestra.run.smithi174.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-16T19:27:20.073 INFO:teuthology.orchestra.run.smithi174.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 151 kB/s | 21 kB 00:00 2024-03-16T19:27:20.099 INFO:teuthology.orchestra.run.smithi174.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 179 kB/s | 25 kB 00:00 2024-03-16T19:27:20.249 INFO:teuthology.orchestra.run.smithi174.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-16T19:27:20.274 INFO:teuthology.orchestra.run.smithi174.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 248 kB 00:00 2024-03-16T19:27:20.341 INFO:teuthology.orchestra.run.smithi174.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.3 MB/s | 216 kB 00:00 2024-03-16T19:27:20.400 INFO:teuthology.orchestra.run.smithi174.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-16T19:27:20.458 INFO:teuthology.orchestra.run.smithi174.stdout:(102/174): qatlib-service-23.11.0-1.el9.x86_64. 190 kB/s | 35 kB 00:00 2024-03-16T19:27:20.483 INFO:teuthology.orchestra.run.smithi174.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-03-16T19:27:20.584 INFO:teuthology.orchestra.run.smithi174.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-03-16T19:27:20.664 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-16T19:27:20.664 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-16T19:27:21.326 INFO:teuthology.orchestra.run.smithi174.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-03-16T19:27:21.384 INFO:teuthology.orchestra.run.smithi174.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.0 MB/s | 4.1 MB 00:01 2024-03-16T19:27:21.418 INFO:teuthology.orchestra.run.smithi174.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 658 kB/s | 548 kB 00:00 2024-03-16T19:27:21.768 INFO:teuthology.orchestra.run.smithi174.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 51 kB/s | 19 kB 00:00 2024-03-16T19:27:21.927 INFO:teuthology.orchestra.run.smithi174.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.2 MB/s | 4.4 MB 00:00 2024-03-16T19:27:21.952 INFO:teuthology.orchestra.run.smithi174.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 141 kB/s | 26 kB 00:00 2024-03-16T19:27:22.019 INFO:teuthology.orchestra.run.smithi174.stdout:(111/174): gperftools-libs-2.9.1-2.el9.x86_64.r 446 kB/s | 309 kB 00:00 2024-03-16T19:27:22.036 INFO:teuthology.orchestra.run.smithi174.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-03-16T19:27:22.161 INFO:teuthology.orchestra.run.smithi174.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2024-03-16T19:27:22.194 INFO:teuthology.orchestra.run.smithi174.stdout:(114/174): libunwind-1.6.2-1.el9.x86_64.rpm 278 kB/s | 67 kB 00:00 2024-03-16T19:27:22.220 INFO:teuthology.orchestra.run.smithi174.stdout:(115/174): luarocks-3.9.2-1.el9.noarch.rpm 752 kB/s | 151 kB 00:00 2024-03-16T19:27:22.286 INFO:teuthology.orchestra.run.smithi174.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-03-16T19:27:22.313 INFO:teuthology.orchestra.run.smithi174.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 368 kB/s | 43 kB 00:00 2024-03-16T19:27:22.330 INFO:teuthology.orchestra.run.smithi174.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 294 kB/s | 32 kB 00:00 2024-03-16T19:27:22.438 INFO:teuthology.orchestra.run.smithi174.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 99 kB/s | 15 kB 00:00 2024-03-16T19:27:22.480 INFO:teuthology.orchestra.run.smithi174.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.0 MB/s | 172 kB 00:00 2024-03-16T19:27:22.589 INFO:teuthology.orchestra.run.smithi174.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-03-16T19:27:22.614 INFO:teuthology.orchestra.run.smithi174.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 1.2 MB/s | 358 kB 00:00 2024-03-16T19:27:22.647 INFO:teuthology.orchestra.run.smithi174.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 211 kB/s | 35 kB 00:00 2024-03-16T19:27:22.698 INFO:teuthology.orchestra.run.smithi174.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-16T19:27:22.807 INFO:teuthology.orchestra.run.smithi174.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-03-16T19:27:22.831 INFO:teuthology.orchestra.run.smithi174.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 787 kB/s | 144 kB 00:00 2024-03-16T19:27:22.915 INFO:teuthology.orchestra.run.smithi174.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-03-16T19:27:22.941 INFO:teuthology.orchestra.run.smithi174.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-16T19:27:22.982 INFO:teuthology.orchestra.run.smithi174.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 5.5 MB/s | 2.0 MB 00:00 2024-03-16T19:27:23.024 INFO:teuthology.orchestra.run.smithi174.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-16T19:27:23.050 INFO:teuthology.orchestra.run.smithi174.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-16T19:27:23.125 INFO:teuthology.orchestra.run.smithi174.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-03-16T19:27:23.150 INFO:teuthology.orchestra.run.smithi174.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 369 kB/s | 46 kB 00:00 2024-03-16T19:27:23.175 INFO:teuthology.orchestra.run.smithi174.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 628 kB/s | 79 kB 00:00 2024-03-16T19:27:23.226 INFO:teuthology.orchestra.run.smithi174.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-16T19:27:23.276 INFO:teuthology.orchestra.run.smithi174.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-16T19:27:23.310 INFO:teuthology.orchestra.run.smithi174.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 123 kB/s | 16 kB 00:00 2024-03-16T19:27:23.335 INFO:teuthology.orchestra.run.smithi174.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 826 kB/s | 90 kB 00:00 2024-03-16T19:27:23.402 INFO:teuthology.orchestra.run.smithi174.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 225 kB/s | 28 kB 00:00 2024-03-16T19:27:23.418 INFO:teuthology.orchestra.run.smithi174.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-03-16T19:27:23.444 INFO:teuthology.orchestra.run.smithi174.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-16T19:27:23.536 INFO:teuthology.orchestra.run.smithi174.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 442 kB/s | 59 kB 00:00 2024-03-16T19:27:23.553 INFO:teuthology.orchestra.run.smithi174.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 267 kB/s | 36 kB 00:00 2024-03-16T19:27:23.570 INFO:teuthology.orchestra.run.smithi174.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 407 kB/s | 51 kB 00:00 2024-03-16T19:27:23.645 INFO:teuthology.orchestra.run.smithi174.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-16T19:27:23.678 INFO:teuthology.orchestra.run.smithi174.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 715 kB/s | 90 kB 00:00 2024-03-16T19:27:23.712 INFO:teuthology.orchestra.run.smithi174.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-03-16T19:27:23.746 INFO:teuthology.orchestra.run.smithi174.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-03-16T19:27:23.837 INFO:teuthology.orchestra.run.smithi174.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-16T19:27:23.888 INFO:teuthology.orchestra.run.smithi174.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-16T19:27:24.030 INFO:teuthology.orchestra.run.smithi174.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2024-03-16T19:27:24.313 INFO:teuthology.orchestra.run.smithi174.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 8.1 MB/s | 2.3 MB 00:00 2024-03-16T19:27:24.480 INFO:teuthology.orchestra.run.smithi174.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 11 MB/s | 1.8 MB 00:00 2024-03-16T19:27:24.556 INFO:teuthology.orchestra.run.smithi174.stdout:(154/174): librbd1-19.0.0-1681.gaba3c1d2.el9.x8 4.4 MB/s | 3.1 MB 00:00 2024-03-16T19:27:24.572 INFO:teuthology.orchestra.run.smithi174.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 462 kB/s | 42 kB 00:00 2024-03-16T19:27:24.614 INFO:teuthology.orchestra.run.smithi174.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.2 MB/s | 248 kB 00:00 2024-03-16T19:27:24.656 INFO:teuthology.orchestra.run.smithi174.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 855 kB/s | 71 kB 00:00 2024-03-16T19:27:24.740 INFO:teuthology.orchestra.run.smithi174.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-16T19:27:24.773 INFO:teuthology.orchestra.run.smithi174.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 230 kB/s | 36 kB 00:00 2024-03-16T19:27:24.807 INFO:teuthology.orchestra.run.smithi174.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 256 kB 00:00 2024-03-16T19:27:24.849 INFO:teuthology.orchestra.run.smithi174.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 305 kB/s | 23 kB 00:00 2024-03-16T19:27:24.899 INFO:teuthology.orchestra.run.smithi174.stdout:(162/174): librados2-19.0.0-1681.gaba3c1d2.el9. 2.9 MB/s | 3.4 MB 00:01 2024-03-16T19:27:24.925 INFO:teuthology.orchestra.run.smithi174.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 923 kB/s | 108 kB 00:00 2024-03-16T19:27:24.950 INFO:teuthology.orchestra.run.smithi174.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-16T19:27:24.998 INFO:teuthology.orchestra.run.smithi174.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-03-16T19:27:25.050 INFO:teuthology.orchestra.run.smithi174.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 509 kB/s | 64 kB 00:00 2024-03-16T19:27:25.075 INFO:teuthology.orchestra.run.smithi174.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 229 kB/s | 29 kB 00:00 2024-03-16T19:27:25.117 INFO:teuthology.orchestra.run.smithi174.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2024-03-16T19:27:25.218 INFO:teuthology.orchestra.run.smithi174.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 16 MB/s | 2.3 MB 00:00 2024-03-16T19:27:25.285 INFO:teuthology.orchestra.run.smithi174.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.2 MB/s | 2.2 MB 00:00 2024-03-16T19:27:25.326 INFO:teuthology.orchestra.run.smithi174.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 368 kB/s | 31 kB 00:00 2024-03-16T19:27:25.368 INFO:teuthology.orchestra.run.smithi174.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-16T19:27:25.410 INFO:teuthology.orchestra.run.smithi174.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 889 kB/s | 74 kB 00:00 2024-03-16T19:27:25.494 INFO:teuthology.orchestra.run.smithi174.stdout:(174/174): util-linux-core-2.37.4-18.el9.x86_64 1.2 MB/s | 465 kB 00:00 2024-03-16T19:27:25.500 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-16T19:27:25.500 INFO:teuthology.orchestra.run.smithi174.stdout:Total 9.0 MB/s | 224 MB 00:24 2024-03-16T19:27:25.950 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-16T19:27:26.220 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-16T19:27:26.220 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-16T19:27:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-16T19:27:27.600 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T19:27:27.733 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T19:27:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T19:27:27.914 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-16T19:27:27.915 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-16T19:27:27.918 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T19:27:27.967 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T19:27:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T19:27:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T19:27:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T19:27:28.135 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T19:27:28.461 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T19:27:28.516 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T19:27:28.663 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T19:27:28.692 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T19:27:28.770 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T19:27:28.805 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T19:27:28.851 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T19:27:29.025 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T19:27:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T19:27:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T19:27:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T19:27:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T19:27:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T19:27:29.311 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T19:27:29.333 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T19:27:29.349 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T19:27:29.515 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T19:27:29.552 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T19:27:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T19:27:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T19:27:29.918 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T19:27:29.978 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T19:27:30.255 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T19:27:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T19:27:30.653 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T19:27:30.758 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T19:27:30.852 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T19:27:30.884 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T19:27:30.914 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T19:27:30.956 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T19:27:31.028 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T19:27:31.082 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T19:27:31.120 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T19:27:31.185 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T19:27:31.219 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T19:27:31.263 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T19:27:31.296 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T19:27:31.324 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T19:27:31.391 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T19:27:31.428 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 39/198 2024-03-16T19:27:31.472 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 40/198 2024-03-16T19:27:31.500 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 41/198 2024-03-16T19:27:31.568 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T19:27:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T19:27:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T19:27:31.823 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T19:27:31.876 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T19:27:32.416 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T19:27:32.498 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T19:27:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T19:27:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T19:27:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T19:27:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T19:27:32.731 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T19:27:32.769 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 53/198 2024-03-16T19:27:32.801 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T19:27:33.104 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T19:27:33.341 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T19:27:33.425 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T19:27:33.469 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T19:27:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T19:27:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T19:27:33.718 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T19:27:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T19:27:33.823 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T19:27:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T19:27:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T19:27:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T19:27:33.985 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T19:27:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T19:27:34.161 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T19:27:34.207 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T19:27:34.237 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T19:27:34.278 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T19:27:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T19:27:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T19:27:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T19:27:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T19:27:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T19:27:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T19:27:34.723 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T19:27:34.828 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T19:27:34.869 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T19:27:34.918 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T19:27:34.982 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T19:27:35.041 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-16T19:27:35.139 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T19:27:35.173 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T19:27:35.173 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T19:27:35.204 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T19:27:35.213 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T19:27:35.223 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T19:27:35.253 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T19:27:35.306 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T19:27:35.340 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T19:27:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T19:27:35.374 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T19:27:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T19:27:35.416 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T19:27:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T19:27:35.450 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T19:27:35.478 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T19:27:35.513 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T19:27:35.552 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T19:27:35.585 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T19:27:35.613 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T19:27:35.717 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T19:27:35.736 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T19:27:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T19:27:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T19:27:35.783 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T19:27:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T19:27:35.858 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T19:27:35.905 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T19:27:35.934 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T19:27:35.956 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T19:27:36.024 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T19:27:36.045 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T19:27:36.055 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T19:27:36.089 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T19:27:36.097 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T19:27:36.120 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T19:27:36.167 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T19:27:36.265 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T19:27:36.271 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T19:27:36.319 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T19:27:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T19:27:36.347 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T19:27:36.433 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T19:27:36.457 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T19:27:36.504 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T19:27:36.543 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T19:27:36.566 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T19:27:36.582 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T19:27:36.723 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T19:27:36.755 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T19:27:36.787 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T19:27:36.892 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T19:27:36.991 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T19:27:37.041 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T19:27:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T19:27:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T19:27:37.116 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T19:27:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T19:27:37.175 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T19:27:37.183 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T19:27:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T19:27:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T19:27:37.387 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T19:27:37.490 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T19:27:37.496 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T19:27:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T19:27:37.584 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T19:27:37.604 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T19:27:37.613 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T19:27:37.640 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T19:27:37.683 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T19:27:37.757 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T19:27:37.811 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T19:27:37.850 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T19:27:37.913 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T19:27:37.949 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T19:27:37.992 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T19:27:38.025 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T19:27:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T19:27:38.283 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T19:27:38.284 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T19:27:38.284 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 39/198 2024-03-16T19:27:38.284 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 40/198 2024-03-16T19:27:38.284 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 41/198 2024-03-16T19:27:38.299 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T19:27:38.343 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T19:27:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T19:27:38.417 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T19:27:38.549 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T19:27:38.605 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T19:27:39.121 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T19:27:39.199 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T19:27:39.273 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T19:27:39.327 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T19:27:39.335 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T19:27:39.367 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T19:27:39.416 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T19:27:39.454 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 53/198 2024-03-16T19:27:39.484 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T19:27:39.584 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T19:27:39.667 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T19:27:39.698 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T19:27:39.719 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T19:27:39.728 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T19:27:39.756 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T19:27:39.904 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T19:27:39.969 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T19:27:40.012 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T19:27:40.066 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T19:27:40.094 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T19:27:40.148 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T19:27:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T19:27:40.194 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T19:27:40.224 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 125/198 2024-03-16T19:27:40.245 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T19:27:40.261 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T19:27:40.319 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T19:27:40.358 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T19:27:40.394 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T19:27:40.441 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T19:27:40.477 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T19:27:40.519 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T19:27:40.562 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T19:27:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T19:27:40.716 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T19:27:40.739 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T19:27:40.773 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T19:27:40.793 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T19:27:40.803 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T19:27:40.824 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 129/198 2024-03-16T19:27:40.844 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T19:27:40.939 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T19:27:40.978 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T19:27:41.017 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T19:27:41.043 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T19:27:41.068 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T19:27:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T19:27:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T19:27:41.368 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T19:27:41.409 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T19:27:41.457 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T19:27:41.512 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T19:27:41.696 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T19:27:41.738 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T19:27:41.877 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T19:27:41.911 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T19:27:41.942 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T19:27:41.973 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T19:27:42.001 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T19:27:42.036 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T19:27:42.074 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T19:27:42.106 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T19:27:42.136 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T19:27:42.212 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T19:27:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T19:27:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T19:27:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T19:27:42.356 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T19:27:42.444 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T19:27:42.534 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T19:27:42.578 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T19:27:42.609 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T19:27:42.656 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T19:27:42.754 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T19:27:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T19:27:43.075 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T19:27:43.112 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T19:27:43.530 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T19:27:43.574 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T19:27:43.608 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T19:27:43.655 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T19:27:43.669 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T19:27:43.706 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T19:27:43.734 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T19:27:43.766 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T19:27:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T19:27:43.804 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T19:27:43.804 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T19:27:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:27:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T19:27:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T19:27:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T19:27:44.066 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T19:27:44.692 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T19:27:44.793 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T19:27:46.000 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T19:27:46.080 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T19:27:46.111 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T19:27:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T19:27:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T19:27:46.475 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T19:27:46.508 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T19:27:46.597 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T19:27:46.638 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 125/198 2024-03-16T19:27:46.673 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T19:27:47.124 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T19:27:47.152 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T19:27:47.223 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T19:27:47.246 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 129/198 2024-03-16T19:27:49.471 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T19:27:49.508 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T19:27:50.049 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T19:27:50.302 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T19:27:50.518 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T19:27:50.518 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T19:27:50.519 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T19:27:50.519 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:27:50.545 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T19:27:55.198 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T19:27:55.198 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2024-03-16T19:27:55.198 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2024-03-16T19:27:55.198 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2024-03-16T19:27:55.198 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2024-03-16T19:27:55.198 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2024-03-16T19:27:55.198 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /root 2024-03-16T19:27:55.198 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2024-03-16T19:27:55.198 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:27:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T19:27:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T19:27:55.375 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:27:55.375 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T19:27:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:27:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T19:27:56.880 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T19:27:56.880 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:27:56.880 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T19:27:56.880 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T19:27:56.880 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T19:27:56.881 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:27:57.122 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T19:27:57.145 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T19:27:57.145 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:27:57.146 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T19:27:57.146 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T19:27:57.146 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T19:27:57.146 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:27:57.590 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T19:27:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T19:27:57.613 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:27:57.613 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T19:27:57.613 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T19:27:57.613 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T19:27:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:27:57.708 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T19:27:58.135 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T19:27:58.165 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T19:27:58.214 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 139/198 2024-03-16T19:27:58.274 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 140/198 2024-03-16T19:27:58.305 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T19:27:58.538 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T19:27:58.572 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T19:27:59.538 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T19:27:59.564 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T19:27:59.665 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T19:27:59.826 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 144/198 2024-03-16T19:27:59.858 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T19:27:59.882 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T19:27:59.882 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:27:59.882 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T19:27:59.882 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T19:27:59.882 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T19:27:59.882 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:27:59.933 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T19:27:59.945 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T19:27:59.992 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 147/198 2024-03-16T19:28:00.839 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T19:28:00.869 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T19:28:00.891 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T19:28:00.892 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:00.892 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T19:28:00.892 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T19:28:00.892 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T19:28:00.892 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:00.944 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T19:28:00.966 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T19:28:00.966 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:00.966 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T19:28:00.967 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:01.020 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T19:28:01.043 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T19:28:01.043 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:01.043 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T19:28:01.043 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T19:28:01.043 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T19:28:01.043 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T19:28:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T19:28:01.377 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:01.377 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T19:28:01.377 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T19:28:01.377 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T19:28:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:01.956 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T19:28:01.956 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-03-16T19:28:01.956 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-03-16T19:28:01.956 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-03-16T19:28:01.956 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-03-16T19:28:01.956 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-03-16T19:28:01.956 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-03-16T19:28:01.956 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-03-16T19:28:01.957 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T19:28:02.129 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T19:28:02.129 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:02.129 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T19:28:02.129 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:03.611 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T19:28:03.636 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T19:28:03.636 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:03.636 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T19:28:03.636 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T19:28:03.636 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T19:28:03.637 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:03.861 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T19:28:03.883 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T19:28:03.883 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:03.883 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T19:28:03.884 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T19:28:03.884 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T19:28:03.884 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:04.328 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T19:28:04.351 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T19:28:04.351 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:04.351 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T19:28:04.351 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T19:28:04.351 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T19:28:04.351 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:04.438 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T19:28:04.806 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T19:28:04.836 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T19:28:04.867 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 139/198 2024-03-16T19:28:04.928 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 140/198 2024-03-16T19:28:04.951 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T19:28:05.181 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T19:28:05.210 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T19:28:05.808 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T19:28:05.849 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 154/198 2024-03-16T19:28:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 155/198 2024-03-16T19:28:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T19:28:06.095 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 157/198 2024-03-16T19:28:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T19:28:06.194 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T19:28:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T19:28:06.246 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 159/198 2024-03-16T19:28:06.293 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T19:28:06.300 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T19:28:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T19:28:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T19:28:06.439 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 163/198 2024-03-16T19:28:06.457 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 144/198 2024-03-16T19:28:06.480 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T19:28:06.488 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T19:28:06.509 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T19:28:06.510 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T19:28:06.510 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:06.510 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T19:28:06.510 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T19:28:06.511 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T19:28:06.511 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:06.554 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T19:28:06.561 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T19:28:06.565 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T19:28:06.610 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 147/198 2024-03-16T19:28:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T19:28:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T19:28:07.255 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T19:28:07.289 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T19:28:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T19:28:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T19:28:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T19:28:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T19:28:07.512 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:07.512 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T19:28:07.512 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T19:28:07.512 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T19:28:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T19:28:07.579 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T19:28:07.579 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:07.579 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T19:28:07.579 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:07.624 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T19:28:07.646 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T19:28:07.646 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:07.646 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T19:28:07.647 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T19:28:07.647 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T19:28:07.647 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:07.967 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T19:28:07.989 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T19:28:07.989 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T19:28:07.989 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T19:28:07.990 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T19:28:07.990 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T19:28:07.990 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:08.175 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 170/198 2024-03-16T19:28:08.260 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T19:28:08.293 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T19:28:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T19:28:08.431 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T19:28:08.455 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T19:28:08.481 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T19:28:08.507 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T19:28:08.507 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T19:28:08.620 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T19:28:08.650 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T19:28:08.650 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T19:28:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T19:28:08.773 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T19:28:08.793 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T19:28:08.819 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T19:28:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T19:28:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T19:28:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T19:28:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T19:28:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T19:28:09.074 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T19:28:09.099 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T19:28:09.125 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T19:28:09.132 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T19:28:09.132 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T19:28:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T19:28:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T19:28:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T19:28:09.419 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T19:28:09.444 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T19:28:09.493 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T19:28:09.528 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T19:28:09.594 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T19:28:12.362 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T19:28:12.403 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 154/198 2024-03-16T19:28:12.434 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 155/198 2024-03-16T19:28:12.461 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T19:28:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 157/198 2024-03-16T19:28:12.768 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T19:28:12.808 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 159/198 2024-03-16T19:28:12.848 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T19:28:12.892 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T19:28:12.927 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T19:28:12.959 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 163/198 2024-03-16T19:28:13.000 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T19:28:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T19:28:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T19:28:13.153 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T19:28:13.186 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T19:28:13.770 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T19:28:13.809 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T19:28:13.828 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T19:28:14.464 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T19:28:14.464 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 1/198 2024-03-16T19:28:14.464 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 3/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 4/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 5/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 6/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 7/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 8/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 9/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 13/198 2024-03-16T19:28:14.465 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 14/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 15/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 17/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 18/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 19/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 20/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 21/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 22/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 23/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 24/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 25/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 26/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 27/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 28/198 2024-03-16T19:28:14.466 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 29/198 2024-03-16T19:28:14.481 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 30/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 31/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 32/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 33/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 34/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 35/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 36/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 37/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 38/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 39/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T19:28:14.482 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T19:28:14.483 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T19:28:14.484 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T19:28:14.485 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T19:28:14.486 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T19:28:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T19:28:14.488 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T19:28:14.489 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T19:28:14.490 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T19:28:14.491 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T19:28:14.492 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T19:28:14.493 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T19:28:14.494 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T19:28:14.494 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T19:28:14.494 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T19:28:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 170/198 2024-03-16T19:28:14.720 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T19:28:14.760 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T19:28:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T19:28:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T19:28:14.898 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T19:28:14.924 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T19:28:14.950 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T19:28:14.950 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T19:28:15.021 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T19:28:15.051 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T19:28:15.051 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T19:28:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T19:28:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T19:28:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T19:28:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T19:28:15.270 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T19:28:15.303 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T19:28:15.351 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T19:28:15.391 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T19:28:15.417 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T19:28:15.442 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T19:28:15.467 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T19:28:15.492 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T19:28:15.499 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T19:28:15.499 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T19:28:15.687 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T19:28:15.687 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T19:28:15.762 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T19:28:15.786 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T19:28:15.811 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T19:28:15.844 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T19:28:15.871 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T19:28:15.935 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T19:28:17.396 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T19:28:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:17.397 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2024-03-16T19:28:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T19:28:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T19:28:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T19:28:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T19:28:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T19:28:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T19:28:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T19:28:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T19:28:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T19:28:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T19:28:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T19:28:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T19:28:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T19:28:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.6-16.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T19:28:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T19:28:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T19:28:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T19:28:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T19:28:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T19:28:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T19:28:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T19:28:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T19:28:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T19:28:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T19:28:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T19:28:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T19:28:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T19:28:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T19:28:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:17.412 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-16T19:28:17.853 DEBUG:teuthology.parallel:result is None 2024-03-16T19:28:20.691 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T19:28:20.691 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 1/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 3/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 4/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 5/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 6/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 7/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 8/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 9/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 13/198 2024-03-16T19:28:20.692 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 14/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 15/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 17/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 18/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 19/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 20/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 21/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 22/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 23/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 24/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 25/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 26/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 27/198 2024-03-16T19:28:20.693 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 28/198 2024-03-16T19:28:20.694 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 29/198 2024-03-16T19:28:20.694 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 30/198 2024-03-16T19:28:20.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 31/198 2024-03-16T19:28:20.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 32/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 33/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 34/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 35/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 36/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 37/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 38/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 39/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T19:28:20.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T19:28:20.711 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T19:28:20.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T19:28:20.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T19:28:20.714 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T19:28:20.715 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T19:28:20.716 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T19:28:20.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T19:28:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T19:28:20.719 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T19:28:20.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T19:28:20.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T19:28:20.722 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T19:28:20.722 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T19:28:20.722 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T19:28:20.722 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T19:28:20.722 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T19:28:20.722 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T19:28:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T19:28:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: jq-1.6-16.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T19:28:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T19:28:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T19:28:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T19:28:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T19:28:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T19:28:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T19:28:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T19:28:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T19:28:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T19:28:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:23.488 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-16T19:28:23.936 DEBUG:teuthology.parallel:result is None 2024-03-16T19:28:23.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:28:24.242 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T19:28:24.268 INFO:teuthology.orchestra.run.smithi172.stdout:19.0.0-1681.gaba3c1d2.el9 2024-03-16T19:28:24.268 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1681.gaba3c1d2.el9 2024-03-16T19:28:24.268 INFO:teuthology.task.install:The correct ceph version 19.0.0-1681.gaba3c1d2 is installed. 2024-03-16T19:28:24.270 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:28:24.574 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T19:28:24.597 INFO:teuthology.orchestra.run.smithi174.stdout:19.0.0-1681.gaba3c1d2.el9 2024-03-16T19:28:24.597 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1681.gaba3c1d2.el9 2024-03-16T19:28:24.597 INFO:teuthology.task.install:The correct ceph version 19.0.0-1681.gaba3c1d2 is installed. 2024-03-16T19:28:24.599 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-16T19:28:24.599 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:24.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T19:28:24.634 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:24.634 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T19:28:24.669 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-16T19:28:24.670 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:24.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T19:28:24.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T19:28:24.774 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:24.774 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T19:28:24.802 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T19:28:24.870 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-16T19:28:24.870 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:24.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T19:28:24.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T19:28:24.969 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:24.969 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T19:28:24.997 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T19:28:25.065 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-16T19:28:25.065 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:25.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T19:28:25.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T19:28:25.161 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:25.161 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T19:28:25.189 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T19:28:25.256 INFO:teuthology.run_tasks:Running task ceph... 2024-03-16T19:28:25.339 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-16T19:28:25.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2024-03-16T19:28:25.341 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /var/log/ceph 2024-03-16T19:28:25.368 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-16T19:28:25.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-16T19:28:25.409 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-16T19:28:25.436 INFO:tasks.ceph:Creating extra log directories... 2024-03-16T19:28:25.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-16T19:28:25.478 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-16T19:28:25.510 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-16T19:28:25.510 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-16T19:28:25.510 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-16T19:28:25.510 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:25.511 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-16T19:28:25.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-16T19:28:25.677 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-16T19:28:25.704 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:25.704 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-16T19:28:25.719 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-16T19:28:25.837 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-16T19:28:25.869 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-16T19:28:25.869 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-16T19:28:25.869 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602108', 'branch': 'wip-lusov-qdb-squid', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}', 'email': None, 'first_in_suite': False, 'job_id': '7602108', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb-squid'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb-squid', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 6918, 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'sleep_before_teardown': 0, 'subset': '111/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb-squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRsXkXxfZSNA6a7epG+n39p3eYTeaH0kgIkfHYLXj4ijGrI5/lmYi+jnEcuid+o40QXqd1woHEZLdI87ygM77Y=', 'smithi174.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBADm3bpJk5YJJHDFYSEn665kH7RZaDY4QpUnVqq1EXzhLtmsb4pxAn/p5mB2nA/tywJfgLoA78z8nMrDdcGtL8w='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'cluster': 'ceph'}}, {'kclient': None}, {'thrashosds': None}, {'workunit': {'clients': {'all': ['fs/misc/trivial_sync.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-15_11:44:52', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-16T19:28:25.870 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-16T19:28:25.872 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-16T19:28:25.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-03-16T19:28:25.927 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/run/ceph 2024-03-16T19:28:25.985 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:25.985 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T19:28:26.007 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T19:28:26.007 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-03-16T19:28:26.061 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T19:28:26.061 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:28:26.062 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-16T19:28:26.062 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:28:26.062 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T19:28:26.062 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-16 19:28:09.948073366 +0000 2024-03-16T19:28:26.062 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-16 19:25:32.008803075 +0000 2024-03-16T19:28:26.062 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-16 19:25:32.008803075 +0000 2024-03-16T19:28:26.062 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-16 19:25:32.008803075 +0000 2024-03-16T19:28:26.062 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T19:28:26.127 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-16T19:28:26.127 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-16T19:28:26.128 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000204846 s, 2.5 MB/s 2024-03-16T19:28:26.129 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T19:28:26.184 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-03-16T19:28:26.239 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T19:28:26.240 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:28:26.240 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-16T19:28:26.240 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:28:26.240 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T19:28:26.240 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-16 19:28:09.948073366 +0000 2024-03-16T19:28:26.240 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-16 19:25:32.349795379 +0000 2024-03-16T19:28:26.240 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-16 19:25:32.349795379 +0000 2024-03-16T19:28:26.240 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-16 19:25:32.349795379 +0000 2024-03-16T19:28:26.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T19:28:26.305 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-16T19:28:26.305 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-16T19:28:26.305 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000207315 s, 2.5 MB/s 2024-03-16T19:28:26.306 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T19:28:26.362 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-03-16T19:28:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T19:28:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:28:26.417 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-16T19:28:26.417 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:28:26.417 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T19:28:26.417 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-16 19:28:09.949073343 +0000 2024-03-16T19:28:26.417 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-16 19:25:32.689787706 +0000 2024-03-16T19:28:26.417 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-16 19:25:32.689787706 +0000 2024-03-16T19:28:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-16 19:25:32.689787706 +0000 2024-03-16T19:28:26.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T19:28:26.482 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-16T19:28:26.482 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-16T19:28:26.482 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000192766 s, 2.7 MB/s 2024-03-16T19:28:26.483 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T19:28:26.539 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-03-16T19:28:26.595 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T19:28:26.595 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:28:26.595 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-16T19:28:26.595 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:28:26.595 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T19:28:26.595 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-16 19:28:09.949073343 +0000 2024-03-16T19:28:26.595 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-16 19:25:33.038779830 +0000 2024-03-16T19:28:26.595 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-16 19:25:33.038779830 +0000 2024-03-16T19:28:26.595 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-16 19:25:33.038779830 +0000 2024-03-16T19:28:26.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T19:28:26.660 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-16T19:28:26.660 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-16T19:28:26.660 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000163566 s, 3.1 MB/s 2024-03-16T19:28:26.661 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T19:28:26.716 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:26.717 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:26.717 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T19:28:26.733 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T19:28:26.733 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-03-16T19:28:26.788 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T19:28:26.788 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:28:26.788 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-16T19:28:26.788 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:28:26.788 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T19:28:26.788 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-16 19:28:16.281435335 +0000 2024-03-16T19:28:26.788 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-16 19:26:09.848261792 +0000 2024-03-16T19:28:26.789 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-16 19:26:09.848261792 +0000 2024-03-16T19:28:26.789 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-16 19:26:09.848261792 +0000 2024-03-16T19:28:26.789 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T19:28:26.854 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-16T19:28:26.855 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-16T19:28:26.855 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000263432 s, 1.9 MB/s 2024-03-16T19:28:26.856 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T19:28:26.913 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-03-16T19:28:26.968 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T19:28:26.968 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:28:26.968 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-16T19:28:26.968 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:28:26.968 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T19:28:26.969 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-16 19:28:16.281435335 +0000 2024-03-16T19:28:26.969 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-16 19:26:10.200256831 +0000 2024-03-16T19:28:26.969 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-16 19:26:10.200256831 +0000 2024-03-16T19:28:26.969 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-16 19:26:10.200256831 +0000 2024-03-16T19:28:26.969 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T19:28:27.034 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-16T19:28:27.034 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-16T19:28:27.034 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000200625 s, 2.6 MB/s 2024-03-16T19:28:27.035 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T19:28:27.090 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-03-16T19:28:27.144 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T19:28:27.145 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:28:27.145 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-16T19:28:27.145 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:28:27.145 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T19:28:27.145 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-16 19:28:16.281435335 +0000 2024-03-16T19:28:27.145 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-16 19:26:10.518252350 +0000 2024-03-16T19:28:27.145 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-16 19:26:10.518252350 +0000 2024-03-16T19:28:27.145 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-16 19:26:10.518252350 +0000 2024-03-16T19:28:27.145 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T19:28:27.210 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-16T19:28:27.210 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-16T19:28:27.210 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000191164 s, 2.7 MB/s 2024-03-16T19:28:27.211 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T19:28:27.267 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-03-16T19:28:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T19:28:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T19:28:27.321 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-16T19:28:27.322 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:28:27.322 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T19:28:27.322 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-16 19:28:16.282435321 +0000 2024-03-16T19:28:27.322 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-16 19:26:10.860247531 +0000 2024-03-16T19:28:27.322 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-16 19:26:10.860247531 +0000 2024-03-16T19:28:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-16 19:26:10.860247531 +0000 2024-03-16T19:28:27.322 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T19:28:27.387 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-16T19:28:27.387 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-16T19:28:27.387 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000192866 s, 2.7 MB/s 2024-03-16T19:28:27.388 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T19:28:27.444 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:27.444 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-16T19:28:27.444 INFO:tasks.ceph:Generating config... 2024-03-16T19:28:27.445 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-16T19:28:27.445 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-03-16T19:28:27.445 INFO:tasks.ceph:[client] debug client = 20 2024-03-16T19:28:27.445 INFO:tasks.ceph:[client] debug ms = 1 2024-03-16T19:28:27.446 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-16T19:28:27.446 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-16T19:28:27.446 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2024-03-16T19:28:27.446 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] ms inject delay max = 1 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-16T19:28:27.447 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-16T19:28:27.448 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-16T19:28:27.448 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-16T19:28:27.448 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] ms inject delay max = 1 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-16T19:28:27.449 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-16T19:28:27.449 INFO:tasks.ceph:Setting up mon.a... 2024-03-16T19:28:27.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-16T19:28:27.503 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.keyring 2024-03-16T19:28:27.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-16T19:28:27.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-16T19:28:27.576 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.172'), ('mon.b', '172.21.15.174'), ('mon.c', '[v2:172.21.15.174:3301,v1:172.21.15.174:6790]')] 2024-03-16T19:28:27.577 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.172,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-16T19:28:27.577 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:27.578 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-16T19:28:27.630 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.172 --add b 172.21.15.174 --addv c '[v2:172.21.15.174:3301,v1:172.21.15.174:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:27.715 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:27.715 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: generated fsid 6c00e600-a879-4eb6-acee-0fdf0f76922a 2024-03-16T19:28:27.715 INFO:teuthology.orchestra.run.smithi172.stdout:setting min_mon_release = quincy 2024-03-16T19:28:27.715 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 0 2024-03-16T19:28:27.715 INFO:teuthology.orchestra.run.smithi172.stdout:fsid 6c00e600-a879-4eb6-acee-0fdf0f76922a 2024-03-16T19:28:27.715 INFO:teuthology.orchestra.run.smithi172.stdout:last_changed 2024-03-16T19:28:27.734612+0000 2024-03-16T19:28:27.715 INFO:teuthology.orchestra.run.smithi172.stdout:created 2024-03-16T19:28:27.734612+0000 2024-03-16T19:28:27.715 INFO:teuthology.orchestra.run.smithi172.stdout:min_mon_release 17 (quincy) 2024-03-16T19:28:27.715 INFO:teuthology.orchestra.run.smithi172.stdout:election_strategy: 1 2024-03-16T19:28:27.716 INFO:teuthology.orchestra.run.smithi172.stdout:0: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.a 2024-03-16T19:28:27.716 INFO:teuthology.orchestra.run.smithi172.stdout:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-03-16T19:28:27.716 INFO:teuthology.orchestra.run.smithi172.stdout:2: [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] mon.c 2024-03-16T19:28:27.716 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-16T19:28:27.717 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-16T19:28:27.732 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6c00e600-a879-4eb6-acee-0fdf0f76922a... 2024-03-16T19:28:27.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-16T19:28:27.773 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-16T19:28:27.821 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-03-16T19:28:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-03-16T19:28:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-03-16T19:28:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-16T19:28:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = "172.21.15.172,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]" 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on skipped message = False 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 6c00e600-a879-4eb6-acee-0fdf0f76922a 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-03-16T19:28:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug trim objects = true 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: bdev async discard = True 2024-03-16T19:28:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: bdev enable discard = True 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore allocator = bitmap 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 1/20 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 1/20 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 4/10 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject delay max = 1 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject delay probability = 0.005 2024-03-16T19:28:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject delay type = client mds 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 2500 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: osd blocked scrub grace period = 3600 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: osd max markdown count = 1000 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: osd op complaint time = 180 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-03-16T19:28:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-03-16T19:28:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: mon op complaint time = 120 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-03-16T19:28:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: client mount timeout = 600 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: client_shutdown_timeout = 120 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: debug client = 20 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: rados mon op timeout = 900 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: rados osd op timeout = 900 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.b] 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.c] 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.a] 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.c] 2024-03-16T19:28:27.831 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.e] 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.b] 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.d] 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.f] 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout:[mds] 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds = 20 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds balancer = 20 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: mds bal fragment size max = 10000 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: mds bal merge size = 5 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: mds bal split bits = 3 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: mds bal split size = 100 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: mds debug frag = True 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: mds debug scatterstat = True 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: mds oft prefetch dirfrags = False 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: mds op complaint time = 180 2024-03-16T19:28:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: mds verify scatter = True 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: mds_dir_prefetch = False 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: mds_sleep_rank_change = 5000000.0 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject delay max = 1 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject delay probability = 0.005 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject delay type = client mds osd 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 2500 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: osd op complaint time = 180 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: rados mon op timeout = 900 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: rados osd op timeout = 900 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2024-03-16T19:28:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = cephx 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2024-03-16T19:28:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: mon_allow_pool_size_one = true 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = "172.21.15.172,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]" 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on skipped message = False 2024-03-16T19:28:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = 6c00e600-a879-4eb6-acee-0fdf0f76922a 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 600 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock profile = high_recovery_ops 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2024-03-16T19:28:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug trim objects = true 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2024-03-16T19:28:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: bdev async discard = True 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: bdev enable discard = True 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore allocator = bitmap 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore block size = 96636764160 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore fsck on mount = True 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluefs = 1/20 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluestore = 1/20 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 20 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: debug rocksdb = 4/10 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: mon client directed command retry = 5 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd backfillfull_ratio = 0.85 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd full ratio = 0.9 2024-03-16T19:28:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd nearfull ratio = 0.8 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay max = 1 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay probability = 0.005 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay type = client mds 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject socket failures = 2500 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd blocked scrub grace period = 3600 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd failsafe full ratio = 0.95 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd max markdown count = 1000 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = bluestore 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-03-16T19:28:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 20 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2024-03-16T19:28:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: # 1m isn't quite enough 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: mon_down_mkfs_grace = 2m 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_filestore_osds = false 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: mon op complaint time = 120 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: client mount timeout = 600 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: client_shutdown_timeout = 120 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: debug client = 20 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 900 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 900 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.b] 2024-03-16T19:28:27.843 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.c] 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.a] 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.c] 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.e] 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.b] 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.d] 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.f] 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout:[mds] 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds = 20 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds balancer = 20 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal fragment size max = 10000 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal merge size = 5 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split bits = 3 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split size = 100 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug frag = True 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug scatterstat = True 2024-03-16T19:28:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: mds oft prefetch dirfrags = False 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: mds op complaint time = 180 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: mds verify scatter = True 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: mds_dir_prefetch = False 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: mds_sleep_rank_change = 5000000.0 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: mon client directed command retry = 5 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay max = 1 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay probability = 0.005 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay type = client mds osd 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject socket failures = 2500 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 900 2024-03-16T19:28:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 900 2024-03-16T19:28:27.847 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-16T19:28:27.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-16T19:28:27.902 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-16T19:28:27.902 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:27.902 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-16T19:28:27.916 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:27.916 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-16T19:28:27.971 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-03-16T19:28:27.971 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:27.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-16T19:28:27.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-16T19:28:28.050 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:28.050 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:28.105 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi174.front.sepia.ceph.com 2024-03-16T19:28:28.105 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:28.105 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-16T19:28:28.105 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-16T19:28:28.145 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:28.145 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:28.199 INFO:tasks.ceph:Setting up mon nodes... 2024-03-16T19:28:28.199 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-16T19:28:28.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-16T19:28:28.255 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-16T19:28:28.257 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-16T19:28:28.316 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-16T19:28:28.319 INFO:tasks.ceph:Setting up mds nodes... 2024-03-16T19:28:28.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-16T19:28:28.374 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-16T19:28:28.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-16T19:28:28.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-16T19:28:28.497 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-16T19:28:28.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-16T19:28:28.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-16T19:28:28.619 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-16T19:28:28.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-16T19:28:28.648 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-16T19:28:28.703 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-16T19:28:28.706 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-16T19:28:28.731 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-16T19:28:28.827 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-16T19:28:28.829 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-16T19:28:28.856 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-16T19:28:28.950 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-16T19:28:28.952 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-16T19:28:28.979 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-16T19:28:28.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-16T19:28:29.021 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-16T19:28:29.036 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-16T19:28:29.037 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-16T19:28:29.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-16T19:28:29.104 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:29.104 INFO:tasks.ceph:role: osd.0 2024-03-16T19:28:29.104 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-16T19:28:29.104 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-16T19:28:29.178 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:28:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:28:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:28:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-16T19:28:29.178 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:28:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:28:29.179 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:28:29.179 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-16T19:28:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:28:29.179 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:28:29.656 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-16T19:28:29.658 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-16T19:28:29.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-16T19:28:29.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-16T19:28:29.807 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-16T19:28:29.876 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:29.876 INFO:tasks.ceph:role: osd.1 2024-03-16T19:28:29.876 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-16T19:28:29.876 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-16T19:28:29.952 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:28:29.952 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:28:29.953 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:28:29.953 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-16T19:28:29.953 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:28:29.953 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:28:29.953 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:28:29.953 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-16T19:28:29.953 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:28:29.953 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:28:30.410 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-16T19:28:30.413 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-16T19:28:30.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-16T19:28:30.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-16T19:28:30.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-16T19:28:30.627 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:30.627 INFO:tasks.ceph:role: osd.2 2024-03-16T19:28:30.628 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-16T19:28:30.628 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-16T19:28:30.701 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:28:30.701 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:28:30.702 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:28:30.702 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-16T19:28:30.702 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:28:30.702 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:28:30.702 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:28:30.702 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-16T19:28:30.702 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:28:30.702 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:28:31.169 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-16T19:28:31.172 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-16T19:28:31.172 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-16T19:28:31.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-16T19:28:31.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-16T19:28:31.364 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:31.364 INFO:tasks.ceph:role: osd.3 2024-03-16T19:28:31.364 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-16T19:28:31.364 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-16T19:28:31.438 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:28:31.438 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:28:31.438 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:28:31.439 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-16T19:28:31.439 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:28:31.439 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:28:31.439 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:28:31.439 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-16T19:28:31.439 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:28:31.439 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:28:31.948 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-16T19:28:31.950 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-16T19:28:31.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-16T19:28:32.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-16T19:28:32.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:32.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:32.214+0000 7f3549811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-16T19:28:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:32.221+0000 7f3549811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-16T19:28:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:32.221+0000 7f3549811640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-16T19:28:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:32.221+0000 7f3549811640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-16T19:28:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:32.221+0000 7f3549811640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-16T19:28:32.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:32.226+0000 7f3549811640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-16T19:28:35.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-16T19:28:35.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:35.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:35.191+0000 7efca5ff8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-16T19:28:35.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:35.192+0000 7efca5ff8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-16T19:28:35.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:35.192+0000 7efca5ff8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-16T19:28:35.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:35.192+0000 7efca5ff8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-16T19:28:35.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:35.192+0000 7efca5ff8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-16T19:28:35.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:35.192+0000 7efca5ff8640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-16T19:28:38.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-16T19:28:38.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:38.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:38.153+0000 7f3c6b791640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-16T19:28:38.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:38.154+0000 7f3c6b791640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-16T19:28:38.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:38.154+0000 7f3c6b791640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-16T19:28:38.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:38.154+0000 7f3c6b791640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-16T19:28:38.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:38.154+0000 7f3c6b791640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-16T19:28:38.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:38.154+0000 7f3c6b791640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-16T19:28:40.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-16T19:28:41.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:41.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:41.124+0000 7fa6f20d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-16T19:28:41.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:41.125+0000 7fa6f20d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-16T19:28:41.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:41.125+0000 7fa6f20d9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-16T19:28:41.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:41.125+0000 7fa6f20d9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-16T19:28:41.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:41.125+0000 7fa6f20d9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-16T19:28:41.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:28:41.125+0000 7fa6f20d9640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-16T19:28:43.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-16T19:28:43.996 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-16T19:28:44.028 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:44.028 INFO:tasks.ceph:role: osd.4 2024-03-16T19:28:44.028 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-16T19:28:44.028 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:28:44.106 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:28:44.631 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-03-16T19:28:44.634 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-03-16T19:28:44.634 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-16T19:28:44.747 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-16T19:28:44.781 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-16T19:28:44.855 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:44.855 INFO:tasks.ceph:role: osd.5 2024-03-16T19:28:44.856 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-16T19:28:44.856 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-16T19:28:44.933 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:28:44.933 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:28:44.933 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:28:44.933 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-16T19:28:44.933 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:28:44.933 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:28:44.933 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:28:44.933 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-16T19:28:44.933 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:28:44.934 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:28:45.375 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-03-16T19:28:45.377 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-03-16T19:28:45.378 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-16T19:28:45.468 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-16T19:28:45.498 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-16T19:28:45.564 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:45.564 INFO:tasks.ceph:role: osd.6 2024-03-16T19:28:45.565 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-16T19:28:45.565 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-16T19:28:45.638 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:28:45.639 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:28:45.639 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:28:45.639 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-16T19:28:45.639 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:28:45.639 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:28:45.639 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:28:45.639 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-16T19:28:45.639 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:28:45.639 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:28:46.133 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-03-16T19:28:46.135 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-03-16T19:28:46.135 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-16T19:28:46.282 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-16T19:28:46.314 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-16T19:28:46.381 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-16T19:28:46.381 INFO:tasks.ceph:role: osd.7 2024-03-16T19:28:46.382 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-16T19:28:46.382 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-16T19:28:46.456 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-16T19:28:46.971 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-03-16T19:28:46.973 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-03-16T19:28:46.973 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-16T19:28:47.119 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-16T19:28:47.151 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:47.254 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:47.272+0000 7fd38cdac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-16T19:28:47.254 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:47.273+0000 7fd38cdac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-16T19:28:47.254 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:47.273+0000 7fd38cdac640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-16T19:28:47.254 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:47.273+0000 7fd38cdac640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-16T19:28:47.254 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:47.273+0000 7fd38cdac640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-16T19:28:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:47.273+0000 7fd38cdac640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-16T19:28:50.103 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-16T19:28:50.134 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:50.227 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:50.245+0000 7fab92212640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-16T19:28:50.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:50.246+0000 7fab92212640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-16T19:28:50.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:50.246+0000 7fab92212640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-16T19:28:50.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:50.246+0000 7fab92212640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-16T19:28:50.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:50.246+0000 7fab92212640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-16T19:28:50.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:50.246+0000 7fab92212640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-16T19:28:53.080 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-16T19:28:53.111 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:53.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:53.218+0000 7f31d5e0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-16T19:28:53.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:53.219+0000 7f31d5e0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-16T19:28:53.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:53.219+0000 7f31d5e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-16T19:28:53.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:53.219+0000 7f31d5e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-16T19:28:53.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:53.219+0000 7f31d5e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-16T19:28:53.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:53.219+0000 7f31d5e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-16T19:28:55.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:28:55.874 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:28:56.051 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-16T19:28:56.078 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:28:56.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:56.181+0000 7ff76a4a8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-16T19:28:56.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:56.182+0000 7ff76a4a8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-16T19:28:56.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:56.182+0000 7ff76a4a8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-16T19:28:56.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:56.182+0000 7ff76a4a8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-16T19:28:56.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:56.182+0000 7ff76a4a8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-16T19:28:56.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:28:56.182+0000 7ff76a4a8640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-16T19:28:59.016 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-16T19:28:59.047 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-16T19:28:59.047 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:59.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-16T19:28:59.079 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:59.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-16T19:28:59.149 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:59.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-16T19:28:59.219 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:59.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-16T19:28:59.287 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:59.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-16T19:28:59.355 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:59.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-16T19:28:59.422 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:59.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-16T19:28:59.490 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:28:59.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-16T19:28:59.556 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:59.556 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-16T19:28:59.587 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:59.587 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-16T19:28:59.658 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:59.658 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-16T19:28:59.728 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:59.728 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-16T19:28:59.796 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:59.796 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-16T19:28:59.864 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:59.864 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-16T19:28:59.932 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:28:59.932 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-16T19:29:00.000 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:29:00.000 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-16T19:29:00.066 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:29:00.067 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-16T19:29:00.082 INFO:tasks.ceph:Adding keys to all mons... 2024-03-16T19:29:00.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-16T19:29:00.125 DEBUG:teuthology.orchestra.run.smithi174:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.y] 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDc8vVlEIFbEBAAEx0IlS9nAJ9Z7zwgYbcmuw== 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.a] 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDc8vVl5xt0FxAAcMJdoSVBwVPjhk45fFgUBg== 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.c] 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDc8vVlrcHFHhAA67i60MrMSWwfuQkbpm92zQ== 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.e] 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDc8vVl3PQZJhAAqFLQ1AIUxJY7ptOyiOlYiw== 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDg8vVl/PHiDBAAakiH9iabWCEwfhp5/BP6DQ== 2024-03-16T19:29:00.149 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDj8vVlCGV5CxAAEI4UeB8T9uPVFzILbVKbnQ== 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDm8vVl4QM0CRAAWkjIeoPc131VnaGkbKiAqQ== 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDp8vVlL8V4BxAAh5O661tEsL2sw1hu3t+nYg== 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDc8vVlRO3+ExAAGVcps1X/BaFaTdl4hclq7A== 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.b] 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDc8vVloNMUKxAAtJOILi/7CDzOlSRLmuEMaw== 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.d] 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDc8vVlgrB3MhAAfu2081LWMFf36eZpuSxtOw== 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout:[mds.f] 2024-03-16T19:29:00.150 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDc8vVlo03PORAATnLY/wmcTs1rFBSSErwMMA== 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.4] 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDv8vVlq31JEBAAQYEspKfEFI/Uh0EHslan1Q== 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.5] 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDy8vVlqyO+DhAAO1rdRpBaJnSSsRsiub2wnQ== 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.6] 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQD18vVl7hchDRAAZg5QE+fFfDTncijQj9UUGg== 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.7] 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQD48vVlFPrtChAALYdtP8bIq5nJG7GZUxGcaQ== 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2024-03-16T19:29:00.151 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDd8vVl49VyAhAAv0Y8YH7R4VA8yJM86B1CNg== 2024-03-16T19:29:00.152 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.y] 2024-03-16T19:29:00.152 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDc8vVlEIFbEBAAEx0IlS9nAJ9Z7zwgYbcmuw== 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.a] 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDc8vVl5xt0FxAAcMJdoSVBwVPjhk45fFgUBg== 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.c] 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDc8vVlrcHFHhAA67i60MrMSWwfuQkbpm92zQ== 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.e] 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDc8vVl3PQZJhAAqFLQ1AIUxJY7ptOyiOlYiw== 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.0] 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDg8vVl/PHiDBAAakiH9iabWCEwfhp5/BP6DQ== 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.1] 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDj8vVlCGV5CxAAEI4UeB8T9uPVFzILbVKbnQ== 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.2] 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDm8vVl4QM0CRAAWkjIeoPc131VnaGkbKiAqQ== 2024-03-16T19:29:00.153 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.3] 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDp8vVlL8V4BxAAh5O661tEsL2sw1hu3t+nYg== 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.x] 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDc8vVlRO3+ExAAGVcps1X/BaFaTdl4hclq7A== 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.b] 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDc8vVloNMUKxAAtJOILi/7CDzOlSRLmuEMaw== 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.d] 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDc8vVlgrB3MhAAfu2081LWMFf36eZpuSxtOw== 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.f] 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDc8vVlo03PORAATnLY/wmcTs1rFBSSErwMMA== 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.4] 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDv8vVlq31JEBAAQYEspKfEFI/Uh0EHslan1Q== 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.5] 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDy8vVlqyO+DhAAO1rdRpBaJnSSsRsiub2wnQ== 2024-03-16T19:29:00.154 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.6] 2024-03-16T19:29:00.155 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQD18vVl7hchDRAAZg5QE+fFfDTncijQj9UUGg== 2024-03-16T19:29:00.155 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.7] 2024-03-16T19:29:00.155 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQD48vVlFPrtChAALYdtP8bIq5nJG7GZUxGcaQ== 2024-03-16T19:29:00.155 INFO:teuthology.orchestra.run.smithi174.stdout:[client.0] 2024-03-16T19:29:00.155 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDd8vVl49VyAhAAv0Y8YH7R4VA8yJM86B1CNg== 2024-03-16T19:29:00.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-16T19:29:00.197 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-16T19:29:00.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.250 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.306 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.362 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.418 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.471 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.524 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.578 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-16T19:29:00.633 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-16T19:29:00.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.686 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.742 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.800 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-16T19:29:00.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.857 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.910 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:00.964 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:01.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:01.017 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-16T19:29:01.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-16T19:29:01.071 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-16T19:29:01.124 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-16T19:29:01.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-16T19:29:01.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-16T19:29:01.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-16T19:29:01.273 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-16T19:29:01.300 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-16T19:29:01.396 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-16T19:29:01.422 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-16T19:29:01.489 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-16T19:29:01.587 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-16T19:29:01.613 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:29:01.616 DEBUG:teuthology.orchestra.run.smithi174:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-16T19:29:01.668 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-16T19:29:01.668 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-16T19:29:01.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-16T19:29:01.671 INFO:tasks.ceph.mon.a:Started 2024-03-16T19:29:01.671 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-16T19:29:01.671 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-16T19:29:01.710 INFO:tasks.ceph.mon.b:Started 2024-03-16T19:29:01.711 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-16T19:29:01.711 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-16T19:29:01.713 INFO:tasks.ceph.mon.c:Started 2024-03-16T19:29:01.713 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-16T19:29:01.713 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-16T19:29:01.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-16T19:29:01.715 INFO:tasks.ceph.mgr.y:Started 2024-03-16T19:29:01.715 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-16T19:29:01.715 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-16T19:29:01.717 INFO:tasks.ceph.mgr.x:Started 2024-03-16T19:29:01.717 INFO:tasks.ceph:Setting crush tunables to default 2024-03-16T19:29:01.717 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd crush tunables default 2024-03-16T19:29:01.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.836+0000 7f2aba2ae640 1 Processor -- start 2024-03-16T19:29:01.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.836+0000 7f2aba2ae640 1 -- start start 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.836+0000 7f2aba2ae640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2ab415b700 0x7f2ab40b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.836+0000 7f2aba2ae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 0x7f2ab40ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.836+0000 7f2aba2ae640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab406e990 con 0x7f2ab406f640 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.836+0000 7f2aba2ae640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab415bae0 con 0x7f2ab415b700 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.836+0000 7f2aba2ae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab406bfe0 con 0x7f2ab40ce320 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab3fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2ab415b700 0x7f2ab40b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab37fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2ab406f640 0x7f2ab406eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:58944/0 (socket says 172.21.15.172:58944) 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab3fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f2ab415b700 0x7f2ab40b0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42800/0 (socket says 172.21.15.172:42800) 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab37fe640 1 -- 172.21.15.172:0/596882486 learned_addr learned my addr 172.21.15.172:0/596882486 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab3fff640 1 -- 172.21.15.172:0/596882486 >> v2:172.21.15.172:3300/0 conn(0x7f2ab415b700 msgr2=0x7f2ab40b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab3fff640 1 -- 172.21.15.172:0/596882486 >> v2:172.21.15.172:3300/0 conn(0x7f2ab415b700 msgr2=0x7f2ab40b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:29:01.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab3fff640 1 --2- 172.21.15.172:0/596882486 >> v2:172.21.15.172:3300/0 conn(0x7f2ab415b700 0x7f2ab40b0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:29:01.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab884b640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 0x7f2ab40ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:01.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab3fff640 1 --2- 172.21.15.172:0/596882486 >> v2:172.21.15.172:3300/0 conn(0x7f2ab415b700 0x7f2ab40b0df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:29:01.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab884b640 1 -- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 msgr2=0x7f2ab40ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-16T19:29:01.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab884b640 1 -- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 msgr2=0x7f2ab40ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:29:01.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab884b640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 0x7f2ab40ce750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:29:01.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:01.837+0000 7f2ab884b640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 0x7f2ab40ce750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:29:02.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249949781 0 0) 0x7f2aa8001110 con 0x7f2ab406f640 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a9c003520 con 0x7f2ab406f640 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3726616071 0 0) 0x7f2a9c003520 con 0x7f2ab406f640 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 msgr2=0x7f2ab40ce750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 0x7f2ab40ce750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 >> v2:172.21.15.172:3300/0 conn(0x7f2ab415b700 msgr2=0x7f2ab40b0df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 --2- 172.21.15.172:0/596882486 >> v2:172.21.15.172:3300/0 conn(0x7f2ab415b700 0x7f2ab40b0df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab406f260 con 0x7f2ab406f640 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f2aa8001820 con 0x7f2ab406f640 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 >> v1:172.21.15.172:6789/0 conn(0x7f2ab406f640 legacy=0x7f2ab406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c003970 0x7f2a9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 0x7f2a9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2a9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab415bae0 con 0x7f2a9c004d00 2024-03-16T19:29:02.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab406bfe0 con 0x7f2a9c003970 2024-03-16T19:29:02.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c00bae0 con 0x7f2ab40ce320 2024-03-16T19:29:02.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab884b640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2a9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.019+0000 7f2ab3fff640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c003970 0x7f2a9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.020+0000 7f2ab884b640 1 -- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 msgr2=0x7f2a9c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:02.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.020+0000 7f2ab884b640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40ce320 0x7f2a9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.020+0000 7f2ab884b640 1 -- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c003970 msgr2=0x7f2a9c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:02.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.020+0000 7f2ab884b640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c003970 0x7f2a9c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.020+0000 7f2ab884b640 1 -- 172.21.15.172:0/596882486 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c003520 con 0x7f2a9c004d00 2024-03-16T19:29:02.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.021+0000 7f2ab884b640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2a9c005180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009fd0 tx=0x7f2a9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a335b7062a31ae56 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:02.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.021+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac02ccb0 con 0x7f2a9c004d00 2024-03-16T19:29:02.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.021+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aac02ce10 con 0x7f2a9c004d00 2024-03-16T19:29:02.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.021+0000 7f2ab2ffd640 1 -- 172.21.15.172:0/596882486 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac034680 con 0x7f2a9c004d00 2024-03-16T19:29:02.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.022+0000 7f2aba2ae640 1 -- 172.21.15.172:0/596882486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 msgr2=0x7f2a9c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.022+0000 7f2aba2ae640 1 --2- 172.21.15.172:0/596882486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2a9c005180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009fd0 tx=0x7f2a9c0057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.022+0000 7f2ab884b640 1 -- 172.21.15.172:0/596882486 reap_dead start 2024-03-16T19:29:02.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.022+0000 7f2aba2ae640 1 -- 172.21.15.172:0/596882486 shutdown_connections 2024-03-16T19:29:02.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.022+0000 7f2aba2ae640 1 -- 172.21.15.172:0/596882486 >> 172.21.15.172:0/596882486 conn(0x7f2ab4066c20 msgr2=0x7f2ab4067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:02.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.022+0000 7f2aba2ae640 1 -- 172.21.15.172:0/596882486 shutdown_connections 2024-03-16T19:29:02.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.022+0000 7f2aba2ae640 1 -- 172.21.15.172:0/596882486 wait complete. 2024-03-16T19:29:02.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.023+0000 7f2aba2ae640 1 Processor -- start 2024-03-16T19:29:02.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.023+0000 7f2aba2ae640 1 -- start start 2024-03-16T19:29:02.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2aba2ae640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2ab40c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2aba2ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ab415bc40 0x7f2ab40b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2aba2ae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40b23b0 0x7f2ab40b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2aba2ae640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40b2d70 con 0x7f2a9c004d00 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2aba2ae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40ce0c0 con 0x7f2ab415bc40 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2aba2ae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40cdde0 con 0x7f2ab40b23b0 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2ab3fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2ab40c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2ab3fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2ab40c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42834/0 (socket says 172.21.15.172:42834) 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2ab3fff640 1 -- 172.21.15.172:0/1268194728 learned_addr learned my addr 172.21.15.172:0/1268194728 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2ab37fe640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ab415bc40 0x7f2ab40b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2ab884b640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40b23b0 0x7f2ab40b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.024+0000 7f2ab3fff640 1 -- 172.21.15.172:0/1268194728 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40b23b0 msgr2=0x7f2ab40b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab3fff640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40b23b0 0x7f2ab40b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab3fff640 1 -- 172.21.15.172:0/1268194728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ab415bc40 msgr2=0x7f2ab40b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab3fff640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ab415bc40 0x7f2ab40b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab3fff640 1 -- 172.21.15.172:0/1268194728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab415c7e0 con 0x7f2a9c004d00 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab37fe640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ab415bc40 0x7f2ab40b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab3fff640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2ab40c9450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2aa40108c0 tx=0x7f2aa4010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab0ff9640 1 -- 172.21.15.172:0/1268194728 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa400e9c0 con 0x7f2a9c004d00 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab0ff9640 1 -- 172.21.15.172:0/1268194728 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa400eb20 con 0x7f2a9c004d00 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab0ff9640 1 -- 172.21.15.172:0/1268194728 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa400eeb0 con 0x7f2a9c004d00 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2aba2ae640 1 -- 172.21.15.172:0/1268194728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab415cac0 con 0x7f2a9c004d00 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2aba2ae640 1 -- 172.21.15.172:0/1268194728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab415d080 con 0x7f2a9c004d00 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab0ff9640 1 -- 172.21.15.172:0/1268194728 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2aa4016810 con 0x7f2a9c004d00 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.025+0000 7f2ab0ff9640 1 -- 172.21.15.172:0/1268194728 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2aa4014c40 con 0x7f2a9c004d00 2024-03-16T19:29:02.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.026+0000 7f2aba2ae640 1 -- 172.21.15.172:0/1268194728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab415d240 con 0x7f2a9c004d00 2024-03-16T19:29:02.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.028+0000 7f2ab0ff9640 1 -- 172.21.15.172:0/1268194728 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2aa403edd0 con 0x7f2a9c004d00 2024-03-16T19:29:02.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.094+0000 7f2aba2ae640 1 -- 172.21.15.172:0/1268194728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2ab40aff50 con 0x7f2a9c004d00 2024-03-16T19:29:02.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.096+0000 7f2ab0ff9640 1 -- 172.21.15.172:0/1268194728 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2aa402d020 con 0x7f2a9c004d00 2024-03-16T19:29:02.078 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2024-03-16T19:29:02.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.096+0000 7f2aba2ae640 1 -- 172.21.15.172:0/1268194728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 msgr2=0x7f2ab40c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.096+0000 7f2aba2ae640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2ab40c9450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2aa40108c0 tx=0x7f2aa4010d80 comp rx=0 tx=0).stop 2024-03-16T19:29:02.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.097+0000 7f2aba2ae640 1 -- 172.21.15.172:0/1268194728 shutdown_connections 2024-03-16T19:29:02.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.097+0000 7f2aba2ae640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ab40b23b0 0x7f2ab40b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.097+0000 7f2aba2ae640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ab415bc40 0x7f2ab40b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.097+0000 7f2aba2ae640 1 --2- 172.21.15.172:0/1268194728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a9c004d00 0x7f2ab40c9450 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.097+0000 7f2aba2ae640 1 -- 172.21.15.172:0/1268194728 >> 172.21.15.172:0/1268194728 conn(0x7f2ab4066c20 msgr2=0x7f2ab40cae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:02.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.097+0000 7f2aba2ae640 1 -- 172.21.15.172:0/1268194728 shutdown_connections 2024-03-16T19:29:02.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:02.097+0000 7f2aba2ae640 1 -- 172.21.15.172:0/1268194728 wait complete. 2024-03-16T19:29:02.090 INFO:tasks.ceph:check_enable_crimson: False 2024-03-16T19:29:02.090 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-16T19:29:02.090 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:29:02.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-16T19:29:02.117 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:29:02.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-16T19:29:02.184 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:29:02.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-16T19:29:02.250 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:29:02.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-16T19:29:02.317 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:29:02.318 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-16T19:29:02.345 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:29:02.345 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-16T19:29:02.413 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:29:02.413 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-16T19:29:02.465 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:02.483+0000 7fbad1848040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T19:29:02.479 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-16T19:29:02.479 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-16T19:29:02.514 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:02.532+0000 7f21ef7b3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T19:29:02.521 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:02.539+0000 7fbad1848040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T19:29:02.546 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 460ed9e4-eacf-4544-9199-23ca7208b663 0 2024-03-16T19:29:02.567 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:02.586+0000 7f21ef7b3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T19:29:02.685 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.703+0000 7ff4a833e640 1 Processor -- start 2024-03-16T19:29:02.685 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.703+0000 7ff4a833e640 1 -- start start 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.704+0000 7ff4a833e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff4a00c7290 0x7ff4a00c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.704+0000 7ff4a833e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a0072ed0 con 0x7ff4a0152850 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.704+0000 7ff4a833e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a00c6dc0 con 0x7ff4a00c7ce0 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.704+0000 7ff4a833e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a00ca1f0 con 0x7ff4a00c7290 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.704+0000 7ff4a68db640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff4a00c7290 0x7ff4a00c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.704+0000 7ff4a68db640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff4a00c7290 0x7ff4a00c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:57758/0 (socket says 172.21.15.174:57758) 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.704+0000 7ff4a58d9640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7ff4a00c7ce0 0x7ff4a00c8350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:58474/0 (socket says 172.21.15.174:58474) 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.704+0000 7ff4a68db640 1 -- 172.21.15.174:0/2222660603 learned_addr learned my addr 172.21.15.174:0/2222660603 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.705+0000 7ff4a68db640 1 -- 172.21.15.174:0/2222660603 >> v1:172.21.15.174:6789/0 conn(0x7ff4a00c7ce0 legacy=0x7ff4a00c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.705+0000 7ff4a68db640 1 -- 172.21.15.174:0/2222660603 >> v1:172.21.15.172:6789/0 conn(0x7ff4a0152850 legacy=0x7ff4a0066c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.705+0000 7ff4a68db640 1 -- 172.21.15.174:0/2222660603 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a0071450 con 0x7ff4a00c7290 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.705+0000 7ff4a68db640 1 --2- 172.21.15.174:0/2222660603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 0x7ff4a00c8a60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff498009e50 tx=0x7ff49802c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96c38b0c24afbba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a50d8640 1 -- 172.21.15.174:0/2222660603 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49802cbb0 con 0x7ff4a00c7290 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a50d8640 1 -- 172.21.15.174:0/2222660603 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49802cd10 con 0x7ff4a00c7290 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a50d8640 1 -- 172.21.15.174:0/2222660603 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498034610 con 0x7ff4a00c7290 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a833e640 1 -- 172.21.15.174:0/2222660603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 msgr2=0x7ff4a00c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a833e640 1 --2- 172.21.15.174:0/2222660603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 0x7ff4a00c8a60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff498009e50 tx=0x7ff49802c010 comp rx=0 tx=0).stop 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a833e640 1 -- 172.21.15.174:0/2222660603 shutdown_connections 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a833e640 1 --2- 172.21.15.174:0/2222660603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 0x7ff4a00c8a60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a833e640 1 -- 172.21.15.174:0/2222660603 >> 172.21.15.174:0/2222660603 conn(0x7ff4a00c9390 msgr2=0x7ff4a00c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a833e640 1 -- 172.21.15.174:0/2222660603 shutdown_connections 2024-03-16T19:29:02.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.706+0000 7ff4a833e640 1 -- 172.21.15.174:0/2222660603 wait complete. 2024-03-16T19:29:02.690 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.707+0000 7ff4a833e640 1 Processor -- start 2024-03-16T19:29:02.690 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.707+0000 7ff4a833e640 1 -- start start 2024-03-16T19:29:02.692 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a833e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 0x7ff4a0156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.692 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a833e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4a00c7ce0 0x7ff4a0157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.692 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a833e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4a0152850 0x7ff4a01413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.692 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a833e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01514b0 con 0x7ff4a00c7ce0 2024-03-16T19:29:02.692 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a833e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0157990 con 0x7ff4a00c7290 2024-03-16T19:29:02.692 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a833e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0145730 con 0x7ff4a0152850 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a60da640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 0x7ff4a0156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a60da640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 0x7ff4a0156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:57772/0 (socket says 172.21.15.174:57772) 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a60da640 1 -- 172.21.15.174:0/3802730934 learned_addr learned my addr 172.21.15.174:0/3802730934 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a68db640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4a0152850 0x7ff4a01413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.708+0000 7ff4a58d9640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4a00c7ce0 0x7ff4a0157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff4a68db640 1 -- 172.21.15.174:0/3802730934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 msgr2=0x7ff4a0156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff4a68db640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 0x7ff4a0156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff4a68db640 1 -- 172.21.15.174:0/3802730934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4a00c7ce0 msgr2=0x7ff4a0157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff4a68db640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4a00c7ce0 0x7ff4a0157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff4a68db640 1 -- 172.21.15.174:0/3802730934 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff498009b30 con 0x7ff4a0152850 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff4a60da640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 0x7ff4a0156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff4a68db640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4a0152850 0x7ff4a01413d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff498032000 tx=0x7ff498004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff496ffd640 1 -- 172.21.15.174:0/3802730934 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498009510 con 0x7ff4a0152850 2024-03-16T19:29:02.693 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff496ffd640 1 -- 172.21.15.174:0/3802730934 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49803e450 con 0x7ff4a0152850 2024-03-16T19:29:02.694 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff496ffd640 1 -- 172.21.15.174:0/3802730934 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49803e800 con 0x7ff4a0152850 2024-03-16T19:29:02.694 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff4a833e640 1 -- 172.21.15.174:0/3802730934 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a0150f30 con 0x7ff4a0152850 2024-03-16T19:29:02.694 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff4a833e640 1 -- 172.21.15.174:0/3802730934 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a01511d0 con 0x7ff4a0152850 2024-03-16T19:29:02.694 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff496ffd640 1 -- 172.21.15.174:0/3802730934 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff49803e960 con 0x7ff4a0152850 2024-03-16T19:29:02.694 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.709+0000 7ff496ffd640 1 -- 172.21.15.174:0/3802730934 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff498048bc0 con 0x7ff4a0152850 2024-03-16T19:29:02.694 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.710+0000 7ff4a833e640 1 -- 172.21.15.174:0/3802730934 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a006bce0 con 0x7ff4a0152850 2024-03-16T19:29:02.694 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.712+0000 7ff496ffd640 1 -- 172.21.15.174:0/3802730934 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff498043020 con 0x7ff4a0152850 2024-03-16T19:29:02.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.783+0000 7ff4a833e640 1 -- 172.21.15.174:0/3802730934 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd new", "uuid": "460ed9e4-eacf-4544-9199-23ca7208b663", "id": 0} v 0) v1 -- 0x7ff4a00688d0 con 0x7ff4a0152850 2024-03-16T19:29:02.768 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.786+0000 7ff496ffd640 1 -- 172.21.15.174:0/3802730934 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "460ed9e4-eacf-4544-9199-23ca7208b663", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff4980570b0 con 0x7ff4a0152850 2024-03-16T19:29:02.768 INFO:teuthology.orchestra.run.smithi174.stdout:0 2024-03-16T19:29:02.769 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.787+0000 7ff4a833e640 1 -- 172.21.15.174:0/3802730934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4a0152850 msgr2=0x7ff4a01413d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.769 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.787+0000 7ff4a833e640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4a0152850 0x7ff4a01413d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff498032000 tx=0x7ff498004a40 comp rx=0 tx=0).stop 2024-03-16T19:29:02.769 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.787+0000 7ff4a833e640 1 -- 172.21.15.174:0/3802730934 shutdown_connections 2024-03-16T19:29:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.787+0000 7ff4a833e640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4a0152850 0x7ff4a01413d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.787+0000 7ff4a833e640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4a00c7ce0 0x7ff4a0157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.787+0000 7ff4a833e640 1 --2- 172.21.15.174:0/3802730934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4a00c7290 0x7ff4a0156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.787+0000 7ff4a833e640 1 -- 172.21.15.174:0/3802730934 >> 172.21.15.174:0/3802730934 conn(0x7ff4a00c9390 msgr2=0x7ff4a006f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.787+0000 7ff4a833e640 1 -- 172.21.15.174:0/3802730934 shutdown_connections 2024-03-16T19:29:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.787+0000 7ff4a833e640 1 -- 172.21.15.174:0/3802730934 wait complete. 2024-03-16T19:29:02.781 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new b84ebb50-079d-460f-bf05-71040f17500a 1 2024-03-16T19:29:02.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.899+0000 7f508426f640 1 Processor -- start 2024-03-16T19:29:02.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.899+0000 7f508426f640 1 -- start start 2024-03-16T19:29:02.882 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508426f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f507c15b700 0x7f507c0b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508426f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508426f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c06e990 con 0x7f507c06f640 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508426f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c15bae0 con 0x7f507c15b700 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508426f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c06bfe0 con 0x7f507c0ce320 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508200b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f507c15b700 0x7f507c0b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508280c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508180a640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f507c06f640 0x7f507c06eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:58480/0 (socket says 172.21.15.174:58480) 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508180a640 1 -- 172.21.15.174:0/3936008128 learned_addr learned my addr 172.21.15.174:0/3936008128 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508200b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f507c15b700 0x7f507c0b0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:57782/0 (socket says 172.21.15.174:57782) 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508280c640 1 -- 172.21.15.174:0/3936008128 >> v2:172.21.15.174:3300/0 conn(0x7f507c15b700 msgr2=0x7f507c0b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508280c640 1 --2- 172.21.15.174:0/3936008128 >> v2:172.21.15.174:3300/0 conn(0x7f507c15b700 0x7f507c0b0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508280c640 1 -- 172.21.15.174:0/3936008128 >> v1:172.21.15.174:6789/0 conn(0x7f507c06f640 legacy=0x7f507c06eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508280c640 1 -- 172.21.15.174:0/3936008128 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c06f260 con 0x7f507c0ce320 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f5081009640 1 -- 172.21.15.174:0/3936008128 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342596612 0 0) 0x7f5078001110 con 0x7f507c06f640 2024-03-16T19:29:02.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.900+0000 7f508200b640 1 --2- 172.21.15.174:0/3936008128 >> v2:172.21.15.174:3300/0 conn(0x7f507c15b700 0x7f507c0b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:02.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.901+0000 7f508280c640 1 --2- 172.21.15.174:0/3936008128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0ce750 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5074009d90 tx=0x7f507402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95e15ab62491bfe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:02.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.901+0000 7f5081009640 1 -- 172.21.15.174:0/3936008128 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507402cce0 con 0x7f507c0ce320 2024-03-16T19:29:02.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.901+0000 7f5081009640 1 -- 172.21.15.174:0/3936008128 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507402ce40 con 0x7f507c0ce320 2024-03-16T19:29:02.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.901+0000 7f5081009640 1 -- 172.21.15.174:0/3936008128 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074034660 con 0x7f507c0ce320 2024-03-16T19:29:02.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.902+0000 7f508426f640 1 -- 172.21.15.174:0/3936008128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 msgr2=0x7f507c0ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.902+0000 7f508426f640 1 --2- 172.21.15.174:0/3936008128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0ce750 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5074009d90 tx=0x7f507402bf30 comp rx=0 tx=0).stop 2024-03-16T19:29:02.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.902+0000 7f508426f640 1 -- 172.21.15.174:0/3936008128 shutdown_connections 2024-03-16T19:29:02.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.902+0000 7f508426f640 1 --2- 172.21.15.174:0/3936008128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0ce750 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.902+0000 7f508426f640 1 --2- 172.21.15.174:0/3936008128 >> v2:172.21.15.174:3300/0 conn(0x7f507c15b700 0x7f507c0b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.902+0000 7f508426f640 1 -- 172.21.15.174:0/3936008128 >> 172.21.15.174:0/3936008128 conn(0x7f507c066c20 msgr2=0x7f507c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:02.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.902+0000 7f508426f640 1 -- 172.21.15.174:0/3936008128 shutdown_connections 2024-03-16T19:29:02.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.902+0000 7f508426f640 1 -- 172.21.15.174:0/3936008128 wait complete. 2024-03-16T19:29:02.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.903+0000 7f508426f640 1 Processor -- start 2024-03-16T19:29:02.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.903+0000 7f508426f640 1 -- start start 2024-03-16T19:29:02.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508426f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f507c06f640 0x7f507c1f1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.888 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508426f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508426f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f507c15b700 0x7f507c1537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508426f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0b1a60 con 0x7f507c06f640 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508426f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c1f0900 con 0x7f507c15b700 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508426f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c155ce0 con 0x7f507c0ce320 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508180a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508280c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f507c15b700 0x7f507c1537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508280c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f507c15b700 0x7f507c1537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:57798/0 (socket says 172.21.15.174:57798) 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508280c640 1 -- 172.21.15.174:0/3559935474 learned_addr learned my addr 172.21.15.174:0/3559935474 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508200b640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f507c06f640 0x7f507c1f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508180a640 1 -- 172.21.15.174:0/3559935474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f507c15b700 msgr2=0x7f507c1537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508180a640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f507c15b700 0x7f507c1537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508180a640 1 -- 172.21.15.174:0/3559935474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f507c06f640 msgr2=0x7f507c1f1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508180a640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f507c06f640 0x7f507c1f1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.889 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508180a640 1 -- 172.21.15.174:0/3559935474 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c009400 con 0x7f507c0ce320 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508280c640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f507c15b700 0x7f507c1537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508200b640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f507c06f640 0x7f507c1f1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508180a640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0b2f80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5078005880 tx=0x7f507800c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f5072ffd640 1 -- 172.21.15.174:0/3559935474 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5078005ce0 con 0x7f507c0ce320 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f5072ffd640 1 -- 172.21.15.174:0/3559935474 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5078005e60 con 0x7f507c0ce320 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f5072ffd640 1 -- 172.21.15.174:0/3559935474 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5078007310 con 0x7f507c0ce320 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508426f640 1 -- 172.21.15.174:0/3559935474 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5074009a70 con 0x7f507c0ce320 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.904+0000 7f508426f640 1 -- 172.21.15.174:0/3559935474 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c0b1f20 con 0x7f507c0ce320 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.905+0000 7f5072ffd640 1 -- 172.21.15.174:0/3559935474 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5078007470 con 0x7f507c0ce320 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.905+0000 7f5072ffd640 1 -- 172.21.15.174:0/3559935474 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f50780128d0 con 0x7f507c0ce320 2024-03-16T19:29:02.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.905+0000 7f508426f640 1 -- 172.21.15.174:0/3559935474 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c15aa90 con 0x7f507c0ce320 2024-03-16T19:29:02.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.909+0000 7f5072ffd640 1 -- 172.21.15.174:0/3559935474 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5078017070 con 0x7f507c0ce320 2024-03-16T19:29:02.963 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.980+0000 7f508426f640 1 -- 172.21.15.174:0/3559935474 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b84ebb50-079d-460f-bf05-71040f17500a", "id": 1} v 0) v1 -- 0x7f507c0688d0 con 0x7f507c0ce320 2024-03-16T19:29:02.966 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.984+0000 7f5072ffd640 1 -- 172.21.15.174:0/3559935474 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b84ebb50-079d-460f-bf05-71040f17500a", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f507c06e990 con 0x7f507c0ce320 2024-03-16T19:29:02.966 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-16T19:29:02.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.985+0000 7f508426f640 1 -- 172.21.15.174:0/3559935474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 msgr2=0x7f507c0b2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:02.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.985+0000 7f508426f640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0b2f80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5078005880 tx=0x7f507800c870 comp rx=0 tx=0).stop 2024-03-16T19:29:02.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.985+0000 7f508426f640 1 -- 172.21.15.174:0/3559935474 shutdown_connections 2024-03-16T19:29:02.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.985+0000 7f508426f640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f507c15b700 0x7f507c1537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.985+0000 7f508426f640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f507c0ce320 0x7f507c0b2f80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.985+0000 7f508426f640 1 --2- 172.21.15.174:0/3559935474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f507c06f640 0x7f507c1f1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:02.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.985+0000 7f508426f640 1 -- 172.21.15.174:0/3559935474 >> 172.21.15.174:0/3559935474 conn(0x7f507c066c20 msgr2=0x7f507c0ae8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:02.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.985+0000 7f508426f640 1 -- 172.21.15.174:0/3559935474 shutdown_connections 2024-03-16T19:29:02.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:02.985+0000 7f508426f640 1 -- 172.21.15.174:0/3559935474 wait complete. 2024-03-16T19:29:02.977 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 6774c42c-5373-4543-8f8c-ed7b82c12057 2 2024-03-16T19:29:03.073 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.091+0000 7fbad1848040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T19:29:03.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.096+0000 7fe6f67ac640 1 Processor -- start 2024-03-16T19:29:03.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.096+0000 7fe6f67ac640 1 -- start start 2024-03-16T19:29:03.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.097+0000 7fe6f67ac640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe6f0072850 0x7fe6f015b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.097+0000 7fe6f67ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.097+0000 7fe6f67ac640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f00734b0 con 0x7fe6f0072c30 2024-03-16T19:29:03.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.097+0000 7fe6f67ac640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f00c7ae0 con 0x7fe6f0072850 2024-03-16T19:29:03.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.097+0000 7fe6f67ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00c7d80 con 0x7fe6f006cbe0 2024-03-16T19:29:03.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.097+0000 7fe6f4d49640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.097+0000 7fe6f4d49640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:45492/0 (socket says 172.21.15.174:45492) 2024-03-16T19:29:03.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.097+0000 7fe6f4d49640 1 -- 172.21.15.174:0/106671733 learned_addr learned my addr 172.21.15.174:0/106671733 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:03.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.097+0000 7fe6effff640 1 --2- 172.21.15.174:0/106671733 >> v2:172.21.15.172:3300/0 conn(0x7fe6f0072850 0x7fe6f015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.098+0000 7fe6f4d49640 1 -- 172.21.15.174:0/106671733 >> v2:172.21.15.172:3300/0 conn(0x7fe6f0072850 msgr2=0x7fe6f015b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:03.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.098+0000 7fe6f4d49640 1 --2- 172.21.15.174:0/106671733 >> v2:172.21.15.172:3300/0 conn(0x7fe6f0072850 0x7fe6f015b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.098+0000 7fe6f4d49640 1 -- 172.21.15.174:0/106671733 >> v1:172.21.15.172:6789/0 conn(0x7fe6f0072c30 legacy=0x7fe6f006aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.098+0000 7fe6f4d49640 1 -- 172.21.15.174:0/106671733 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f00c8060 con 0x7fe6f006cbe0 2024-03-16T19:29:03.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.098+0000 7fe6effff640 1 --2- 172.21.15.174:0/106671733 >> v2:172.21.15.172:3300/0 conn(0x7fe6f0072850 0x7fe6f015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:03.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.098+0000 7fe6f4d49640 1 --2- 172.21.15.174:0/106671733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c1ff0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009e50 tx=0x7fe6e802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64596e264538f829 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.098+0000 7fe6eeffd640 1 -- 172.21.15.174:0/106671733 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e802cb00 con 0x7fe6f006cbe0 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.098+0000 7fe6eeffd640 1 -- 172.21.15.174:0/106671733 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e802cc60 con 0x7fe6f006cbe0 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.098+0000 7fe6eeffd640 1 -- 172.21.15.174:0/106671733 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e80343f0 con 0x7fe6f006cbe0 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.099+0000 7fe6f67ac640 1 -- 172.21.15.174:0/106671733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 msgr2=0x7fe6f00c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.099+0000 7fe6f67ac640 1 --2- 172.21.15.174:0/106671733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c1ff0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009e50 tx=0x7fe6e802bf60 comp rx=0 tx=0).stop 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.099+0000 7fe6f67ac640 1 -- 172.21.15.174:0/106671733 shutdown_connections 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.099+0000 7fe6f67ac640 1 --2- 172.21.15.174:0/106671733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c1ff0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.099+0000 7fe6f67ac640 1 --2- 172.21.15.174:0/106671733 >> v2:172.21.15.172:3300/0 conn(0x7fe6f0072850 0x7fe6f015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.099+0000 7fe6f67ac640 1 -- 172.21.15.174:0/106671733 >> 172.21.15.174:0/106671733 conn(0x7fe6f0066c20 msgr2=0x7fe6f0067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.099+0000 7fe6f67ac640 1 -- 172.21.15.174:0/106671733 shutdown_connections 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.099+0000 7fe6f67ac640 1 -- 172.21.15.174:0/106671733 wait complete. 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.100+0000 7fe6f67ac640 1 Processor -- start 2024-03-16T19:29:03.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.100+0000 7fe6f67ac640 1 -- start start 2024-03-16T19:29:03.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6f67ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6f67ac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6f0072850 0x7fe6f014d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6f67ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6f0072c30 0x7fe6f01561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6f67ac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f0157820 con 0x7fe6f0072850 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6f67ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00c8b70 con 0x7fe6f0072c30 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6f67ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00c9780 con 0x7fe6f006cbe0 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6effff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6effff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:45496/0 (socket says 172.21.15.174:45496) 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6effff640 1 -- 172.21.15.174:0/3803210430 learned_addr learned my addr 172.21.15.174:0/3803210430 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6f4d49640 1 --2- 172.21.15.174:0/3803210430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6f0072c30 0x7fe6f01561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6ef7fe640 1 --2- 172.21.15.174:0/3803210430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6f0072850 0x7fe6f014d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6effff640 1 -- 172.21.15.174:0/3803210430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6f0072c30 msgr2=0x7fe6f01561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6effff640 1 --2- 172.21.15.174:0/3803210430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6f0072c30 0x7fe6f01561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6effff640 1 -- 172.21.15.174:0/3803210430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6f0072850 msgr2=0x7fe6f014d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6effff640 1 --2- 172.21.15.174:0/3803210430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6f0072850 0x7fe6f014d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.101+0000 7fe6effff640 1 -- 172.21.15.174:0/3803210430 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e8009b30 con 0x7fe6f006cbe0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.102+0000 7fe6effff640 1 --2- 172.21.15.174:0/3803210430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c8630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400e810 tx=0x7fe6e400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.102+0000 7fe6ecff9640 1 -- 172.21.15.174:0/3803210430 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4004030 con 0x7fe6f006cbe0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.102+0000 7fe6ecff9640 1 -- 172.21.15.174:0/3803210430 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e4004190 con 0x7fe6f006cbe0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.102+0000 7fe6ecff9640 1 -- 172.21.15.174:0/3803210430 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4004520 con 0x7fe6f006cbe0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.102+0000 7fe6f67ac640 1 -- 172.21.15.174:0/3803210430 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f00c99d0 con 0x7fe6f006cbe0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.102+0000 7fe6ecff9640 1 -- 172.21.15.174:0/3803210430 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe6e400ca60 con 0x7fe6f006cbe0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.102+0000 7fe6f67ac640 1 -- 172.21.15.174:0/3803210430 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f00c9f90 con 0x7fe6f006cbe0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.102+0000 7fe6ecff9640 1 -- 172.21.15.174:0/3803210430 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe6e4003a30 con 0x7fe6f006cbe0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.103+0000 7fe6f67ac640 1 -- 172.21.15.174:0/3803210430 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f00688d0 con 0x7fe6f006cbe0 2024-03-16T19:29:03.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.105+0000 7fe6ecff9640 1 -- 172.21.15.174:0/3803210430 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe6e403add0 con 0x7fe6f006cbe0 2024-03-16T19:29:03.088 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.107+0000 7f21ef7b3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T19:29:03.130 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.148+0000 7fbad1848040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T19:29:03.142 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.160+0000 7f21ef7b3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T19:29:03.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.176+0000 7fe6f67ac640 1 -- 172.21.15.174:0/3803210430 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6774c42c-5373-4543-8f8c-ed7b82c12057", "id": 2} v 0) v1 -- 0x7fe6f006b8a0 con 0x7fe6f006cbe0 2024-03-16T19:29:03.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.182+0000 7fe6ecff9640 1 -- 172.21.15.174:0/3803210430 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6774c42c-5373-4543-8f8c-ed7b82c12057", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe6e4029070 con 0x7fe6f006cbe0 2024-03-16T19:29:03.164 INFO:teuthology.orchestra.run.smithi174.stdout:2 2024-03-16T19:29:03.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.182+0000 7fe6f67ac640 1 -- 172.21.15.174:0/3803210430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 msgr2=0x7fe6f00c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.182+0000 7fe6f67ac640 1 --2- 172.21.15.174:0/3803210430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c8630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400e810 tx=0x7fe6e400ecd0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.183+0000 7fe6f67ac640 1 -- 172.21.15.174:0/3803210430 shutdown_connections 2024-03-16T19:29:03.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.183+0000 7fe6f67ac640 1 --2- 172.21.15.174:0/3803210430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6f0072c30 0x7fe6f01561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.183+0000 7fe6f67ac640 1 --2- 172.21.15.174:0/3803210430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6f0072850 0x7fe6f014d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.183+0000 7fe6f67ac640 1 --2- 172.21.15.174:0/3803210430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6f006cbe0 0x7fe6f00c8630 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.183+0000 7fe6f67ac640 1 -- 172.21.15.174:0/3803210430 >> 172.21.15.174:0/3803210430 conn(0x7fe6f0066c20 msgr2=0x7fe6f0067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.183+0000 7fe6f67ac640 1 -- 172.21.15.174:0/3803210430 shutdown_connections 2024-03-16T19:29:03.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.183+0000 7fe6f67ac640 1 -- 172.21.15.174:0/3803210430 wait complete. 2024-03-16T19:29:03.175 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new b96838ef-d991-43ab-b2d7-16deab58ca78 3 2024-03-16T19:29:03.177 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.195+0000 7fbad1848040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T19:29:03.187 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.205+0000 7f21ef7b3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T19:29:03.234 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.252+0000 7fbad1848040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T19:29:03.240 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.258+0000 7f21ef7b3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T19:29:03.275 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.293+0000 7f2937348640 1 Processor -- start 2024-03-16T19:29:03.275 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.293+0000 7f2937348640 1 -- start start 2024-03-16T19:29:03.276 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f2937348640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29300c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f2937348640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293006b2b0 con 0x7f293006cc50 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f2937348640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293006b590 con 0x7f29300cd290 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f2937348640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293015b470 con 0x7f2930154c90 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f29350e4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29300c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f29350e4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29300c79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:45508/0 (socket says 172.21.15.174:45508) 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f29350e4640 1 -- 172.21.15.174:0/208348408 learned_addr learned my addr 172.21.15.174:0/208348408 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f29358e5640 1 --1- 172.21.15.174:0/208348408 >> v1:172.21.15.174:6789/0 conn(0x7f29300cd290 0x7f2930165300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:58482/0 (socket says 172.21.15.174:58482) 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f2923fff640 1 -- 172.21.15.174:0/208348408 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516971386 0 0) 0x7f293006b590 con 0x7f29300cd290 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f29350e4640 1 -- 172.21.15.174:0/208348408 >> v1:172.21.15.174:6789/0 conn(0x7f29300cd290 legacy=0x7f2930165300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.277 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f29350e4640 1 -- 172.21.15.174:0/208348408 >> v1:172.21.15.172:6789/0 conn(0x7f293006cc50 legacy=0x7f29300c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.278 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.294+0000 7f29350e4640 1 -- 172.21.15.174:0/208348408 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293015adb0 con 0x7f2930154c90 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.295+0000 7f29350e4640 1 --2- 172.21.15.174:0/208348408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29300c79d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f29240098a0 tx=0x7f292402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afc4f82919ef7856 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.295+0000 7f2923fff640 1 -- 172.21.15.174:0/208348408 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292402f830 con 0x7f2930154c90 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.295+0000 7f2923fff640 1 -- 172.21.15.174:0/208348408 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292402f990 con 0x7f2930154c90 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.295+0000 7f2923fff640 1 -- 172.21.15.174:0/208348408 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292402fd20 con 0x7f2930154c90 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.296+0000 7f2937348640 1 -- 172.21.15.174:0/208348408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 msgr2=0x7f29300c79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.296+0000 7f2937348640 1 --2- 172.21.15.174:0/208348408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29300c79d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f29240098a0 tx=0x7f292402ba10 comp rx=0 tx=0).stop 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.296+0000 7f2937348640 1 -- 172.21.15.174:0/208348408 shutdown_connections 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.296+0000 7f2937348640 1 --2- 172.21.15.174:0/208348408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29300c79d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.296+0000 7f2937348640 1 -- 172.21.15.174:0/208348408 >> 172.21.15.174:0/208348408 conn(0x7f2930066c20 msgr2=0x7f2930067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.279 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.296+0000 7f2937348640 1 -- 172.21.15.174:0/208348408 shutdown_connections 2024-03-16T19:29:03.280 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.296+0000 7f2937348640 1 -- 172.21.15.174:0/208348408 wait complete. 2024-03-16T19:29:03.280 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.297+0000 7fbad1848040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T19:29:03.282 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.297+0000 7f2937348640 1 Processor -- start 2024-03-16T19:29:03.282 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.297+0000 7f2937348640 1 -- start start 2024-03-16T19:29:03.282 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f2937348640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f293006cc50 0x7f2930151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.282 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f2937348640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29300cd290 0x7f29301f0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.282 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f2937348640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29301f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.282 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f2937348640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29300c7010 con 0x7f29300cd290 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f2937348640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29300c72b0 con 0x7f293006cc50 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f2937348640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29300c9510 con 0x7f2930154c90 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29350e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f293006cc50 0x7f2930151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29350e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f293006cc50 0x7f2930151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:57820/0 (socket says 172.21.15.174:57820) 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29350e4640 1 -- 172.21.15.174:0/333320256 learned_addr learned my addr 172.21.15.174:0/333320256 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29358e5640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29301f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29348e3640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29300cd290 0x7f29301f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29350e4640 1 -- 172.21.15.174:0/333320256 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 msgr2=0x7f29301f0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29350e4640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29301f0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29350e4640 1 -- 172.21.15.174:0/333320256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29300cd290 msgr2=0x7f29301f0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29350e4640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29300cd290 0x7f29301f0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29350e4640 1 -- 172.21.15.174:0/333320256 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29240094d0 con 0x7f293006cc50 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29358e5640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29301f0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f29350e4640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f293006cc50 0x7f2930151450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f29240098a0 tx=0x7f2924004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f2921ffb640 1 -- 172.21.15.174:0/333320256 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292403c040 con 0x7f293006cc50 2024-03-16T19:29:03.283 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f2921ffb640 1 -- 172.21.15.174:0/333320256 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2924037cd0 con 0x7f293006cc50 2024-03-16T19:29:03.284 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.298+0000 7f2937348640 1 -- 172.21.15.174:0/333320256 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29300c97f0 con 0x7f293006cc50 2024-03-16T19:29:03.284 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.299+0000 7f2921ffb640 1 -- 172.21.15.174:0/333320256 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2924004270 con 0x7f293006cc50 2024-03-16T19:29:03.284 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.299+0000 7f2937348640 1 -- 172.21.15.174:0/333320256 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29300c9db0 con 0x7f293006cc50 2024-03-16T19:29:03.284 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.299+0000 7f2921ffb640 1 -- 172.21.15.174:0/333320256 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f29240043d0 con 0x7f293006cc50 2024-03-16T19:29:03.284 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.299+0000 7f2921ffb640 1 -- 172.21.15.174:0/333320256 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f29240428c0 con 0x7f293006cc50 2024-03-16T19:29:03.284 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.300+0000 7f2937348640 1 -- 172.21.15.174:0/333320256 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29301f0390 con 0x7f293006cc50 2024-03-16T19:29:03.284 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.300+0000 7f21ef7b3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T19:29:03.284 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.302+0000 7f2921ffb640 1 -- 172.21.15.174:0/333320256 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2924047070 con 0x7f293006cc50 2024-03-16T19:29:03.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.374+0000 7f2937348640 1 -- 172.21.15.174:0/333320256 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b96838ef-d991-43ab-b2d7-16deab58ca78", "id": 3} v 0) v1 -- 0x7f29300688d0 con 0x7f293006cc50 2024-03-16T19:29:03.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.379+0000 7f2921ffb640 1 -- 172.21.15.174:0/333320256 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b96838ef-d991-43ab-b2d7-16deab58ca78", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2924042500 con 0x7f293006cc50 2024-03-16T19:29:03.362 INFO:teuthology.orchestra.run.smithi174.stdout:3 2024-03-16T19:29:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.380+0000 7f2937348640 1 -- 172.21.15.174:0/333320256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f293006cc50 msgr2=0x7f2930151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.380+0000 7f2937348640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f293006cc50 0x7f2930151450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f29240098a0 tx=0x7f2924004730 comp rx=0 tx=0).stop 2024-03-16T19:29:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.380+0000 7f2937348640 1 -- 172.21.15.174:0/333320256 shutdown_connections 2024-03-16T19:29:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.380+0000 7f2937348640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2930154c90 0x7f29301f0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.380+0000 7f2937348640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29300cd290 0x7f29301f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.380+0000 7f2937348640 1 --2- 172.21.15.174:0/333320256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f293006cc50 0x7f2930151450 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.380+0000 7f2937348640 1 -- 172.21.15.174:0/333320256 >> 172.21.15.174:0/333320256 conn(0x7f2930066c20 msgr2=0x7f2930153af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.380+0000 7f2937348640 1 -- 172.21.15.174:0/333320256 shutdown_connections 2024-03-16T19:29:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.380+0000 7f2937348640 1 -- 172.21.15.174:0/333320256 wait complete. 2024-03-16T19:29:03.373 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new f5c78559-d22e-4847-be3f-2f597d7f6789 4 2024-03-16T19:29:03.397 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.415+0000 7f21ef7b3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T19:29:03.400 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.418+0000 7fbad1848040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T19:29:03.466 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.484+0000 7f21ef7b3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T19:29:03.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.490+0000 7ff5326c3640 1 Processor -- start 2024-03-16T19:29:03.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.490+0000 7ff5326c3640 1 -- start start 2024-03-16T19:29:03.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.491+0000 7ff5326c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0c9380 0x7ff52c0ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.491+0000 7ff5326c3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff52c15b5f0 con 0x7ff52c06eb10 2024-03-16T19:29:03.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.491+0000 7ff5326c3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff52c15b8d0 con 0x7ff52c0ce340 2024-03-16T19:29:03.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.491+0000 7ff5326c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c06cd50 con 0x7ff52c0c9380 2024-03-16T19:29:03.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.491+0000 7ff52b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0c9380 0x7ff52c0ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.491+0000 7ff52bfff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7ff52c0ce340 0x7ff52c06c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:58498/0 (socket says 172.21.15.174:58498) 2024-03-16T19:29:03.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.491+0000 7ff52bfff640 1 -- 172.21.15.174:0/1868497665 learned_addr learned my addr 172.21.15.174:0/1868497665 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-16T19:29:03.474 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.492+0000 7fbad1848040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T19:29:03.476 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.492+0000 7ff52b7fe640 1 -- 172.21.15.174:0/1868497665 >> v1:172.21.15.174:6789/0 conn(0x7ff52c0ce340 legacy=0x7ff52c06c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.476 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.492+0000 7ff52b7fe640 1 -- 172.21.15.174:0/1868497665 >> v1:172.21.15.172:6789/0 conn(0x7ff52c06eb10 legacy=0x7ff52c165410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.476 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.492+0000 7ff52b7fe640 1 -- 172.21.15.174:0/1868497665 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c06dbc0 con 0x7ff52c0c9380 2024-03-16T19:29:03.476 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.492+0000 7ff52affd640 1 -- 172.21.15.174:0/1868497665 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922882590 0 0) 0x7ff52c15b8d0 con 0x7ff52c0ce340 2024-03-16T19:29:03.476 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.492+0000 7ff52b7fe640 1 --2- 172.21.15.174:0/1868497665 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0c9380 0x7ff52c0ce720 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0097d0 tx=0x7ff51c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ede671ecb0ecba06 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.492+0000 7ff52affd640 1 -- 172.21.15.174:0/1868497665 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c031890 con 0x7ff52c0c9380 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.492+0000 7ff52affd640 1 -- 172.21.15.174:0/1868497665 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51c0319f0 con 0x7ff52c0c9380 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.492+0000 7ff52affd640 1 -- 172.21.15.174:0/1868497665 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c031d80 con 0x7ff52c0c9380 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.493+0000 7ff5326c3640 1 -- 172.21.15.174:0/1868497665 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0c9380 msgr2=0x7ff52c0ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.493+0000 7ff5326c3640 1 --2- 172.21.15.174:0/1868497665 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0c9380 0x7ff52c0ce720 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0097d0 tx=0x7ff51c02b930 comp rx=0 tx=0).stop 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.493+0000 7ff5326c3640 1 -- 172.21.15.174:0/1868497665 shutdown_connections 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.493+0000 7ff5326c3640 1 --2- 172.21.15.174:0/1868497665 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0c9380 0x7ff52c0ce720 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.493+0000 7ff5326c3640 1 -- 172.21.15.174:0/1868497665 >> 172.21.15.174:0/1868497665 conn(0x7ff52c066c20 msgr2=0x7ff52c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.493+0000 7ff5326c3640 1 -- 172.21.15.174:0/1868497665 shutdown_connections 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.493+0000 7ff5326c3640 1 -- 172.21.15.174:0/1868497665 wait complete. 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.494+0000 7ff5326c3640 1 Processor -- start 2024-03-16T19:29:03.477 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.494+0000 7ff5326c3640 1 -- start start 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff5326c3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff52c06eb10 0x7ff52c1f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff5326c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff52c0c9380 0x7ff52c1f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff5326c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0ce340 0x7ff52c0c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff5326c3640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1f0820 con 0x7ff52c06eb10 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff5326c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1f3f10 con 0x7ff52c0c9380 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff5326c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1f41d0 con 0x7ff52c0ce340 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff52b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff52c0c9380 0x7ff52c1f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff530c60640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0ce340 0x7ff52c0c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff530c60640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0ce340 0x7ff52c0c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:45534/0 (socket says 172.21.15.174:45534) 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff530c60640 1 -- 172.21.15.174:0/1298191990 learned_addr learned my addr 172.21.15.174:0/1298191990 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff52bfff640 1 --2- 172.21.15.174:0/1298191990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff52c06eb10 0x7ff52c1f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff530c60640 1 -- 172.21.15.174:0/1298191990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff52c0c9380 msgr2=0x7ff52c1f1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff530c60640 1 --2- 172.21.15.174:0/1298191990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff52c0c9380 0x7ff52c1f1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff530c60640 1 -- 172.21.15.174:0/1298191990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff52c06eb10 msgr2=0x7ff52c1f0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff530c60640 1 --2- 172.21.15.174:0/1298191990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff52c06eb10 0x7ff52c1f0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.495+0000 7ff530c60640 1 -- 172.21.15.174:0/1298191990 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c009400 con 0x7ff52c0ce340 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.496+0000 7ff530c60640 1 --2- 172.21.15.174:0/1298191990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0ce340 0x7ff52c0c7ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff52400c6b0 tx=0x7ff52400cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.496+0000 7ff528ff9640 1 -- 172.21.15.174:0/1298191990 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff524003b70 con 0x7ff52c0ce340 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.496+0000 7ff528ff9640 1 -- 172.21.15.174:0/1298191990 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff524003cf0 con 0x7ff52c0ce340 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.496+0000 7ff5326c3640 1 -- 172.21.15.174:0/1298191990 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52c1f44b0 con 0x7ff52c0ce340 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.496+0000 7ff528ff9640 1 -- 172.21.15.174:0/1298191990 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff524003ea0 con 0x7ff52c0ce340 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.496+0000 7ff5326c3640 1 -- 172.21.15.174:0/1298191990 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52c1f4a70 con 0x7ff52c0ce340 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.496+0000 7ff528ff9640 1 -- 172.21.15.174:0/1298191990 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff524005310 con 0x7ff52c0ce340 2024-03-16T19:29:03.481 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.496+0000 7ff528ff9640 1 -- 172.21.15.174:0/1298191990 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7ff52400ab50 con 0x7ff52c0ce340 2024-03-16T19:29:03.482 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.497+0000 7ff5326c3640 1 -- 172.21.15.174:0/1298191990 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52c1f0970 con 0x7ff52c0ce340 2024-03-16T19:29:03.482 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.499+0000 7ff528ff9640 1 -- 172.21.15.174:0/1298191990 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff52c1f0970 con 0x7ff52c0ce340 2024-03-16T19:29:03.512 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.530+0000 7f21ef7b3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T19:29:03.522 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.540+0000 7fbad1848040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T19:29:03.553 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.571+0000 7ff5326c3640 1 -- 172.21.15.174:0/1298191990 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f5c78559-d22e-4847-be3f-2f597d7f6789", "id": 4} v 0) v1 -- 0x7ff52c0688d0 con 0x7ff52c0ce340 2024-03-16T19:29:03.558 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.576+0000 7ff528ff9640 1 -- 172.21.15.174:0/1298191990 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f5c78559-d22e-4847-be3f-2f597d7f6789", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff524005690 con 0x7ff52c0ce340 2024-03-16T19:29:03.558 INFO:teuthology.orchestra.run.smithi174.stdout:4 2024-03-16T19:29:03.559 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.577+0000 7ff5326c3640 1 -- 172.21.15.174:0/1298191990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0ce340 msgr2=0x7ff52c0c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.560 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.577+0000 7ff5326c3640 1 --2- 172.21.15.174:0/1298191990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0ce340 0x7ff52c0c7ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff52400c6b0 tx=0x7ff52400cb70 comp rx=0 tx=0).stop 2024-03-16T19:29:03.560 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.577+0000 7ff5326c3640 1 -- 172.21.15.174:0/1298191990 shutdown_connections 2024-03-16T19:29:03.560 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.577+0000 7ff5326c3640 1 --2- 172.21.15.174:0/1298191990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff52c0ce340 0x7ff52c0c7ed0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.560 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.577+0000 7ff5326c3640 1 --2- 172.21.15.174:0/1298191990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff52c0c9380 0x7ff52c1f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.560 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.577+0000 7ff5326c3640 1 --2- 172.21.15.174:0/1298191990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff52c06eb10 0x7ff52c1f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.560 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.577+0000 7ff5326c3640 1 -- 172.21.15.174:0/1298191990 >> 172.21.15.174:0/1298191990 conn(0x7ff52c066c20 msgr2=0x7ff52c15ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.560 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.577+0000 7ff5326c3640 1 -- 172.21.15.174:0/1298191990 shutdown_connections 2024-03-16T19:29:03.560 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.577+0000 7ff5326c3640 1 -- 172.21.15.174:0/1298191990 wait complete. 2024-03-16T19:29:03.564 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.583+0000 7f21ef7b3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T19:29:03.570 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new e1f4797f-31c3-4292-a4a1-88fa0ab64f9a 5 2024-03-16T19:29:03.578 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.596+0000 7fbad1848040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T19:29:03.611 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:03.629+0000 7f21ef7b3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T19:29:03.629 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:03.647+0000 7fbad1848040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T19:29:03.679 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.697+0000 7f366e233640 1 Processor -- start 2024-03-16T19:29:03.679 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.697+0000 7f366e233640 1 -- start start 2024-03-16T19:29:03.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.697+0000 7f366e233640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3668152c80 0x7f366815c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.697+0000 7f366e233640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366806f720 con 0x7f36681528a0 2024-03-16T19:29:03.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.697+0000 7f366e233640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36680c4fb0 con 0x7f366806f2d0 2024-03-16T19:29:03.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.697+0000 7f366e233640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36680c5290 con 0x7f3668152c80 2024-03-16T19:29:03.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.698+0000 7f3666ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3668152c80 0x7f366815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.698+0000 7f3667fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f366806f2d0 0x7f36680ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:58506/0 (socket says 172.21.15.174:58506) 2024-03-16T19:29:03.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.698+0000 7f3667fff640 1 -- 172.21.15.174:0/729851080 learned_addr learned my addr 172.21.15.174:0/729851080 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-16T19:29:03.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.698+0000 7f3666ffd640 1 -- 172.21.15.174:0/729851080 >> v1:172.21.15.174:6789/0 conn(0x7f366806f2d0 legacy=0x7f36680ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.698+0000 7f3666ffd640 1 -- 172.21.15.174:0/729851080 >> v1:172.21.15.172:6789/0 conn(0x7f36681528a0 legacy=0x7f36680c33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.698+0000 7f36667fc640 1 -- 172.21.15.174:0/729851080 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962907907 0 0) 0x7f36680c4fb0 con 0x7f366806f2d0 2024-03-16T19:29:03.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.698+0000 7f3666ffd640 1 -- 172.21.15.174:0/729851080 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36680c5570 con 0x7f3668152c80 2024-03-16T19:29:03.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.699+0000 7f3666ffd640 1 --2- 172.21.15.174:0/729851080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 0x7f366815c860 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f365c0097a0 tx=0x7f365c02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f7ae0677a5e927 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.699+0000 7f36667fc640 1 -- 172.21.15.174:0/729851080 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c02f810 con 0x7f3668152c80 2024-03-16T19:29:03.682 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.699+0000 7f36667fc640 1 -- 172.21.15.174:0/729851080 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365c02f970 con 0x7f3668152c80 2024-03-16T19:29:03.682 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.699+0000 7f36667fc640 1 -- 172.21.15.174:0/729851080 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c02fd00 con 0x7f3668152c80 2024-03-16T19:29:03.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.700+0000 7f366e233640 1 -- 172.21.15.174:0/729851080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 msgr2=0x7f366815c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.700+0000 7f366e233640 1 --2- 172.21.15.174:0/729851080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 0x7f366815c860 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f365c0097a0 tx=0x7f365c02b9f0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.700+0000 7f366e233640 1 -- 172.21.15.174:0/729851080 shutdown_connections 2024-03-16T19:29:03.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.700+0000 7f366e233640 1 --2- 172.21.15.174:0/729851080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 0x7f366815c860 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.700+0000 7f366e233640 1 -- 172.21.15.174:0/729851080 >> 172.21.15.174:0/729851080 conn(0x7f3668066c20 msgr2=0x7f3668067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.700+0000 7f366e233640 1 -- 172.21.15.174:0/729851080 shutdown_connections 2024-03-16T19:29:03.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.700+0000 7f366e233640 1 -- 172.21.15.174:0/729851080 wait complete. 2024-03-16T19:29:03.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.701+0000 7f366e233640 1 Processor -- start 2024-03-16T19:29:03.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.701+0000 7f366e233640 1 -- start start 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f366e233640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f366806f2d0 0x7f36681f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f366e233640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36681528a0 0x7f366815a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f366e233640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 0x7f36680b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f366e233640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36681f19d0 con 0x7f36681528a0 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f366e233640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36681f13a0 con 0x7f3668152c80 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f366e233640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36681f2330 con 0x7f366806f2d0 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f36677fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f366806f2d0 0x7f36681f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f3667fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 0x7f36680b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f36677fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f366806f2d0 0x7f36681f1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:45546/0 (socket says 172.21.15.174:45546) 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f36677fe640 1 -- 172.21.15.174:0/3559121443 learned_addr learned my addr 172.21.15.174:0/3559121443 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f3667fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 0x7f36680b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:57854/0 (socket says 172.21.15.174:57854) 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f3666ffd640 1 --2- 172.21.15.174:0/3559121443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36681528a0 0x7f366815a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f3667fff640 1 -- 172.21.15.174:0/3559121443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f366806f2d0 msgr2=0x7f36681f1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f3667fff640 1 --2- 172.21.15.174:0/3559121443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f366806f2d0 0x7f36681f1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f3667fff640 1 -- 172.21.15.174:0/3559121443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36681528a0 msgr2=0x7f366815a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f3667fff640 1 --2- 172.21.15.174:0/3559121443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36681528a0 0x7f366815a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f3667fff640 1 -- 172.21.15.174:0/3559121443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c009400 con 0x7f3668152c80 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f36677fe640 1 --2- 172.21.15.174:0/3559121443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f366806f2d0 0x7f36681f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.702+0000 7f3667fff640 1 --2- 172.21.15.174:0/3559121443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 0x7f36680b2960 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f366000c700 tx=0x7f366000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.703+0000 7f364ffff640 1 -- 172.21.15.174:0/3559121443 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366000ce10 con 0x7f3668152c80 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.703+0000 7f364ffff640 1 -- 172.21.15.174:0/3559121443 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366000a640 con 0x7f3668152c80 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.703+0000 7f364ffff640 1 -- 172.21.15.174:0/3559121443 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366000a9f0 con 0x7f3668152c80 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.703+0000 7f366e233640 1 -- 172.21.15.174:0/3559121443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36680b1ca0 con 0x7f3668152c80 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.703+0000 7f366e233640 1 -- 172.21.15.174:0/3559121443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36681eebd0 con 0x7f3668152c80 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.703+0000 7f364ffff640 1 -- 172.21.15.174:0/3559121443 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f366000ab50 con 0x7f3668152c80 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.703+0000 7f364ffff640 1 -- 172.21.15.174:0/3559121443 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f3660012980 con 0x7f3668152c80 2024-03-16T19:29:03.687 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.704+0000 7f366e233640 1 -- 172.21.15.174:0/3559121443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36681f1b10 con 0x7f3668152c80 2024-03-16T19:29:03.688 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.706+0000 7f364ffff640 1 -- 172.21.15.174:0/3559121443 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f36681f1b10 con 0x7f3668152c80 2024-03-16T19:29:03.760 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.778+0000 7f366e233640 1 -- 172.21.15.174:0/3559121443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e1f4797f-31c3-4292-a4a1-88fa0ab64f9a", "id": 5} v 0) v1 -- 0x7f366815c020 con 0x7f3668152c80 2024-03-16T19:29:03.765 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.783+0000 7f364ffff640 1 -- 172.21.15.174:0/3559121443 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e1f4797f-31c3-4292-a4a1-88fa0ab64f9a", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3660018070 con 0x7f3668152c80 2024-03-16T19:29:03.765 INFO:teuthology.orchestra.run.smithi174.stdout:5 2024-03-16T19:29:03.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.784+0000 7f366e233640 1 -- 172.21.15.174:0/3559121443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 msgr2=0x7f36680b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.784+0000 7f366e233640 1 --2- 172.21.15.174:0/3559121443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 0x7f36680b2960 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f366000c700 tx=0x7f366000cbc0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.766 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.784+0000 7f366e233640 1 -- 172.21.15.174:0/3559121443 shutdown_connections 2024-03-16T19:29:03.767 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.784+0000 7f366e233640 1 --2- 172.21.15.174:0/3559121443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3668152c80 0x7f36680b2960 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.767 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.784+0000 7f366e233640 1 --2- 172.21.15.174:0/3559121443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36681528a0 0x7f366815a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.767 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.784+0000 7f366e233640 1 --2- 172.21.15.174:0/3559121443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f366806f2d0 0x7f36681f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.767 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.784+0000 7f366e233640 1 -- 172.21.15.174:0/3559121443 >> 172.21.15.174:0/3559121443 conn(0x7f3668066c20 msgr2=0x7f3668150b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.767 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.784+0000 7f366e233640 1 -- 172.21.15.174:0/3559121443 shutdown_connections 2024-03-16T19:29:03.767 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.784+0000 7f366e233640 1 -- 172.21.15.174:0/3559121443 wait complete. 2024-03-16T19:29:03.776 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new ea3e888e-c4c6-444a-8ebe-e26cddb2084b 6 2024-03-16T19:29:03.877 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.895+0000 7f563ebac640 1 Processor -- start 2024-03-16T19:29:03.877 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.895+0000 7f563ebac640 1 -- start start 2024-03-16T19:29:03.878 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.895+0000 7f563ebac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5638066fa0 0x7f5638072e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.878 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.895+0000 7f563ebac640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f56380c87c0 0x7f56380c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.895+0000 7f563ebac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5638152ff0 con 0x7f5638152c10 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.895+0000 7f563ebac640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5638158f50 con 0x7f56380c87c0 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f563ebac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56380ca260 con 0x7f5638066fa0 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f5637fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5638066fa0 0x7f5638072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f563d149640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f56380c87c0 0x7f56380c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f563c948640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f5638152c10 0x7f5638158840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:58512/0 (socket says 172.21.15.174:58512) 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f5637fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5638066fa0 0x7f5638072e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:45556/0 (socket says 172.21.15.174:45556) 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f563c948640 1 -- 172.21.15.174:0/856023814 learned_addr learned my addr 172.21.15.174:0/856023814 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f5637fff640 1 -- 172.21.15.174:0/856023814 >> v2:172.21.15.174:3300/0 conn(0x7f56380c87c0 msgr2=0x7f56380c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f5637fff640 1 --2- 172.21.15.174:0/856023814 >> v2:172.21.15.174:3300/0 conn(0x7f56380c87c0 0x7f56380c8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f5637fff640 1 -- 172.21.15.174:0/856023814 >> v1:172.21.15.174:6789/0 conn(0x7f5638152c10 legacy=0x7f5638158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f56377fe640 1 -- 172.21.15.174:0/856023814 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430957000 0 0) 0x7f5638152ff0 con 0x7f5638152c10 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f5637fff640 1 -- 172.21.15.174:0/856023814 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56380c8180 con 0x7f5638066fa0 2024-03-16T19:29:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.896+0000 7f563d149640 1 --2- 172.21.15.174:0/856023814 >> v2:172.21.15.174:3300/0 conn(0x7f56380c87c0 0x7f56380c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.897+0000 7f5637fff640 1 --2- 172.21.15.174:0/856023814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5638066fa0 0x7f5638072e90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f56200097a0 tx=0x7f562002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=635c11b755af720f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.897+0000 7f56377fe640 1 -- 172.21.15.174:0/856023814 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620031830 con 0x7f5638066fa0 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.897+0000 7f56377fe640 1 -- 172.21.15.174:0/856023814 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5620031990 con 0x7f5638066fa0 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.897+0000 7f56377fe640 1 -- 172.21.15.174:0/856023814 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620031d20 con 0x7f5638066fa0 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.898+0000 7f563ebac640 1 -- 172.21.15.174:0/856023814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5638066fa0 msgr2=0x7f5638072e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.898+0000 7f563ebac640 1 --2- 172.21.15.174:0/856023814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5638066fa0 0x7f5638072e90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f56200097a0 tx=0x7f562002b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.898+0000 7f563ebac640 1 -- 172.21.15.174:0/856023814 shutdown_connections 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.898+0000 7f563ebac640 1 --2- 172.21.15.174:0/856023814 >> v2:172.21.15.174:3300/0 conn(0x7f56380c87c0 0x7f56380c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.898+0000 7f563ebac640 1 --2- 172.21.15.174:0/856023814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5638066fa0 0x7f5638072e90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.898+0000 7f563ebac640 1 -- 172.21.15.174:0/856023814 >> 172.21.15.174:0/856023814 conn(0x7f56380c9390 msgr2=0x7f56380c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.882 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.898+0000 7f563ebac640 1 -- 172.21.15.174:0/856023814 shutdown_connections 2024-03-16T19:29:03.882 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.898+0000 7f563ebac640 1 -- 172.21.15.174:0/856023814 wait complete. 2024-03-16T19:29:03.884 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.899+0000 7f563ebac640 1 Processor -- start 2024-03-16T19:29:03.884 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.899+0000 7f563ebac640 1 -- start start 2024-03-16T19:29:03.884 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f563ebac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5638066fa0 0x7f563814ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.884 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f563ebac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56380c87c0 0x7f5638148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.884 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f563ebac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5638152c10 0x7f56381ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f563ebac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56381564a0 con 0x7f5638152c10 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f563ebac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638156d00 con 0x7f5638066fa0 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f563ebac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5638146d00 con 0x7f56380c87c0 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f5637fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56380c87c0 0x7f5638148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f563c948640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5638066fa0 0x7f563814ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f5637fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56380c87c0 0x7f5638148980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:45568/0 (socket says 172.21.15.174:45568) 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f5637fff640 1 -- 172.21.15.174:0/1516126521 learned_addr learned my addr 172.21.15.174:0/1516126521 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.900+0000 7f563d149640 1 --2- 172.21.15.174:0/1516126521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5638152c10 0x7f56381ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f563c948640 1 -- 172.21.15.174:0/1516126521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56380c87c0 msgr2=0x7f5638148980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f563c948640 1 --2- 172.21.15.174:0/1516126521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56380c87c0 0x7f5638148980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f563c948640 1 -- 172.21.15.174:0/1516126521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5638152c10 msgr2=0x7f56381ee670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f563c948640 1 --2- 172.21.15.174:0/1516126521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5638152c10 0x7f56381ee670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f563c948640 1 -- 172.21.15.174:0/1516126521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5628009400 con 0x7f5638066fa0 2024-03-16T19:29:03.885 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f5637fff640 1 --2- 172.21.15.174:0/1516126521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56380c87c0 0x7f5638148980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f563c948640 1 --2- 172.21.15.174:0/1516126521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5638066fa0 0x7f563814ce70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5630002230 tx=0x7f563000cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f56357fa640 1 -- 172.21.15.174:0/1516126521 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630004d60 con 0x7f5638066fa0 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f56357fa640 1 -- 172.21.15.174:0/1516126521 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563000a640 con 0x7f5638066fa0 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f56357fa640 1 -- 172.21.15.174:0/1516126521 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563000a9f0 con 0x7f5638066fa0 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f563ebac640 1 -- 172.21.15.174:0/1516126521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5620009400 con 0x7f5638066fa0 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f563ebac640 1 -- 172.21.15.174:0/1516126521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56381566b0 con 0x7f5638066fa0 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f56357fa640 1 -- 172.21.15.174:0/1516126521 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f56300070d0 con 0x7f5638066fa0 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.901+0000 7f56357fa640 1 -- 172.21.15.174:0/1516126521 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f56300129f0 con 0x7f5638066fa0 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.902+0000 7f563ebac640 1 -- 172.21.15.174:0/1516126521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56380688d0 con 0x7f5638066fa0 2024-03-16T19:29:03.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.904+0000 7f56357fa640 1 -- 172.21.15.174:0/1516126521 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5630018070 con 0x7f5638066fa0 2024-03-16T19:29:03.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.977+0000 7f563ebac640 1 -- 172.21.15.174:0/1516126521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ea3e888e-c4c6-444a-8ebe-e26cddb2084b", "id": 6} v 0) v1 -- 0x7f5638150430 con 0x7f5638066fa0 2024-03-16T19:29:03.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.981+0000 7f56357fa640 1 -- 172.21.15.174:0/1516126521 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea3e888e-c4c6-444a-8ebe-e26cddb2084b", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5630012400 con 0x7f5638066fa0 2024-03-16T19:29:03.963 INFO:teuthology.orchestra.run.smithi174.stdout:6 2024-03-16T19:29:03.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.982+0000 7f563ebac640 1 -- 172.21.15.174:0/1516126521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5638066fa0 msgr2=0x7f563814ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:03.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.982+0000 7f563ebac640 1 --2- 172.21.15.174:0/1516126521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5638066fa0 0x7f563814ce70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5630002230 tx=0x7f563000cb90 comp rx=0 tx=0).stop 2024-03-16T19:29:03.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.982+0000 7f563ebac640 1 -- 172.21.15.174:0/1516126521 shutdown_connections 2024-03-16T19:29:03.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.982+0000 7f563ebac640 1 --2- 172.21.15.174:0/1516126521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5638152c10 0x7f56381ee670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.982+0000 7f563ebac640 1 --2- 172.21.15.174:0/1516126521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56380c87c0 0x7f5638148980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.982+0000 7f563ebac640 1 --2- 172.21.15.174:0/1516126521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5638066fa0 0x7f563814ce70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:03.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.982+0000 7f563ebac640 1 -- 172.21.15.174:0/1516126521 >> 172.21.15.174:0/1516126521 conn(0x7f56380c9390 msgr2=0x7f563806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:03.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.982+0000 7f563ebac640 1 -- 172.21.15.174:0/1516126521 shutdown_connections 2024-03-16T19:29:03.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:03.982+0000 7f563ebac640 1 -- 172.21.15.174:0/1516126521 wait complete. 2024-03-16T19:29:03.975 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new bb0252a5-3aa1-43f0-be38-37413819720b 7 2024-03-16T19:29:03.988 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:04.006+0000 7f21ef7b3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T19:29:04.030 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:04.048+0000 7fbad1848040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T19:29:04.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.094+0000 7f7589c23640 1 Processor -- start 2024-03-16T19:29:04.077 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.094+0000 7f7589c23640 1 -- start start 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.095+0000 7f7589c23640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75841528a0 0x7f758415c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.095+0000 7f7589c23640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758406f760 con 0x7f758406f2d0 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.095+0000 7f7589c23640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840c4fb0 con 0x7f7584152c80 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.095+0000 7f7589c23640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75840c5290 con 0x7f75841528a0 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.095+0000 7f75837fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75841528a0 0x7f758415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.095+0000 7f7582ffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f7584152c80 0x7f75840c33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:58520/0 (socket says 172.21.15.174:58520) 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.095+0000 7f7582ffd640 1 -- 172.21.15.174:0/222624219 learned_addr learned my addr 172.21.15.174:0/222624219 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.095+0000 7f75827fc640 1 -- 172.21.15.174:0/222624219 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146930552 0 0) 0x7f75840c4fb0 con 0x7f7584152c80 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.096+0000 7f75837fe640 1 -- 172.21.15.174:0/222624219 >> v1:172.21.15.174:6789/0 conn(0x7f7584152c80 legacy=0x7f75840c33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.096+0000 7f75837fe640 1 -- 172.21.15.174:0/222624219 >> v1:172.21.15.172:6789/0 conn(0x7f758406f2d0 legacy=0x7f75840ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.078 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.096+0000 7f75837fe640 1 -- 172.21.15.174:0/222624219 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75840c5570 con 0x7f75841528a0 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.096+0000 7f75837fe640 1 --2- 172.21.15.174:0/222624219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75841528a0 0x7f758415c860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f75780097d0 tx=0x7f757802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3789f4a9a63f051 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.096+0000 7f75827fc640 1 -- 172.21.15.174:0/222624219 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578030880 con 0x7f75841528a0 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.096+0000 7f75827fc640 1 -- 172.21.15.174:0/222624219 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75780309e0 con 0x7f75841528a0 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.096+0000 7f75827fc640 1 -- 172.21.15.174:0/222624219 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578030d70 con 0x7f75841528a0 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.097+0000 7f7589c23640 1 -- 172.21.15.174:0/222624219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75841528a0 msgr2=0x7f758415c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.097+0000 7f7589c23640 1 --2- 172.21.15.174:0/222624219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75841528a0 0x7f758415c860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f75780097d0 tx=0x7f757802b9c0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.097+0000 7f7589c23640 1 -- 172.21.15.174:0/222624219 shutdown_connections 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.097+0000 7f7589c23640 1 --2- 172.21.15.174:0/222624219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75841528a0 0x7f758415c860 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.097+0000 7f7589c23640 1 -- 172.21.15.174:0/222624219 >> 172.21.15.174:0/222624219 conn(0x7f7584066c20 msgr2=0x7f7584067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.097+0000 7f7589c23640 1 -- 172.21.15.174:0/222624219 shutdown_connections 2024-03-16T19:29:04.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.097+0000 7f7589c23640 1 -- 172.21.15.174:0/222624219 wait complete. 2024-03-16T19:29:04.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.098+0000 7f7589c23640 1 Processor -- start 2024-03-16T19:29:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.098+0000 7f7589c23640 1 -- start start 2024-03-16T19:29:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7589c23640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f758406f2d0 0x7f75840ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7589c23640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75841528a0 0x7f75840b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7589c23640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7584152c80 0x7f75840b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7589c23640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758406d290 con 0x7f75841528a0 2024-03-16T19:29:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7589c23640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758406c3a0 con 0x7f7584152c80 2024-03-16T19:29:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7589c23640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758406c680 con 0x7f758406f2d0 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7583fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7584152c80 0x7f75840b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f75837fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f758406f2d0 0x7f75840ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f75837fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f758406f2d0 0x7f75840ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:45590/0 (socket says 172.21.15.174:45590) 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f75837fe640 1 -- 172.21.15.174:0/2512253975 learned_addr learned my addr 172.21.15.174:0/2512253975 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7583fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7584152c80 0x7f75840b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:57868/0 (socket says 172.21.15.174:57868) 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7582ffd640 1 --2- 172.21.15.174:0/2512253975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75841528a0 0x7f75840b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7583fff640 1 -- 172.21.15.174:0/2512253975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f758406f2d0 msgr2=0x7f75840ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7583fff640 1 --2- 172.21.15.174:0/2512253975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f758406f2d0 0x7f75840ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7583fff640 1 -- 172.21.15.174:0/2512253975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75841528a0 msgr2=0x7f75840b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7583fff640 1 --2- 172.21.15.174:0/2512253975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75841528a0 0x7f75840b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f7583fff640 1 -- 172.21.15.174:0/2512253975 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7578009400 con 0x7f7584152c80 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.099+0000 7f75837fe640 1 --2- 172.21.15.174:0/2512253975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f758406f2d0 0x7f75840ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.100+0000 7f7583fff640 1 --2- 172.21.15.174:0/2512253975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7584152c80 0x7f75840b12f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f757c00c730 tx=0x7f757c00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.100+0000 7f756bfff640 1 -- 172.21.15.174:0/2512253975 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c006d50 con 0x7f7584152c80 2024-03-16T19:29:04.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.100+0000 7f756bfff640 1 -- 172.21.15.174:0/2512253975 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c003970 con 0x7f7584152c80 2024-03-16T19:29:04.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.100+0000 7f7589c23640 1 -- 172.21.15.174:0/2512253975 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758406c960 con 0x7f7584152c80 2024-03-16T19:29:04.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.100+0000 7f756bfff640 1 -- 172.21.15.174:0/2512253975 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c003d60 con 0x7f7584152c80 2024-03-16T19:29:04.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.100+0000 7f7589c23640 1 -- 172.21.15.174:0/2512253975 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758406cf20 con 0x7f7584152c80 2024-03-16T19:29:04.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.100+0000 7f756bfff640 1 -- 172.21.15.174:0/2512253975 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f757c005460 con 0x7f7584152c80 2024-03-16T19:29:04.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.100+0000 7f756bfff640 1 -- 172.21.15.174:0/2512253975 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f757c012a80 con 0x7f7584152c80 2024-03-16T19:29:04.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.101+0000 7f7589c23640 1 -- 172.21.15.174:0/2512253975 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758406d400 con 0x7f7584152c80 2024-03-16T19:29:04.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.103+0000 7f756bfff640 1 -- 172.21.15.174:0/2512253975 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f757c021060 con 0x7f7584152c80 2024-03-16T19:29:04.123 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:04.141+0000 7f21ef7b3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T19:29:04.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.176+0000 7f7589c23640 1 -- 172.21.15.174:0/2512253975 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bb0252a5-3aa1-43f0-be38-37413819720b", "id": 7} v 0) v1 -- 0x7f75840688d0 con 0x7f7584152c80 2024-03-16T19:29:04.163 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.181+0000 7f756bfff640 1 -- 172.21.15.174:0/2512253975 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bb0252a5-3aa1-43f0-be38-37413819720b", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f757c016020 con 0x7f7584152c80 2024-03-16T19:29:04.163 INFO:teuthology.orchestra.run.smithi174.stdout:7 2024-03-16T19:29:04.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.182+0000 7f7589c23640 1 -- 172.21.15.174:0/2512253975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7584152c80 msgr2=0x7f75840b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.182+0000 7f7589c23640 1 --2- 172.21.15.174:0/2512253975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7584152c80 0x7f75840b12f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f757c00c730 tx=0x7f757c00cbf0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.182+0000 7f7589c23640 1 -- 172.21.15.174:0/2512253975 shutdown_connections 2024-03-16T19:29:04.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.182+0000 7f7589c23640 1 --2- 172.21.15.174:0/2512253975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7584152c80 0x7f75840b12f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.182+0000 7f7589c23640 1 --2- 172.21.15.174:0/2512253975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75841528a0 0x7f75840b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.182+0000 7f7589c23640 1 --2- 172.21.15.174:0/2512253975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f758406f2d0 0x7f75840ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.182+0000 7f7589c23640 1 -- 172.21.15.174:0/2512253975 >> 172.21.15.174:0/2512253975 conn(0x7f7584066c20 msgr2=0x7f7584150130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:04.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.182+0000 7f7589c23640 1 -- 172.21.15.174:0/2512253975 shutdown_connections 2024-03-16T19:29:04.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-16T19:29:04.182+0000 7f7589c23640 1 -- 172.21.15.174:0/2512253975 wait complete. 2024-03-16T19:29:04.172 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:04.190+0000 7fbad1848040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T19:29:04.174 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-16T19:29:04.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-16T19:29:04.177 INFO:tasks.ceph.osd.0:Started 2024-03-16T19:29:04.177 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-16T19:29:04.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-16T19:29:04.179 INFO:tasks.ceph.osd.1:Started 2024-03-16T19:29:04.179 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-16T19:29:04.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-16T19:29:04.182 INFO:tasks.ceph.osd.2:Started 2024-03-16T19:29:04.182 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-16T19:29:04.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-16T19:29:04.184 INFO:tasks.ceph.osd.3:Started 2024-03-16T19:29:04.184 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-16T19:29:04.184 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-16T19:29:04.186 INFO:tasks.ceph.osd.4:Started 2024-03-16T19:29:04.186 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-16T19:29:04.186 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-16T19:29:04.188 INFO:tasks.ceph.osd.5:Started 2024-03-16T19:29:04.188 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-16T19:29:04.189 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-16T19:29:04.190 INFO:tasks.ceph.osd.6:Started 2024-03-16T19:29:04.191 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-16T19:29:04.191 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-16T19:29:04.193 INFO:tasks.ceph.osd.7:Started 2024-03-16T19:29:04.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:04.251 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:04.268+0000 7f21ef7b3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T19:29:04.301 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:04.319+0000 7fbad1848040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T19:29:04.310 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:04.328+0000 7f21ef7b3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T19:29:04.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.341+0000 7fb9f0965640 1 Processor -- start 2024-03-16T19:29:04.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.341+0000 7fb9f0965640 1 -- start start 2024-03-16T19:29:04.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.342+0000 7fb9f0965640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb9ec154b60 0x7fb9ec15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.342+0000 7fb9f0965640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ec0c7750 con 0x7fb9ec154f40 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.342+0000 7fb9f0965640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ec0c7190 con 0x7fb9ec0c7f80 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.342+0000 7fb9f0965640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ec0c6e10 con 0x7fb9ec154b60 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.342+0000 7fb9e9d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb9ec154f40 0x7fb9ec06c1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:58964/0 (socket says 172.21.15.172:58964) 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.342+0000 7fb9e9d9b640 1 -- 172.21.15.172:0/189321443 learned_addr learned my addr 172.21.15.172:0/189321443 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9ea59c640 1 --2- 172.21.15.172:0/189321443 >> v2:172.21.15.174:3300/0 conn(0x7fb9ec154b60 0x7fb9ec15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431581994 0 0) 0x7fb9ec0c7750 con 0x7fb9ec154f40 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d4003520 con 0x7fb9ec154f40 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263165816 0 0) 0x7fb9ec0c7190 con 0x7fb9ec0c7f80 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ec0c7750 con 0x7fb9ec0c7f80 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1772115195 0 0) 0x7fb9d4003520 con 0x7fb9ec154f40 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 >> v2:172.21.15.174:3300/0 conn(0x7fb9ec154b60 msgr2=0x7fb9ec15aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 --2- 172.21.15.172:0/189321443 >> v2:172.21.15.174:3300/0 conn(0x7fb9ec154b60 0x7fb9ec15aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 >> v1:172.21.15.174:6789/0 conn(0x7fb9ec0c7f80 legacy=0x7fb9ec071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec0c7470 con 0x7fb9ec154f40 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7fb9dc001670 con 0x7fb9ec154f40 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 >> v1:172.21.15.172:6789/0 conn(0x7fb9ec154f40 legacy=0x7fb9ec06c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4003b00 0x7fb9d4003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d40047c0 0x7fb9d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9d4005180 0x7fb9d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d400bfd0 con 0x7fb9d40047c0 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d400c140 con 0x7fb9d4003b00 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4014b10 con 0x7fb9d4005180 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9e9d9b640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d40047c0 0x7fb9d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9ea59c640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4003b00 0x7fb9d4003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9ead9d640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9d4005180 0x7fb9d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9ea59c640 1 -- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9d4005180 msgr2=0x7fb9d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:04.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9ea59c640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9d4005180 0x7fb9d400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9ea59c640 1 -- 172.21.15.172:0/189321443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d40047c0 msgr2=0x7fb9d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:04.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9ea59c640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d40047c0 0x7fb9d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9ea59c640 1 -- 172.21.15.172:0/189321443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d4003520 con 0x7fb9d4003b00 2024-03-16T19:29:04.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.343+0000 7fb9ead9d640 1 -- 172.21.15.172:0/189321443 reap_dead start 2024-03-16T19:29:04.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.344+0000 7fb9ea59c640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4003b00 0x7fb9d4003f00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0009ac0 tx=0x7fb9d40057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66e4010aea729ff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:04.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.344+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e002c6b0 con 0x7fb9d4003b00 2024-03-16T19:29:04.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.344+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e002c810 con 0x7fb9d4003b00 2024-03-16T19:29:04.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.344+0000 7fb9e959a640 1 -- 172.21.15.172:0/189321443 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e002cba0 con 0x7fb9d4003b00 2024-03-16T19:29:04.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.345+0000 7fb9f0965640 1 -- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4003b00 msgr2=0x7fb9d4003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.345+0000 7fb9f0965640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4003b00 0x7fb9d4003f00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0009ac0 tx=0x7fb9d40057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.345+0000 7fb9f0965640 1 -- 172.21.15.172:0/189321443 shutdown_connections 2024-03-16T19:29:04.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.345+0000 7fb9f0965640 1 --2- 172.21.15.172:0/189321443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4003b00 0x7fb9d4003f00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.345+0000 7fb9f0965640 1 -- 172.21.15.172:0/189321443 >> 172.21.15.172:0/189321443 conn(0x7fb9ec06e8d0 msgr2=0x7fb9ec06ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:04.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.345+0000 7fb9f0965640 1 -- 172.21.15.172:0/189321443 shutdown_connections 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.345+0000 7fb9f0965640 1 -- 172.21.15.172:0/189321443 wait complete. 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.346+0000 7fb9f0965640 1 Processor -- start 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.346+0000 7fb9f0965640 1 -- start start 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9f0965640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d4003b00 0x7fb9ec14c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9f0965640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9ec1ec4b0 0x7fb9ec15b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9f0965640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9ec14e270 0x7fb9ec14ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9f0965640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec14dde0 con 0x7fb9d4003b00 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9f0965640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec1588b0 con 0x7fb9ec14e270 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9f0965640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec14c910 con 0x7fb9ec1ec4b0 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9ea59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d4003b00 0x7fb9ec14c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9ea59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d4003b00 0x7fb9ec14c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42892/0 (socket says 172.21.15.172:42892) 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9ea59c640 1 -- 172.21.15.172:0/388347293 learned_addr learned my addr 172.21.15.172:0/388347293 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9e9d9b640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9ec1ec4b0 0x7fb9ec15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9ead9d640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9ec14e270 0x7fb9ec14ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.347+0000 7fb9ea59c640 1 -- 172.21.15.172:0/388347293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9ec1ec4b0 msgr2=0x7fb9ec15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9ea59c640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9ec1ec4b0 0x7fb9ec15b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9ea59c640 1 -- 172.21.15.172:0/388347293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9ec14e270 msgr2=0x7fb9ec14ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9ea59c640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9ec14e270 0x7fb9ec14ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9ea59c640 1 -- 172.21.15.172:0/388347293 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec14cbf0 con 0x7fb9d4003b00 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9e9d9b640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9ec1ec4b0 0x7fb9ec15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9ea59c640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d4003b00 0x7fb9ec14c260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0000c00 tx=0x7fb9e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9daffd640 1 -- 172.21.15.172:0/388347293 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0003810 con 0x7fb9d4003b00 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9daffd640 1 -- 172.21.15.172:0/388347293 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e0003970 con 0x7fb9d4003b00 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9f0965640 1 -- 172.21.15.172:0/388347293 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ec14d790 con 0x7fb9d4003b00 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9f0965640 1 -- 172.21.15.172:0/388347293 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec074b40 con 0x7fb9d4003b00 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.348+0000 7fb9daffd640 1 -- 172.21.15.172:0/388347293 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0003d20 con 0x7fb9d4003b00 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.349+0000 7fb9daffd640 1 -- 172.21.15.172:0/388347293 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb9e0047450 con 0x7fb9d4003b00 2024-03-16T19:29:04.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.349+0000 7fb9daffd640 1 -- 172.21.15.172:0/388347293 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb9e0034b60 con 0x7fb9d4003b00 2024-03-16T19:29:04.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.349+0000 7fb9f0965640 1 -- 172.21.15.172:0/388347293 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ec14d5f0 con 0x7fb9d4003b00 2024-03-16T19:29:04.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.351+0000 7fb9daffd640 1 -- 172.21.15.172:0/388347293 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb9e0066dd0 con 0x7fb9d4003b00 2024-03-16T19:29:04.348 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:04.366+0000 7fbad1848040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T19:29:04.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.412+0000 7fb9f0965640 1 -- 172.21.15.172:0/388347293 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9ec0688d0 con 0x7fb9d4003b00 2024-03-16T19:29:04.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.413+0000 7fb9daffd640 1 -- 172.21.15.172:0/388347293 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fb9e0055020 con 0x7fb9d4003b00 2024-03-16T19:29:04.395 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:04.395 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":10,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:04.178546+0000","last_up_change":"0.000000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:04.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.413+0000 7fb9f0965640 1 -- 172.21.15.172:0/388347293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d4003b00 msgr2=0x7fb9ec14c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:04.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.413+0000 7fb9f0965640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d4003b00 0x7fb9ec14c260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0000c00 tx=0x7fb9e0004730 comp rx=0 tx=0).stop 2024-03-16T19:29:04.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.413+0000 7fb9f0965640 1 -- 172.21.15.172:0/388347293 shutdown_connections 2024-03-16T19:29:04.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.413+0000 7fb9f0965640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9ec14e270 0x7fb9ec14ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.413+0000 7fb9f0965640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9ec1ec4b0 0x7fb9ec15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.413+0000 7fb9f0965640 1 --2- 172.21.15.172:0/388347293 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb9d4003b00 0x7fb9ec14c260 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:04.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.413+0000 7fb9f0965640 1 -- 172.21.15.172:0/388347293 >> 172.21.15.172:0/388347293 conn(0x7fb9ec06e8d0 msgr2=0x7fb9ec0c5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:04.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.414+0000 7fb9f0965640 1 -- 172.21.15.172:0/388347293 shutdown_connections 2024-03-16T19:29:04.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:04.414+0000 7fb9f0965640 1 -- 172.21.15.172:0/388347293 wait complete. 2024-03-16T19:29:04.407 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-16T19:29:04.407 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-16T19:29:04.428 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:04.446+0000 7f21ef7b3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T19:29:04.459 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:04.477+0000 7fbad1848040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T19:29:04.524 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:04.542+0000 7f21ef7b3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T19:29:04.555 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:04.573+0000 7fbad1848040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T19:29:04.678 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:04.696+0000 7f21ef7b3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T19:29:04.709 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:04.727+0000 7fbad1848040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T19:29:04.720 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:04.738+0000 7f21ef7b3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T19:29:04.751 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:04.769+0000 7fbad1848040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T19:29:04.790 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:04.808+0000 7ff5415e1640 -1 Falling back to public interface 2024-03-16T19:29:04.792 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:04.810+0000 7f3904a15640 -1 Falling back to public interface 2024-03-16T19:29:04.794 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:04.812+0000 7ff525321640 -1 Falling back to public interface 2024-03-16T19:29:04.797 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:04.815+0000 7fda8a981640 -1 Falling back to public interface 2024-03-16T19:29:04.800 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:04.818+0000 7f9f7d19a640 -1 Falling back to public interface 2024-03-16T19:29:04.801 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:04.819+0000 7fb3ac0ce640 -1 Falling back to public interface 2024-03-16T19:29:04.801 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:04.819+0000 7fbcdc338640 -1 Falling back to public interface 2024-03-16T19:29:04.803 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:04.821+0000 7f198f21f640 -1 Falling back to public interface 2024-03-16T19:29:04.879 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:04.897+0000 7f21ef7b3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T19:29:04.909 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:04.927+0000 7fbad1848040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T19:29:05.214 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:05.305 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.323+0000 7fea1550d640 1 Processor -- start 2024-03-16T19:29:05.305 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.323+0000 7fea1550d640 1 -- start start 2024-03-16T19:29:05.307 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.323+0000 7fea1550d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fea10154e60 0x7fea100cb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:05.307 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea1550d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fea100d0740 0x7fea100d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:05.307 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea1550d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea100c5d10 con 0x7fea1015e000 2024-03-16T19:29:05.307 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea1550d640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea10155240 con 0x7fea100d0740 2024-03-16T19:29:05.307 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea1550d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea100c6e70 con 0x7fea10154e60 2024-03-16T19:29:05.307 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea0f7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fea100d0740 0x7fea100d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:05.307 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea0f7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fea100d0740 0x7fea100d0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42908/0 (socket says 172.21.15.172:42908) 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea0f7fe640 1 -- 172.21.15.172:0/330783035 learned_addr learned my addr 172.21.15.172:0/330783035 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea0effd640 1 --2- 172.21.15.172:0/330783035 >> v2:172.21.15.174:3300/0 conn(0x7fea10154e60 0x7fea100cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea0f7fe640 1 -- 172.21.15.172:0/330783035 >> v2:172.21.15.174:3300/0 conn(0x7fea10154e60 msgr2=0x7fea100cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea0f7fe640 1 --2- 172.21.15.172:0/330783035 >> v2:172.21.15.174:3300/0 conn(0x7fea10154e60 0x7fea100cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea0f7fe640 1 -- 172.21.15.172:0/330783035 >> v1:172.21.15.174:6789/0 conn(0x7fea1015e000 legacy=0x7fea100c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.324+0000 7fea0f7fe640 1 -- 172.21.15.172:0/330783035 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1015e7d0 con 0x7fea100d0740 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.325+0000 7fea0f7fe640 1 --2- 172.21.15.172:0/330783035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 0x7fea100d0bd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea1008e150 tx=0x7fea0802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4b455b2841a5aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.325+0000 7fea0dffb640 1 -- 172.21.15.172:0/330783035 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0802cb20 con 0x7fea100d0740 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.325+0000 7fea0dffb640 1 -- 172.21.15.172:0/330783035 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0802cc80 con 0x7fea100d0740 2024-03-16T19:29:05.308 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.325+0000 7fea0dffb640 1 -- 172.21.15.172:0/330783035 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea080343f0 con 0x7fea100d0740 2024-03-16T19:29:05.311 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.326+0000 7fea1550d640 1 -- 172.21.15.172:0/330783035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 msgr2=0x7fea100d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:05.311 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.326+0000 7fea1550d640 1 --2- 172.21.15.172:0/330783035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 0x7fea100d0bd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea1008e150 tx=0x7fea0802bf80 comp rx=0 tx=0).stop 2024-03-16T19:29:05.311 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.326+0000 7fea1550d640 1 -- 172.21.15.172:0/330783035 shutdown_connections 2024-03-16T19:29:05.311 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.326+0000 7fea1550d640 1 --2- 172.21.15.172:0/330783035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 0x7fea100d0bd0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:05.311 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.326+0000 7fea1550d640 1 --2- 172.21.15.172:0/330783035 >> v2:172.21.15.174:3300/0 conn(0x7fea10154e60 0x7fea100cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:05.311 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.326+0000 7fea1550d640 1 -- 172.21.15.172:0/330783035 >> 172.21.15.172:0/330783035 conn(0x7fea1006bbd0 msgr2=0x7fea1006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:05.311 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.326+0000 7fea1550d640 1 -- 172.21.15.172:0/330783035 shutdown_connections 2024-03-16T19:29:05.311 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.326+0000 7fea1550d640 1 -- 172.21.15.172:0/330783035 wait complete. 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.327+0000 7fea1550d640 1 Processor -- start 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.327+0000 7fea1550d640 1 -- start start 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea1550d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 0x7fea10159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea1550d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10154e60 0x7fea10151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea1550d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1015e000 0x7fea101588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea1550d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea101f37c0 con 0x7fea100d0740 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea1550d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10151490 con 0x7fea1015e000 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea1550d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1014c280 con 0x7fea10154e60 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0effd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 0x7fea10159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1015e000 0x7fea101588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0effd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 0x7fea10159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42920/0 (socket says 172.21.15.172:42920) 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1015e000 0x7fea101588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:53464/0 (socket says 172.21.15.172:53464) 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0effd640 1 -- 172.21.15.172:0/2013004043 learned_addr learned my addr 172.21.15.172:0/2013004043 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0e7fc640 1 --2- 172.21.15.172:0/2013004043 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10154e60 0x7fea10151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:05.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0effd640 1 -- 172.21.15.172:0/2013004043 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10154e60 msgr2=0x7fea10151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0effd640 1 --2- 172.21.15.172:0/2013004043 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10154e60 0x7fea10151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0effd640 1 -- 172.21.15.172:0/2013004043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1015e000 msgr2=0x7fea101588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0effd640 1 --2- 172.21.15.172:0/2013004043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1015e000 0x7fea101588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0effd640 1 -- 172.21.15.172:0/2013004043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08009b30 con 0x7fea100d0740 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0f7fe640 1 --2- 172.21.15.172:0/2013004043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1015e000 0x7fea101588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea0effd640 1 --2- 172.21.15.172:0/2013004043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 0x7fea10159d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea0000e9b0 tx=0x7fea0000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fe9f77fe640 1 -- 172.21.15.172:0/2013004043 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea00009670 con 0x7fea100d0740 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fe9f77fe640 1 -- 172.21.15.172:0/2013004043 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0000c9a0 con 0x7fea100d0740 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fe9f77fe640 1 -- 172.21.15.172:0/2013004043 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0000cd50 con 0x7fea100d0740 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.328+0000 7fea1550d640 1 -- 172.21.15.172:0/2013004043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1015b8b0 con 0x7fea100d0740 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.329+0000 7fea1550d640 1 -- 172.21.15.172:0/2013004043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1015b2f0 con 0x7fea100d0740 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.329+0000 7fe9f77fe640 1 -- 172.21.15.172:0/2013004043 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fea00013470 con 0x7fea100d0740 2024-03-16T19:29:05.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.329+0000 7fe9f77fe640 1 -- 172.21.15.172:0/2013004043 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fea000123d0 con 0x7fea100d0740 2024-03-16T19:29:05.314 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.329+0000 7fea1550d640 1 -- 172.21.15.172:0/2013004043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea10068af0 con 0x7fea100d0740 2024-03-16T19:29:05.314 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.331+0000 7fe9f77fe640 1 -- 172.21.15.172:0/2013004043 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fea00021060 con 0x7fea100d0740 2024-03-16T19:29:05.375 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.394+0000 7fea1550d640 1 -- 172.21.15.172:0/2013004043 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea100cbbd0 con 0x7fea100d0740 2024-03-16T19:29:05.376 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.394+0000 7fe9f77fe640 1 -- 172.21.15.172:0/2013004043 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fea0002a090 con 0x7fea100d0740 2024-03-16T19:29:05.376 INFO:teuthology.misc.health.smithi172.stdout: 2024-03-16T19:29:05.376 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":10,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:04.178546+0000","last_up_change":"0.000000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:05.377 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.395+0000 7fea1550d640 1 -- 172.21.15.172:0/2013004043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 msgr2=0x7fea10159d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:05.378 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.395+0000 7fea1550d640 1 --2- 172.21.15.172:0/2013004043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 0x7fea10159d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea0000e9b0 tx=0x7fea0000ee70 comp rx=0 tx=0).stop 2024-03-16T19:29:05.378 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.395+0000 7fea1550d640 1 -- 172.21.15.172:0/2013004043 shutdown_connections 2024-03-16T19:29:05.378 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.395+0000 7fea1550d640 1 --2- 172.21.15.172:0/2013004043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1015e000 0x7fea101588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:05.378 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.395+0000 7fea1550d640 1 --2- 172.21.15.172:0/2013004043 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10154e60 0x7fea10151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:05.378 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.395+0000 7fea1550d640 1 --2- 172.21.15.172:0/2013004043 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea100d0740 0x7fea10159d50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:05.378 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.395+0000 7fea1550d640 1 -- 172.21.15.172:0/2013004043 >> 172.21.15.172:0/2013004043 conn(0x7fea1006bbd0 msgr2=0x7fea100cabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:05.378 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.395+0000 7fea1550d640 1 -- 172.21.15.172:0/2013004043 shutdown_connections 2024-03-16T19:29:05.378 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:05.395+0000 7fea1550d640 1 -- 172.21.15.172:0/2013004043 wait complete. 2024-03-16T19:29:05.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-16T19:29:05.711 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:05.729+0000 7f21ef7b3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T19:29:05.989 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:06.006+0000 7f21ef7b3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T19:29:05.991 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:06.009+0000 7fbad1848040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T19:29:06.041 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:06.059+0000 7f21ef7b3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T19:29:06.157 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:06.175+0000 7f21ef7b3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T19:29:06.271 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:06.289+0000 7fbad1848040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T19:29:06.322 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:06.340+0000 7fbad1848040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T19:29:06.437 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:29:06.455+0000 7fbad1848040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T19:29:07.183 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:07.201+0000 7f218f037640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-16T19:29:07.184 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-16T19:29:07.201+0000 7f218f037640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-16T19:29:08.254 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:08.272+0000 7ff525321640 -1 osd.4 0 log_to_monitors true 2024-03-16T19:29:08.261 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:08.279+0000 7ff5415e1640 -1 osd.7 0 log_to_monitors true 2024-03-16T19:29:08.282 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:08.300+0000 7fb3ac0ce640 -1 osd.6 0 log_to_monitors true 2024-03-16T19:29:08.305 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:08.323+0000 7f198f21f640 -1 osd.2 0 log_to_monitors true 2024-03-16T19:29:08.320 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:08.338+0000 7f9f7d19a640 -1 osd.5 0 log_to_monitors true 2024-03-16T19:29:08.322 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:08.340+0000 7f3904a15640 -1 osd.0 0 log_to_monitors true 2024-03-16T19:29:08.330 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:08.348+0000 7fda8a981640 -1 osd.1 0 log_to_monitors true 2024-03-16T19:29:08.337 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:08.355+0000 7fbcdc338640 -1 osd.3 0 log_to_monitors true 2024-03-16T19:29:12.159 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:12.177+0000 7ff520a94640 -1 osd.4 0 waiting for initial osdmap 2024-03-16T19:29:12.171 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:12.189+0000 7ff5190c6640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:29:12.194 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:12.228 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:12.246+0000 7f198a465640 -1 osd.2 0 waiting for initial osdmap 2024-03-16T19:29:12.229 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:12.248+0000 7fbcd7aab640 -1 osd.3 0 waiting for initial osdmap 2024-03-16T19:29:12.246 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:12.264+0000 7fbcd00dd640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:29:12.246 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:12.264+0000 7f1982a97640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:29:12.258 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:12.276+0000 7f9f7890d640 -1 osd.5 0 waiting for initial osdmap 2024-03-16T19:29:12.264 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:12.282+0000 7f38ffba5640 -1 osd.0 0 waiting for initial osdmap 2024-03-16T19:29:12.268 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:12.286+0000 7f9f70f3f640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:29:12.274 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:12.293+0000 7ff53e58c640 -1 osd.7 0 waiting for initial osdmap 2024-03-16T19:29:12.277 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:12.295+0000 7f38f81d7640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:29:12.279 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:12.298+0000 7fda860f4640 -1 osd.1 0 waiting for initial osdmap 2024-03-16T19:29:12.282 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:12.301+0000 7fb3a9079640 -1 osd.6 0 waiting for initial osdmap 2024-03-16T19:29:12.288 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:12.307+0000 7ff535386640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:29:12.293 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:12.311+0000 7fda7e726640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:29:12.297 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:12.315+0000 7fb39fe73640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:29:12.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.330+0000 7f58bfd77640 1 Processor -- start 2024-03-16T19:29:12.312 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.330+0000 7f58bfd77640 1 -- start start 2024-03-16T19:29:12.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bfd77640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f58b806e510 0x7f58b815b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:12.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bfd77640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b815e670 con 0x7f58b80c7c30 2024-03-16T19:29:12.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bfd77640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b815e950 con 0x7f58b8154ce0 2024-03-16T19:29:12.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bfd77640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b815a8d0 con 0x7f58b806e510 2024-03-16T19:29:12.313 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bd312640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f58b806e510 0x7f58b815b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:12.314 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bd312640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f58b806e510 0x7f58b815b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:51924/0 (socket says 172.21.15.172:51924) 2024-03-16T19:29:12.314 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bd312640 1 -- 172.21.15.172:0/4036301998 learned_addr learned my addr 172.21.15.172:0/4036301998 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:12.314 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58be314640 1 --1- 172.21.15.172:0/4036301998 >> v1:172.21.15.172:6789/0 conn(0x7f58b80c7c30 0x7f58b806aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:48416/0 (socket says 172.21.15.172:48416) 2024-03-16T19:29:12.314 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bd312640 1 -- 172.21.15.172:0/4036301998 >> v1:172.21.15.174:6789/0 conn(0x7f58b8154ce0 legacy=0x7f58b815dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:12.314 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bd312640 1 -- 172.21.15.172:0/4036301998 >> v1:172.21.15.172:6789/0 conn(0x7f58b80c7c30 legacy=0x7f58b806aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:12.314 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bd312640 1 -- 172.21.15.172:0/4036301998 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b815ec30 con 0x7f58b806e510 2024-03-16T19:29:12.314 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.331+0000 7f58bcb11640 1 -- 172.21.15.172:0/4036301998 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969147281 0 0) 0x7f58b815e670 con 0x7f58b80c7c30 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bd312640 1 --2- 172.21.15.172:0/4036301998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b806e510 0x7f58b815b2e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009870 tx=0x7f58ac02bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7575f1294ebb248 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bcb11640 1 -- 172.21.15.172:0/4036301998 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac02d870 con 0x7f58b806e510 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bcb11640 1 -- 172.21.15.172:0/4036301998 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac02d9d0 con 0x7f58b806e510 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bcb11640 1 -- 172.21.15.172:0/4036301998 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac02dd60 con 0x7f58b806e510 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bfd77640 1 -- 172.21.15.172:0/4036301998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b806e510 msgr2=0x7f58b815b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bfd77640 1 --2- 172.21.15.172:0/4036301998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b806e510 0x7f58b815b2e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009870 tx=0x7f58ac02bab0 comp rx=0 tx=0).stop 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bfd77640 1 -- 172.21.15.172:0/4036301998 shutdown_connections 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bfd77640 1 --2- 172.21.15.172:0/4036301998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b806e510 0x7f58b815b2e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bfd77640 1 -- 172.21.15.172:0/4036301998 >> 172.21.15.172:0/4036301998 conn(0x7f58b806bff0 msgr2=0x7f58b806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:12.316 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bfd77640 1 -- 172.21.15.172:0/4036301998 shutdown_connections 2024-03-16T19:29:12.317 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.332+0000 7f58bfd77640 1 -- 172.21.15.172:0/4036301998 wait complete. 2024-03-16T19:29:12.317 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bfd77640 1 Processor -- start 2024-03-16T19:29:12.317 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bfd77640 1 -- start start 2024-03-16T19:29:12.319 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bfd77640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58b806e510 0x7f58b815d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:12.319 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bfd77640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b80c7c30 0x7f58b80c5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:12.319 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bfd77640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58b8154ce0 0x7f58b80c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:12.319 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bfd77640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b80c5040 con 0x7f58b80c7c30 2024-03-16T19:29:12.319 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bfd77640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b814d9e0 con 0x7f58b8154ce0 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bfd77640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b814dc30 con 0x7f58b806e510 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bd312640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b80c7c30 0x7f58b80c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bd312640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b80c7c30 0x7f58b80c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:51936/0 (socket says 172.21.15.172:51936) 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bd312640 1 -- 172.21.15.172:0/877097602 learned_addr learned my addr 172.21.15.172:0/877097602 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58bdb13640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58b806e510 0x7f58b815d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.334+0000 7f58be314640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58b8154ce0 0x7f58b80c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58bd312640 1 -- 172.21.15.172:0/877097602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58b806e510 msgr2=0x7f58b815d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58bd312640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58b806e510 0x7f58b815d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58bd312640 1 -- 172.21.15.172:0/877097602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58b8154ce0 msgr2=0x7f58b80c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58bd312640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58b8154ce0 0x7f58b80c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58bd312640 1 -- 172.21.15.172:0/877097602 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac0094d0 con 0x7f58b80c7c30 2024-03-16T19:29:12.320 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58be314640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58b8154ce0 0x7f58b80c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58bd312640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b80c7c30 0x7f58b80c5460 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009990 tx=0x7f58ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58aa7fc640 1 -- 172.21.15.172:0/877097602 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac03c040 con 0x7f58b80c7c30 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58aa7fc640 1 -- 172.21.15.172:0/877097602 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac037cc0 con 0x7f58b80c7c30 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58aa7fc640 1 -- 172.21.15.172:0/877097602 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac02e680 con 0x7f58b80c7c30 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b81f0ac0 con 0x7f58b80c7c30 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.335+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b81e6960 con 0x7f58b80c7c30 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.336+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b8074f50 con 0x7f58b80c7c30 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.336+0000 7f58aa7fc640 1 -- 172.21.15.172:0/877097602 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 4) v1 ==== 89406+0+0 (secure 0 0 0) 0x7f58ac04b430 con 0x7f58b80c7c30 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.337+0000 7f58aa7fc640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f589c06b320 0x7f589c06d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.337+0000 7f58aa7fc640 1 -- 172.21.15.172:0/877097602 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f58ac032070 con 0x7f58b80c7c30 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.337+0000 7f58bdb13640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f589c06b320 0x7f589c06d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:12.321 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.337+0000 7f58bdb13640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f589c06b320 0x7f589c06d7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f58b4000f60 tx=0x7f58b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:12.323 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.341+0000 7f58aa7fc640 1 -- 172.21.15.172:0/877097602 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58ac076500 con 0x7f58b80c7c30 2024-03-16T19:29:12.496 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.514+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58b80c5180 con 0x7f58b80c7c30 2024-03-16T19:29:12.497 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.514+0000 7f58aa7fc640 1 -- 172.21.15.172:0/877097602 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f58b80c5180 con 0x7f58b80c7c30 2024-03-16T19:29:12.497 INFO:teuthology.misc.health.smithi172.stdout: 2024-03-16T19:29:12.497 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":12,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:10.199783+0000","last_up_change":"0.000000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:12.499 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f589c06b320 msgr2=0x7f589c06d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:12.499 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f589c06b320 0x7f589c06d7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f58b4000f60 tx=0x7f58b4009040 comp rx=0 tx=0).stop 2024-03-16T19:29:12.499 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b80c7c30 msgr2=0x7f58b80c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:12.499 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b80c7c30 0x7f58b80c5460 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009990 tx=0x7f58ac004730 comp rx=0 tx=0).stop 2024-03-16T19:29:12.499 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 shutdown_connections 2024-03-16T19:29:12.500 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58b8154ce0 0x7f58b80c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:12.500 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58b80c7c30 0x7f58b80c5460 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:12.500 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f589c06b320 0x7f589c06d7e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:12.500 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 --2- 172.21.15.172:0/877097602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58b806e510 0x7f58b815d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:12.500 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.516+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 >> 172.21.15.172:0/877097602 conn(0x7f58b806bff0 msgr2=0x7f58b8070080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:12.500 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.517+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 shutdown_connections 2024-03-16T19:29:12.500 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:12.517+0000 7f58bfd77640 1 -- 172.21.15.172:0/877097602 wait complete. 2024-03-16T19:29:12.508 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-16T19:29:19.315 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:19.408 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.427+0000 7fad255ac640 1 Processor -- start 2024-03-16T19:29:19.408 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.427+0000 7fad255ac640 1 -- start start 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.427+0000 7fad255ac640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fad200cb290 0x7fad200c6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.427+0000 7fad255ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad200c6ca0 0x7fad200d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.427+0000 7fad255ac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad200c5d10 con 0x7fad20154e60 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.427+0000 7fad255ac640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad20155240 con 0x7fad200cb290 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.427+0000 7fad255ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2015e7d0 con 0x7fad200c6ca0 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.428+0000 7fad1e7fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fad200cb290 0x7fad200c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.428+0000 7fad1f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad200c6ca0 0x7fad200d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.428+0000 7fad1effd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fad20154e60 0x7fad2015e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:45998/0 (socket says 172.21.15.172:45998) 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.428+0000 7fad1e7fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fad200cb290 0x7fad200c6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:47268/0 (socket says 172.21.15.172:47268) 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.428+0000 7fad1effd640 1 -- 172.21.15.172:0/390859842 learned_addr learned my addr 172.21.15.172:0/390859842 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.428+0000 7fad1e7fc640 1 -- 172.21.15.172:0/390859842 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad200c6ca0 msgr2=0x7fad200d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:19.411 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.428+0000 7fad1e7fc640 1 --2- 172.21.15.172:0/390859842 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad200c6ca0 0x7fad200d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.428+0000 7fad1e7fc640 1 -- 172.21.15.172:0/390859842 >> v1:172.21.15.174:6789/0 conn(0x7fad20154e60 legacy=0x7fad2015e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.428+0000 7fad1e7fc640 1 -- 172.21.15.172:0/390859842 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2015eab0 con 0x7fad200cb290 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad1e7fc640 1 --2- 172.21.15.172:0/390859842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad200cb290 0x7fad200c6760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad100097a0 tx=0x7fad1002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44f2967e91238963 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad1dffb640 1 -- 172.21.15.172:0/390859842 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10031830 con 0x7fad200cb290 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad1dffb640 1 -- 172.21.15.172:0/390859842 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad10031990 con 0x7fad200cb290 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad1dffb640 1 -- 172.21.15.172:0/390859842 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10031d20 con 0x7fad200cb290 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad255ac640 1 -- 172.21.15.172:0/390859842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad200cb290 msgr2=0x7fad200c6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad255ac640 1 --2- 172.21.15.172:0/390859842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad200cb290 0x7fad200c6760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad100097a0 tx=0x7fad1002b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad255ac640 1 -- 172.21.15.172:0/390859842 shutdown_connections 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad255ac640 1 --2- 172.21.15.172:0/390859842 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad200c6ca0 0x7fad200d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad255ac640 1 --2- 172.21.15.172:0/390859842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad200cb290 0x7fad200c6760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad255ac640 1 -- 172.21.15.172:0/390859842 >> 172.21.15.172:0/390859842 conn(0x7fad2006bbd0 msgr2=0x7fad2006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad255ac640 1 -- 172.21.15.172:0/390859842 shutdown_connections 2024-03-16T19:29:19.412 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.429+0000 7fad255ac640 1 -- 172.21.15.172:0/390859842 wait complete. 2024-03-16T19:29:19.415 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.431+0000 7fad255ac640 1 Processor -- start 2024-03-16T19:29:19.415 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.431+0000 7fad255ac640 1 -- start start 2024-03-16T19:29:19.415 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.431+0000 7fad255ac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad200c6ca0 0x7fad20157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.431+0000 7fad255ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad200cb290 0x7fad20155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.431+0000 7fad255ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad20154e60 0x7fad2015bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.431+0000 7fad255ac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad20156410 con 0x7fad200c6ca0 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.431+0000 7fad255ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad201f1250 con 0x7fad200cb290 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.431+0000 7fad255ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad201f04d0 con 0x7fad20154e60 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1effd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad200c6ca0 0x7fad20157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1effd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad200c6ca0 0x7fad20157800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52166/0 (socket says 172.21.15.172:52166) 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1effd640 1 -- 172.21.15.172:0/1232356618 learned_addr learned my addr 172.21.15.172:0/1232356618 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1e7fc640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad200cb290 0x7fad20155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1f7fe640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad20154e60 0x7fad2015bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1effd640 1 -- 172.21.15.172:0/1232356618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad20154e60 msgr2=0x7fad2015bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1effd640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad20154e60 0x7fad2015bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.416 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1effd640 1 -- 172.21.15.172:0/1232356618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad200cb290 msgr2=0x7fad20155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1effd640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad200cb290 0x7fad20155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1effd640 1 -- 172.21.15.172:0/1232356618 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad10009400 con 0x7fad200c6ca0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad1effd640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad200c6ca0 0x7fad20157800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad1400c3c0 tx=0x7fad1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad077fe640 1 -- 172.21.15.172:0/1232356618 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14005ce0 con 0x7fad200c6ca0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad077fe640 1 -- 172.21.15.172:0/1232356618 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad14005e40 con 0x7fad200c6ca0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad077fe640 1 -- 172.21.15.172:0/1232356618 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad140072d0 con 0x7fad200c6ca0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad20155ca0 con 0x7fad200c6ca0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.432+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad201f0a50 con 0x7fad200c6ca0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.433+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2004f2f0 con 0x7fad200c6ca0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.434+0000 7fad077fe640 1 -- 172.21.15.172:0/1232356618 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fad140074f0 con 0x7fad200c6ca0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.434+0000 7fad077fe640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad0c06b3b0 0x7fad0c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.434+0000 7fad1e7fc640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad0c06b3b0 0x7fad0c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.434+0000 7fad077fe640 1 -- 172.21.15.172:0/1232356618 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad14088e40 con 0x7fad200c6ca0 2024-03-16T19:29:19.417 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.435+0000 7fad1e7fc640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad0c06b3b0 0x7fad0c06d870 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fad10036000 tx=0x7fad1003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:19.420 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.439+0000 7fad077fe640 1 -- 172.21.15.172:0/1232356618 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad14053e10 con 0x7fad200c6ca0 2024-03-16T19:29:19.592 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.611+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad20068af0 con 0x7fad200c6ca0 2024-03-16T19:29:19.593 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.612+0000 7fad077fe640 1 -- 172.21.15.172:0/1232356618 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12908 (secure 0 0 0) 0x7fad14017070 con 0x7fad200c6ca0 2024-03-16T19:29:19.593 INFO:teuthology.misc.health.smithi172.stdout: 2024-03-16T19:29:19.593 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":17,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:17.305669+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:19.595 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad0c06b3b0 msgr2=0x7fad0c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.595 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad0c06b3b0 0x7fad0c06d870 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fad10036000 tx=0x7fad1003c040 comp rx=0 tx=0).stop 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad200c6ca0 msgr2=0x7fad20157800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad200c6ca0 0x7fad20157800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad1400c3c0 tx=0x7fad1400c880 comp rx=0 tx=0).stop 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 shutdown_connections 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad0c06b3b0 0x7fad0c06d870 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad20154e60 0x7fad2015bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad200cb290 0x7fad20155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 --2- 172.21.15.172:0/1232356618 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad200c6ca0 0x7fad20157800 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 >> 172.21.15.172:0/1232356618 conn(0x7fad2006bbd0 msgr2=0x7fad200c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 shutdown_connections 2024-03-16T19:29:19.596 INFO:teuthology.misc.health.smithi172.stderr:2024-03-16T19:29:19.614+0000 7fad255ac640 1 -- 172.21.15.172:0/1232356618 wait complete. 2024-03-16T19:29:19.605 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-16T19:29:19.605 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-16T19:29:19.605 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-16T19:29:19.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-16T19:29:19.608 INFO:tasks.ceph.mds.a:Started 2024-03-16T19:29:19.608 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-16T19:29:19.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-16T19:29:19.610 INFO:tasks.ceph.mds.c:Started 2024-03-16T19:29:19.610 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-16T19:29:19.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-03-16T19:29:19.612 INFO:tasks.ceph.mds.e:Started 2024-03-16T19:29:19.612 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-16T19:29:19.612 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-16T19:29:19.615 INFO:tasks.ceph.mds.b:Started 2024-03-16T19:29:19.615 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-16T19:29:19.616 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-16T19:29:19.618 INFO:tasks.ceph.mds.d:Started 2024-03-16T19:29:19.618 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-16T19:29:19.618 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-03-16T19:29:19.620 INFO:tasks.ceph.mds.f:Started 2024-03-16T19:29:19.620 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-16T19:29:19.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:19.686 INFO:tasks.ceph.mds.c.smithi172.stdout:starting mds.c at 2024-03-16T19:29:19.686 INFO:tasks.ceph.mds.a.smithi172.stdout:starting mds.a at 2024-03-16T19:29:19.687 INFO:tasks.ceph.mds.e.smithi172.stdout:starting mds.e at 2024-03-16T19:29:19.704 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2024-03-16T19:29:19.705 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-03-16T19:29:19.705 INFO:tasks.ceph.mds.f.smithi174.stdout:starting mds.f at 2024-03-16T19:29:19.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.747+0000 7f28fc0d4640 1 Processor -- start 2024-03-16T19:29:19.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.748+0000 7f28fc0d4640 1 -- start start 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.748+0000 7f28fc0d4640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f28f40c3760 0x7f28f40c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.748+0000 7f28fc0d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40720b0 0x7f28f4161e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.748+0000 7f28fc0d4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f415b200 con 0x7f28f40afbb0 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.748+0000 7f28fc0d4640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f40ca730 con 0x7f28f40c3760 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.748+0000 7f28fc0d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40c4080 con 0x7f28f40720b0 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.748+0000 7f28f9e70640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f28f40c3760 0x7f28f40c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.748+0000 7f28f9e70640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f28f40c3760 0x7f28f40c3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52220/0 (socket says 172.21.15.172:52220) 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.748+0000 7f28f9e70640 1 -- 172.21.15.172:0/3299608027 learned_addr learned my addr 172.21.15.172:0/3299608027 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.749+0000 7f28fa671640 1 --2- 172.21.15.172:0/3299608027 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40720b0 0x7f28f4161e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.749+0000 7f28f9e70640 1 -- 172.21.15.172:0/3299608027 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40720b0 msgr2=0x7f28f4161e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:19.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.749+0000 7f28f9e70640 1 --2- 172.21.15.172:0/3299608027 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40720b0 0x7f28f4161e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.749+0000 7f28f9e70640 1 -- 172.21.15.172:0/3299608027 >> v1:172.21.15.174:6789/0 conn(0x7f28f40afbb0 legacy=0x7f28f415aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.749+0000 7f28f9e70640 1 -- 172.21.15.172:0/3299608027 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f4072bb0 con 0x7f28f40c3760 2024-03-16T19:29:19.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.749+0000 7f28f9e70640 1 --2- 172.21.15.172:0/3299608027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40c3760 0x7f28f40c3b40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009870 tx=0x7f28e402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c410912aa7160f63 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:19.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28f8e6e640 1 -- 172.21.15.172:0/3299608027 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e402f840 con 0x7f28f40c3760 2024-03-16T19:29:19.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28f8e6e640 1 -- 172.21.15.172:0/3299608027 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e402f9a0 con 0x7f28f40c3760 2024-03-16T19:29:19.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28f8e6e640 1 -- 172.21.15.172:0/3299608027 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e402fd30 con 0x7f28f40c3760 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28fc0d4640 1 -- 172.21.15.172:0/3299608027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40c3760 msgr2=0x7f28f40c3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28fc0d4640 1 --2- 172.21.15.172:0/3299608027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40c3760 0x7f28f40c3b40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009870 tx=0x7f28e402ba20 comp rx=0 tx=0).stop 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28fc0d4640 1 -- 172.21.15.172:0/3299608027 shutdown_connections 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28fc0d4640 1 --2- 172.21.15.172:0/3299608027 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40720b0 0x7f28f4161e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28fc0d4640 1 --2- 172.21.15.172:0/3299608027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40c3760 0x7f28f40c3b40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28fc0d4640 1 -- 172.21.15.172:0/3299608027 >> 172.21.15.172:0/3299608027 conn(0x7f28f406f6f0 msgr2=0x7f28f406fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28fc0d4640 1 -- 172.21.15.172:0/3299608027 shutdown_connections 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.750+0000 7f28fc0d4640 1 -- 172.21.15.172:0/3299608027 wait complete. 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.751+0000 7f28fc0d4640 1 Processor -- start 2024-03-16T19:29:19.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.751+0000 7f28fc0d4640 1 -- start start 2024-03-16T19:29:19.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28fc0d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f40720b0 0x7f28f40c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28fc0d4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40afbb0 0x7f28f40b0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28fc0d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40c3760 0x7f28f40cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28fc0d4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40ce0c0 con 0x7f28f40afbb0 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28fc0d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40c46e0 con 0x7f28f40720b0 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28fc0d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40c49c0 con 0x7f28f40c3760 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28f966f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40afbb0 0x7f28f40b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28f966f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40afbb0 0x7f28f40b0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52232/0 (socket says 172.21.15.172:52232) 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28f966f640 1 -- 172.21.15.172:0/111974901 learned_addr learned my addr 172.21.15.172:0/111974901 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28f9e70640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f40720b0 0x7f28f40c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28fa671640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40c3760 0x7f28f40cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28f966f640 1 -- 172.21.15.172:0/111974901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40c3760 msgr2=0x7f28f40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28f966f640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40c3760 0x7f28f40cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28f966f640 1 -- 172.21.15.172:0/111974901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f40720b0 msgr2=0x7f28f40c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28f966f640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f40720b0 0x7f28f40c5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.752+0000 7f28f966f640 1 -- 172.21.15.172:0/111974901 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e40094d0 con 0x7f28f40afbb0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.753+0000 7f28f966f640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40afbb0 0x7f28f40b0460 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f28f000c450 tx=0x7f28f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.753+0000 7f28ea7fc640 1 -- 172.21.15.172:0/111974901 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f00052d0 con 0x7f28f40afbb0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.753+0000 7f28ea7fc640 1 -- 172.21.15.172:0/111974901 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f0005430 con 0x7f28f40afbb0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.753+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f40c4ca0 con 0x7f28f40afbb0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.753+0000 7f28ea7fc640 1 -- 172.21.15.172:0/111974901 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0012870 con 0x7f28f40afbb0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.753+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f40c5260 con 0x7f28f40afbb0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.754+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f40688d0 con 0x7f28f40afbb0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.754+0000 7f28ea7fc640 1 -- 172.21.15.172:0/111974901 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f28f00129d0 con 0x7f28f40afbb0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.754+0000 7f28ea7fc640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28d806b3b0 0x7f28d806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.755+0000 7f28ea7fc640 1 -- 172.21.15.172:0/111974901 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f28f0089320 con 0x7f28f40afbb0 2024-03-16T19:29:19.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.755+0000 7f28f9e70640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28d806b3b0 0x7f28d806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:19.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.755+0000 7f28f9e70640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28d806b3b0 0x7f28d806d870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f28e4036000 tx=0x7f28e4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:19.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.759+0000 7f28ea7fc640 1 -- 172.21.15.172:0/111974901 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28f00542f0 con 0x7f28f40afbb0 2024-03-16T19:29:19.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.931+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28f4071e70 con 0x7f28f40afbb0 2024-03-16T19:29:19.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.932+0000 7f28ea7fc640 1 -- 172.21.15.172:0/111974901 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12908 (secure 0 0 0) 0x7f28f00579b0 con 0x7f28f40afbb0 2024-03-16T19:29:19.913 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:19.913 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":17,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:17.305669+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:19.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28d806b3b0 msgr2=0x7f28d806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28d806b3b0 0x7f28d806d870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f28e4036000 tx=0x7f28e4008030 comp rx=0 tx=0).stop 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40afbb0 msgr2=0x7f28f40b0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40afbb0 0x7f28f40b0460 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f28f000c450 tx=0x7f28f000c910 comp rx=0 tx=0).stop 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 shutdown_connections 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28d806b3b0 0x7f28d806d870 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28f40c3760 0x7f28f40cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28f40afbb0 0x7f28f40b0460 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 --2- 172.21.15.172:0/111974901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f40720b0 0x7f28f40c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 >> 172.21.15.172:0/111974901 conn(0x7f28f406f6f0 msgr2=0x7f28f4150290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 shutdown_connections 2024-03-16T19:29:19.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:19.934+0000 7f28fc0d4640 1 -- 172.21.15.172:0/111974901 wait complete. 2024-03-16T19:29:19.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:29:14.242742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T19:29:19.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:29:20.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.049+0000 7f21bf2c3640 1 Processor -- start 2024-03-16T19:29:20.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.049+0000 7f21bf2c3640 1 -- start start 2024-03-16T19:29:20.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bf2c3640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21b8154a90 0x7f21b80c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bf2c3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f21b8154e70 0x7f21b806b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bf2c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b80cc280 0x7f21b806be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bf2c3640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b815df50 con 0x7f21b8154a90 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bf2c3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b815e1f0 con 0x7f21b8154e70 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bf2c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b815e4d0 con 0x7f21b80cc280 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bd05f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21b8154a90 0x7f21b80c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bd05f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f21b8154a90 0x7f21b80c50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52244/0 (socket says 172.21.15.172:52244) 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bd05f640 1 -- 172.21.15.172:0/4293725958 learned_addr learned my addr 172.21.15.172:0/4293725958 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bc85e640 1 --2- 172.21.15.172:0/4293725958 >> v2:172.21.15.174:3300/0 conn(0x7f21b8154e70 0x7f21b806b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bd860640 1 --2- 172.21.15.172:0/4293725958 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b80cc280 0x7f21b806be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bd05f640 1 -- 172.21.15.172:0/4293725958 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b80cc280 msgr2=0x7f21b806be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bd05f640 1 --2- 172.21.15.172:0/4293725958 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b80cc280 0x7f21b806be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bd05f640 1 -- 172.21.15.172:0/4293725958 >> v2:172.21.15.174:3300/0 conn(0x7f21b8154e70 msgr2=0x7f21b806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bd05f640 1 --2- 172.21.15.172:0/4293725958 >> v2:172.21.15.174:3300/0 conn(0x7f21b8154e70 0x7f21b806b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.050+0000 7f21bd05f640 1 -- 172.21.15.172:0/4293725958 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b815e7b0 con 0x7f21b8154a90 2024-03-16T19:29:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.051+0000 7f21bd05f640 1 --2- 172.21.15.172:0/4293725958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b8154a90 0x7f21b80c50b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f21b808db80 tx=0x7f21a802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b68cf5d2ea09bd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.051+0000 7f21affff640 1 -- 172.21.15.172:0/4293725958 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8030860 con 0x7f21b8154a90 2024-03-16T19:29:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.051+0000 7f21affff640 1 -- 172.21.15.172:0/4293725958 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a80309c0 con 0x7f21b8154a90 2024-03-16T19:29:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.051+0000 7f21affff640 1 -- 172.21.15.172:0/4293725958 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8030d50 con 0x7f21b8154a90 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.052+0000 7f21bf2c3640 1 -- 172.21.15.172:0/4293725958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b8154a90 msgr2=0x7f21b80c50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.052+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/4293725958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b8154a90 0x7f21b80c50b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f21b808db80 tx=0x7f21a802b9a0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.052+0000 7f21bf2c3640 1 -- 172.21.15.172:0/4293725958 shutdown_connections 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.052+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/4293725958 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b80cc280 0x7f21b806be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.052+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/4293725958 >> v2:172.21.15.174:3300/0 conn(0x7f21b8154e70 0x7f21b806b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.052+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/4293725958 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b8154a90 0x7f21b80c50b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.052+0000 7f21bf2c3640 1 -- 172.21.15.172:0/4293725958 >> 172.21.15.172:0/4293725958 conn(0x7f21b8067250 msgr2=0x7f21b80d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.052+0000 7f21bf2c3640 1 -- 172.21.15.172:0/4293725958 shutdown_connections 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.052+0000 7f21bf2c3640 1 -- 172.21.15.172:0/4293725958 wait complete. 2024-03-16T19:29:20.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.053+0000 7f21bf2c3640 1 Processor -- start 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.053+0000 7f21bf2c3640 1 -- start start 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bf2c3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b80cc280 0x7f21b80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bf2c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b8154a90 0x7f21b8146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bf2c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b8154e70 0x7f21b8153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bf2c3640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b8151e50 con 0x7f21b80cc280 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bf2c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80b5750 con 0x7f21b8154e70 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bf2c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b81f4390 con 0x7f21b8154a90 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bd05f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b80cc280 0x7f21b80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bd05f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b80cc280 0x7f21b80b4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52246/0 (socket says 172.21.15.172:52246) 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bd05f640 1 -- 172.21.15.172:0/1288724619 learned_addr learned my addr 172.21.15.172:0/1288724619 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bc85e640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b8154a90 0x7f21b8146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bd860640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b8154e70 0x7f21b8153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bd05f640 1 -- 172.21.15.172:0/1288724619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b8154a90 msgr2=0x7f21b8146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bd05f640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b8154a90 0x7f21b8146180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bd05f640 1 -- 172.21.15.172:0/1288724619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b8154e70 msgr2=0x7f21b8153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bd05f640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b8154e70 0x7f21b8153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.054+0000 7f21bd05f640 1 -- 172.21.15.172:0/1288724619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80094d0 con 0x7f21b80cc280 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.055+0000 7f21bd05f640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b80cc280 0x7f21b80b4dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21a8000c00 tx=0x7f21a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.055+0000 7f21adffb640 1 -- 172.21.15.172:0/1288724619 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8039d60 con 0x7f21b80cc280 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.055+0000 7f21adffb640 1 -- 172.21.15.172:0/1288724619 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a8044480 con 0x7f21b80cc280 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.055+0000 7f21adffb640 1 -- 172.21.15.172:0/1288724619 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8044830 con 0x7f21b80cc280 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.055+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b81f4820 con 0x7f21b80cc280 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.055+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b81f3a60 con 0x7f21b80cc280 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.056+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b80681c0 con 0x7f21b80cc280 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.056+0000 7f21adffb640 1 -- 172.21.15.172:0/1288724619 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f21a804d3b0 con 0x7f21b80cc280 2024-03-16T19:29:20.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.056+0000 7f21adffb640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f219c06b3b0 0x7f219c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.057+0000 7f21adffb640 1 -- 172.21.15.172:0/1288724619 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f21a803e070 con 0x7f21b80cc280 2024-03-16T19:29:20.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.057+0000 7f21bc85e640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f219c06b3b0 0x7f219c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.057+0000 7f21bc85e640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f219c06b3b0 0x7f219c06d870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21b80b2cc0 tx=0x7f21b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:20.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.061+0000 7f21adffb640 1 -- 172.21.15.172:0/1288724619 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21a8077570 con 0x7f21b80cc280 2024-03-16T19:29:20.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.232+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21b80c5300 con 0x7f21b80cc280 2024-03-16T19:29:20.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.232+0000 7f21adffb640 1 -- 172.21.15.172:0/1288724619 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21a8049020 con 0x7f21b80cc280 2024-03-16T19:29:20.214 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 1 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f219c06b3b0 msgr2=0x7f219c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f219c06b3b0 0x7f219c06d870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21b80b2cc0 tx=0x7f21b4008040 comp rx=0 tx=0).stop 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b80cc280 msgr2=0x7f21b80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b80cc280 0x7f21b80b4dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21a8000c00 tx=0x7f21a8004730 comp rx=0 tx=0).stop 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 shutdown_connections 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b8154e70 0x7f21b8153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f219c06b3b0 0x7f219c06d870 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b8154a90 0x7f21b8146180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 --2- 172.21.15.172:0/1288724619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b80cc280 0x7f21b80b4dc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 >> 172.21.15.172:0/1288724619 conn(0x7f21b8067250 msgr2=0x7f21b80cad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 shutdown_connections 2024-03-16T19:29:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.234+0000 7f21bf2c3640 1 -- 172.21.15.172:0/1288724619 wait complete. 2024-03-16T19:29:20.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:29:20.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.345+0000 7f7cdcffd640 1 Processor -- start 2024-03-16T19:29:20.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.345+0000 7f7cdcffd640 1 -- start start 2024-03-16T19:29:20.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cdcffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7cd8154c90 0x7f7cd815e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cdcffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 0x7f7cd80cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cdcffd640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd815b880 con 0x7f7cd815ea30 2024-03-16T19:29:20.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cdcffd640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd80c6ee0 con 0x7f7cd8154c90 2024-03-16T19:29:20.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cdcffd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8155070 con 0x7f7cd80cc780 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd659c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7cd8154c90 0x7f7cd815e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd659c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7cd8154c90 0x7f7cd815e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52260/0 (socket says 172.21.15.172:52260) 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd6d9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7cd815ea30 0x7f7cd806b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40154/0 (socket says 172.21.15.172:40154) 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd659c640 1 -- 172.21.15.172:0/1138989627 learned_addr learned my addr 172.21.15.172:0/1138989627 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd5d9b640 1 --2- 172.21.15.172:0/1138989627 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 0x7f7cd80cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd659c640 1 -- 172.21.15.172:0/1138989627 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 msgr2=0x7f7cd80cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd659c640 1 --2- 172.21.15.172:0/1138989627 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 0x7f7cd80cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd659c640 1 -- 172.21.15.172:0/1138989627 >> v1:172.21.15.172:6789/0 conn(0x7f7cd815ea30 legacy=0x7f7cd806b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd659c640 1 -- 172.21.15.172:0/1138989627 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd815ad00 con 0x7f7cd8154c90 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.346+0000 7f7cd559a640 1 -- 172.21.15.172:0/1138989627 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171930316 0 0) 0x7f7cd0001110 con 0x7f7cd815ea30 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.347+0000 7f7cd659c640 1 --2- 172.21.15.172:0/1138989627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 0x7f7cd815e4f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009e50 tx=0x7f7cc802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40cefce8a7415dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.347+0000 7f7cd559a640 1 -- 172.21.15.172:0/1138989627 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc802cb00 con 0x7f7cd8154c90 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.347+0000 7f7cd559a640 1 -- 172.21.15.172:0/1138989627 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc802cc60 con 0x7f7cd8154c90 2024-03-16T19:29:20.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.347+0000 7f7cd559a640 1 -- 172.21.15.172:0/1138989627 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc80343f0 con 0x7f7cd8154c90 2024-03-16T19:29:20.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.348+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1138989627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 msgr2=0x7f7cd815e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.348+0000 7f7cdcffd640 1 --2- 172.21.15.172:0/1138989627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 0x7f7cd815e4f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009e50 tx=0x7f7cc802bf60 comp rx=0 tx=0).stop 2024-03-16T19:29:20.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.348+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1138989627 shutdown_connections 2024-03-16T19:29:20.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.348+0000 7f7cdcffd640 1 --2- 172.21.15.172:0/1138989627 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 0x7f7cd80cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.348+0000 7f7cdcffd640 1 --2- 172.21.15.172:0/1138989627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 0x7f7cd815e4f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.348+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1138989627 >> 172.21.15.172:0/1138989627 conn(0x7f7cd8067040 msgr2=0x7f7cd8067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.348+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1138989627 shutdown_connections 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.348+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1138989627 wait complete. 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.349+0000 7f7cdcffd640 1 Processor -- start 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.349+0000 7f7cdcffd640 1 -- start start 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cdcffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 0x7f7cd815a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cdcffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 0x7f7cd81e3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cdcffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cd815ea30 0x7f7cd8156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cdcffd640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd81f4ae0 con 0x7f7cd8154c90 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cdcffd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8155230 con 0x7f7cd815ea30 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cdcffd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd81554f0 con 0x7f7cd80cc780 2024-03-16T19:29:20.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd5d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 0x7f7cd81e3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd5d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 0x7f7cd81e3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52264/0 (socket says 172.21.15.172:52264) 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd5d9b640 1 -- 172.21.15.172:0/1711326424 learned_addr learned my addr 172.21.15.172:0/1711326424 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd659c640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 0x7f7cd815a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd6d9d640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cd815ea30 0x7f7cd8156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd5d9b640 1 -- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 msgr2=0x7f7cd815a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd5d9b640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 0x7f7cd815a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd5d9b640 1 -- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cd815ea30 msgr2=0x7f7cd8156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd5d9b640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cd815ea30 0x7f7cd8156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd5d9b640 1 -- 172.21.15.172:0/1711326424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc8009b30 con 0x7f7cd8154c90 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd659c640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 0x7f7cd815a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd6d9d640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cd815ea30 0x7f7cd8156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.350+0000 7f7cd5d9b640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 0x7f7cd81e3c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c7f0 tx=0x7f7ccc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.351+0000 7f7cc6ffd640 1 -- 172.21.15.172:0/1711326424 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc004220 con 0x7f7cd8154c90 2024-03-16T19:29:20.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.351+0000 7f7cc6ffd640 1 -- 172.21.15.172:0/1711326424 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc0043a0 con 0x7f7cd8154c90 2024-03-16T19:29:20.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.351+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd81f4f70 con 0x7f7cd8154c90 2024-03-16T19:29:20.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.351+0000 7f7cc6ffd640 1 -- 172.21.15.172:0/1711326424 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc007860 con 0x7f7cd8154c90 2024-03-16T19:29:20.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.351+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd81f4490 con 0x7f7cd8154c90 2024-03-16T19:29:20.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.352+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd815e850 con 0x7f7cd8154c90 2024-03-16T19:29:20.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.352+0000 7f7cc6ffd640 1 -- 172.21.15.172:0/1711326424 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7ccc0079c0 con 0x7f7cd8154c90 2024-03-16T19:29:20.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.352+0000 7f7cc6ffd640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7cb806b3e0 0x7f7cb806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:20.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.352+0000 7f7cc6ffd640 1 -- 172.21.15.172:0/1711326424 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7ccc089620 con 0x7f7cd8154c90 2024-03-16T19:29:20.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.352+0000 7f7cd659c640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7cb806b3e0 0x7f7cb806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:20.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.353+0000 7f7cd659c640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7cb806b3e0 0x7f7cb806d8a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8035000 tx=0x7f7cc8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:20.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.357+0000 7f7cc6ffd640 1 -- 172.21.15.172:0/1711326424 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ccc0545d0 con 0x7f7cd8154c90 2024-03-16T19:29:20.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.562+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cd8067420 con 0x7f7cd8154c90 2024-03-16T19:29:20.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.562+0000 7f7cc6ffd640 1 -- 172.21.15.172:0/1711326424 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f7ccc01e020 con 0x7f7cd8154c90 2024-03-16T19:29:20.543 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:20.543 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/420882750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":420882750},{"type":"v1","addr":"172.21.15.172:6837","nonce":420882750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-16T19:29:20.543 INFO:teuthology.orchestra.run.smithi172.stderr:dumped fsmap epoch 2 2024-03-16T19:29:20.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7cb806b3e0 msgr2=0x7f7cb806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7cb806b3e0 0x7f7cb806d8a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8035000 tx=0x7f7cc8030000 comp rx=0 tx=0).stop 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 msgr2=0x7f7cd81e3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 0x7f7cd81e3c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c7f0 tx=0x7f7ccc00ccb0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 shutdown_connections 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cd815ea30 0x7f7cd8156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7cd8154c90 0x7f7cd81e3c90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7cb806b3e0 0x7f7cb806d8a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 --2- 172.21.15.172:0/1711326424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cd80cc780 0x7f7cd815a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 >> 172.21.15.172:0/1711326424 conn(0x7f7cd8067040 msgr2=0x7f7cd80c59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 shutdown_connections 2024-03-16T19:29:20.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:20.564+0000 7f7cdcffd640 1 -- 172.21.15.172:0/1711326424 wait complete. 2024-03-16T19:29:22.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:29:22.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.687+0000 7fc42736c640 1 Processor -- start 2024-03-16T19:29:22.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.688+0000 7fc42736c640 1 -- start start 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.688+0000 7fc42736c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc42006fe50 0x7fc42015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.688+0000 7fc42736c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42015ba20 con 0x7fc420158180 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.688+0000 7fc42736c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc420158560 con 0x7fc4200ce200 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.688+0000 7fc42736c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42006cf90 con 0x7fc42006fe50 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.688+0000 7fc424907640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc42006fe50 0x7fc42015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.688+0000 7fc425108640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc420158180 0x7fc4200c9210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40166/0 (socket says 172.21.15.172:40166) 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.688+0000 7fc425108640 1 -- 172.21.15.172:0/2165186512 learned_addr learned my addr 172.21.15.172:0/2165186512 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.689+0000 7fc424907640 1 -- 172.21.15.172:0/2165186512 >> v1:172.21.15.174:6789/0 conn(0x7fc4200ce200 legacy=0x7fc420165300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.689+0000 7fc424907640 1 -- 172.21.15.172:0/2165186512 >> v1:172.21.15.172:6789/0 conn(0x7fc420158180 legacy=0x7fc4200c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.689+0000 7fc413fff640 1 -- 172.21.15.172:0/2165186512 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081305123 0 0) 0x7fc41c001110 con 0x7fc420158180 2024-03-16T19:29:22.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.689+0000 7fc424907640 1 -- 172.21.15.172:0/2165186512 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4200ca0a0 con 0x7fc42006fe50 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.689+0000 7fc424907640 1 --2- 172.21.15.172:0/2165186512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc42006fe50 0x7fc42015b4e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc4140097a0 tx=0x7fc41402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d04a4170376f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc413fff640 1 -- 172.21.15.172:0/2165186512 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41402f810 con 0x7fc42006fe50 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc413fff640 1 -- 172.21.15.172:0/2165186512 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41402f970 con 0x7fc42006fe50 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc413fff640 1 -- 172.21.15.172:0/2165186512 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41402fd00 con 0x7fc42006fe50 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc42736c640 1 -- 172.21.15.172:0/2165186512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc42006fe50 msgr2=0x7fc42015b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc42736c640 1 --2- 172.21.15.172:0/2165186512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc42006fe50 0x7fc42015b4e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc4140097a0 tx=0x7fc41402b9f0 comp rx=0 tx=0).stop 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc42736c640 1 -- 172.21.15.172:0/2165186512 shutdown_connections 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc42736c640 1 --2- 172.21.15.172:0/2165186512 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc42006fe50 0x7fc42015b4e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc42736c640 1 -- 172.21.15.172:0/2165186512 >> 172.21.15.172:0/2165186512 conn(0x7fc420066c20 msgr2=0x7fc420067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc42736c640 1 -- 172.21.15.172:0/2165186512 shutdown_connections 2024-03-16T19:29:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.690+0000 7fc42736c640 1 -- 172.21.15.172:0/2165186512 wait complete. 2024-03-16T19:29:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.691+0000 7fc42736c640 1 Processor -- start 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc42736c640 1 -- start start 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc42736c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc42006fe50 0x7fc4201f1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc42736c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc4200ce200 0x7fc4200b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc42736c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc420158180 0x7fc4201f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc42736c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42006c5b0 con 0x7fc420158180 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc42736c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4201f1bc0 con 0x7fc42006fe50 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc42736c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc420151560 con 0x7fc4200ce200 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc425909640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc420158180 0x7fc4201f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc425909640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc420158180 0x7fc4201f2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52280/0 (socket says 172.21.15.172:52280) 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc425909640 1 -- 172.21.15.172:0/1456037614 learned_addr learned my addr 172.21.15.172:0/1456037614 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc425108640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc42006fe50 0x7fc4201f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.692+0000 7fc424907640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc4200ce200 0x7fc4200b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc425909640 1 -- 172.21.15.172:0/1456037614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc4200ce200 msgr2=0x7fc4200b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc425909640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc4200ce200 0x7fc4200b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc425909640 1 -- 172.21.15.172:0/1456037614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc42006fe50 msgr2=0x7fc4201f1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc425909640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc42006fe50 0x7fc4201f1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc425909640 1 -- 172.21.15.172:0/1456037614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414009400 con 0x7fc420158180 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc424907640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc4200ce200 0x7fc4200b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc425909640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc420158180 0x7fc4201f2070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc41800c7a0 tx=0x7fc41800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc411ffb640 1 -- 172.21.15.172:0/1456037614 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418003b70 con 0x7fc420158180 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc411ffb640 1 -- 172.21.15.172:0/1456037614 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418003cf0 con 0x7fc420158180 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc420152570 con 0x7fc420158180 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc411ffb640 1 -- 172.21.15.172:0/1456037614 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418003ea0 con 0x7fc420158180 2024-03-16T19:29:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.693+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc420153220 con 0x7fc420158180 2024-03-16T19:29:22.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.694+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4200681c0 con 0x7fc420158180 2024-03-16T19:29:22.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.694+0000 7fc411ffb640 1 -- 172.21.15.172:0/1456037614 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc41800aa90 con 0x7fc420158180 2024-03-16T19:29:22.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.695+0000 7fc411ffb640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc40406b3e0 0x7fc40406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:22.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.695+0000 7fc411ffb640 1 -- 172.21.15.172:0/1456037614 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc4180896e0 con 0x7fc420158180 2024-03-16T19:29:22.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.695+0000 7fc425108640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc40406b3e0 0x7fc40406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:22.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.695+0000 7fc425108640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc40406b3e0 0x7fc40406d8a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc41c000f60 tx=0x7fc41c007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:22.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.699+0000 7fc411ffb640 1 -- 172.21.15.172:0/1456037614 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc418054600 con 0x7fc420158180 2024-03-16T19:29:22.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.907+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc420156500 con 0x7fc420158180 2024-03-16T19:29:22.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.907+0000 7fc411ffb640 1 -- 172.21.15.172:0/1456037614 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5283 (secure 0 0 0) 0x7fc418057cc0 con 0x7fc420158180 2024-03-16T19:29:22.889 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:22.889 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1028682606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1028682606},{"type":"v1","addr":"172.21.15.174:6833","nonce":1028682606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6836/902696086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":902696086},{"type":"v1","addr":"172.21.15.174:6836","nonce":902696086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/864851052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":864851052},{"type":"v1","addr":"172.21.15.172:6836","nonce":864851052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/420882750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":420882750},{"type":"v1","addr":"172.21.15.172:6837","nonce":420882750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/2853574742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":2853574742},{"type":"v1","addr":"172.21.15.172:6839","nonce":2853574742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4297,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2442775894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6835","nonce":2442775894},{"type":"v1","addr":"172.21.15.174:6837","nonce":2442775894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-16T19:29:22.889 INFO:teuthology.orchestra.run.smithi172.stderr:dumped fsmap epoch 3 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc40406b3e0 msgr2=0x7fc40406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc40406b3e0 0x7fc40406d8a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc41c000f60 tx=0x7fc41c007e40 comp rx=0 tx=0).stop 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc420158180 msgr2=0x7fc4201f2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc420158180 0x7fc4201f2070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc41800c7a0 tx=0x7fc41800cc60 comp rx=0 tx=0).stop 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 shutdown_connections 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc420158180 0x7fc4201f2070 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc40406b3e0 0x7fc40406d8a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc4200ce200 0x7fc4200b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 --2- 172.21.15.172:0/1456037614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc42006fe50 0x7fc4201f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 >> 172.21.15.172:0/1456037614 conn(0x7fc420066c20 msgr2=0x7fc420067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:22.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.909+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 shutdown_connections 2024-03-16T19:29:22.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:22.910+0000 7fc42736c640 1 -- 172.21.15.172:0/1456037614 wait complete. 2024-03-16T19:29:22.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:23.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.021+0000 7f7caeceb640 1 Processor -- start 2024-03-16T19:29:23.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.021+0000 7f7caeceb640 1 -- start start 2024-03-16T19:29:23.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caeceb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ca814f490 0x7f7ca8066c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caeceb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7ca80cb0f0 0x7f7ca80c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caeceb640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca80c8d80 con 0x7f7ca80ca590 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caeceb640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca80c75d0 con 0x7f7ca814f490 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caeceb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca80cc1e0 con 0x7f7ca80cb0f0 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caca87640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ca814f490 0x7f7ca8066c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caca87640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7ca814f490 0x7f7ca8066c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52290/0 (socket says 172.21.15.172:52290) 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caca87640 1 -- 172.21.15.172:0/3755126264 learned_addr learned my addr 172.21.15.172:0/3755126264 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7ca7fff640 1 --2- 172.21.15.172:0/3755126264 >> v2:172.21.15.174:3300/0 conn(0x7f7ca80cb0f0 0x7f7ca80c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caca87640 1 -- 172.21.15.172:0/3755126264 >> v2:172.21.15.174:3300/0 conn(0x7f7ca80cb0f0 msgr2=0x7f7ca80c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caca87640 1 --2- 172.21.15.172:0/3755126264 >> v2:172.21.15.174:3300/0 conn(0x7f7ca80cb0f0 0x7f7ca80c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caca87640 1 -- 172.21.15.172:0/3755126264 >> v1:172.21.15.172:6789/0 conn(0x7f7ca80ca590 legacy=0x7f7ca8158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7caca87640 1 -- 172.21.15.172:0/3755126264 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca806b9f0 con 0x7f7ca814f490 2024-03-16T19:29:23.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.022+0000 7f7ca77fe640 1 -- 172.21.15.172:0/3755126264 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603085692 0 0) 0x7f7ca80c8d80 con 0x7f7ca80ca590 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.023+0000 7f7caca87640 1 --2- 172.21.15.172:0/3755126264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca814f490 0x7f7ca8066c20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80c91f0 tx=0x7f7c9c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7518442ac3e775c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.023+0000 7f7ca77fe640 1 -- 172.21.15.172:0/3755126264 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c030860 con 0x7f7ca814f490 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.023+0000 7f7ca77fe640 1 -- 172.21.15.172:0/3755126264 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c0309c0 con 0x7f7ca814f490 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.023+0000 7f7ca77fe640 1 -- 172.21.15.172:0/3755126264 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c030d50 con 0x7f7ca814f490 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.023+0000 7f7caeceb640 1 -- 172.21.15.172:0/3755126264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca814f490 msgr2=0x7f7ca8066c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.024+0000 7f7caeceb640 1 --2- 172.21.15.172:0/3755126264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca814f490 0x7f7ca8066c20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80c91f0 tx=0x7f7c9c02b9a0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.024+0000 7f7caeceb640 1 -- 172.21.15.172:0/3755126264 shutdown_connections 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.024+0000 7f7caeceb640 1 --2- 172.21.15.172:0/3755126264 >> v2:172.21.15.174:3300/0 conn(0x7f7ca80cb0f0 0x7f7ca80c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.024+0000 7f7caeceb640 1 --2- 172.21.15.172:0/3755126264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca814f490 0x7f7ca8066c20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.024+0000 7f7caeceb640 1 -- 172.21.15.172:0/3755126264 >> 172.21.15.172:0/3755126264 conn(0x7f7ca806aed0 msgr2=0x7f7ca806b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:23.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.024+0000 7f7caeceb640 1 -- 172.21.15.172:0/3755126264 shutdown_connections 2024-03-16T19:29:23.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.024+0000 7f7caeceb640 1 -- 172.21.15.172:0/3755126264 wait complete. 2024-03-16T19:29:23.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.025+0000 7f7caeceb640 1 Processor -- start 2024-03-16T19:29:23.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.025+0000 7f7caeceb640 1 -- start start 2024-03-16T19:29:23.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caeceb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca80ca590 0x7f7ca81518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caeceb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ca80cb0f0 0x7f7ca81ee660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caeceb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ca814f490 0x7f7ca8154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caeceb640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8151490 con 0x7f7ca80ca590 2024-03-16T19:29:23.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caeceb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca81eeba0 con 0x7f7ca814f490 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caeceb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8154dd0 con 0x7f7ca80cb0f0 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caca87640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca80ca590 0x7f7ca81518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caca87640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca80ca590 0x7f7ca81518b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52298/0 (socket says 172.21.15.172:52298) 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caca87640 1 -- 172.21.15.172:0/1586214424 learned_addr learned my addr 172.21.15.172:0/1586214424 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7cad288640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ca814f490 0x7f7ca8154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7ca7fff640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ca80cb0f0 0x7f7ca81ee660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caca87640 1 -- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ca80cb0f0 msgr2=0x7f7ca81ee660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caca87640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ca80cb0f0 0x7f7ca81ee660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caca87640 1 -- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ca814f490 msgr2=0x7f7ca8154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caca87640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ca814f490 0x7f7ca8154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caca87640 1 -- 172.21.15.172:0/1586214424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c0094d0 con 0x7f7ca80ca590 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7ca7fff640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ca80cb0f0 0x7f7ca81ee660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:23.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7cad288640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ca814f490 0x7f7ca8154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caca87640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca80ca590 0x7f7ca81518b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c000c00 tx=0x7f7c9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7ca57fa640 1 -- 172.21.15.172:0/1586214424 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c039d60 con 0x7f7ca80ca590 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7ca57fa640 1 -- 172.21.15.172:0/1586214424 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c044480 con 0x7f7ca80ca590 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca8152b00 con 0x7f7ca80ca590 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7ca57fa640 1 -- 172.21.15.172:0/1586214424 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c044830 con 0x7f7ca80ca590 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.026+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca8152540 con 0x7f7ca80ca590 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.027+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca8074f10 con 0x7f7ca80ca590 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.028+0000 7f7ca57fa640 1 -- 172.21.15.172:0/1586214424 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7c9c04d3b0 con 0x7f7ca80ca590 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.028+0000 7f7ca57fa640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7c8c06b3e0 0x7f7c8c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.028+0000 7f7ca57fa640 1 -- 172.21.15.172:0/1586214424 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7c9c03e070 con 0x7f7ca80ca590 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.028+0000 7f7ca7fff640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7c8c06b3e0 0x7f7c8c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.028+0000 7f7ca7fff640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7c8c06b3e0 0x7f7c8c06d8a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7c94006fd0 tx=0x7f7c94008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.032+0000 7f7ca57fa640 1 -- 172.21.15.172:0/1586214424 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c9c077540 con 0x7f7ca80ca590 2024-03-16T19:29:23.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.204+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ca80681c0 con 0x7f7ca80ca590 2024-03-16T19:29:23.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.205+0000 7f7ca57fa640 1 -- 172.21.15.172:0/1586214424 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12908 (secure 0 0 0) 0x7f7c9c07ac00 con 0x7f7ca80ca590 2024-03-16T19:29:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:23.187 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":17,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:17.305669+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7c8c06b3e0 msgr2=0x7f7c8c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7c8c06b3e0 0x7f7c8c06d8a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7c94006fd0 tx=0x7f7c94008040 comp rx=0 tx=0).stop 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca80ca590 msgr2=0x7f7ca81518b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca80ca590 0x7f7ca81518b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c000c00 tx=0x7f7c9c004730 comp rx=0 tx=0).stop 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 shutdown_connections 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ca814f490 0x7f7ca8154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7c8c06b3e0 0x7f7c8c06d8a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ca80cb0f0 0x7f7ca81ee660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 --2- 172.21.15.172:0/1586214424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ca80ca590 0x7f7ca81518b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 >> 172.21.15.172:0/1586214424 conn(0x7f7ca806aed0 msgr2=0x7f7ca815b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 shutdown_connections 2024-03-16T19:29:23.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.207+0000 7f7caeceb640 1 -- 172.21.15.172:0/1586214424 wait complete. 2024-03-16T19:29:23.199 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:29:14.242742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T19:29:23.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:29:23.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.322+0000 7faa20e8f640 1 Processor -- start 2024-03-16T19:29:23.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.322+0000 7faa20e8f640 1 -- start start 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa20e8f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faa1c154c90 0x7faa1c0c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa20e8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c0cd290 0x7faa1c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa20e8f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c06b2b0 con 0x7faa1c06cc50 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa20e8f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1c15b470 con 0x7faa1c154c90 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa20e8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0c2700 con 0x7faa1c0cd290 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa19d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7faa1c06cc50 0x7faa1c0c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40176/0 (socket says 172.21.15.172:40176) 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa19d9b640 1 -- 172.21.15.172:0/3185764707 learned_addr learned my addr 172.21.15.172:0/3185764707 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258429976 0 0) 0x7faa1c06b2b0 con 0x7faa1c06cc50 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa04003520 con 0x7faa1c06cc50 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1a59c640 1 --2- 172.21.15.172:0/3185764707 >> v2:172.21.15.174:3300/0 conn(0x7faa1c154c90 0x7faa1c0c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1ad9d640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c0cd290 0x7faa1c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661413259 0 0) 0x7faa04003520 con 0x7faa1c06cc50 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c0cd290 msgr2=0x7faa1c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c0cd290 0x7faa1c0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 >> v2:172.21.15.174:3300/0 conn(0x7faa1c154c90 msgr2=0x7faa1c0c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 --2- 172.21.15.172:0/3185764707 >> v2:172.21.15.174:3300/0 conn(0x7faa1c154c90 0x7faa1c0c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c06b590 con 0x7faa1c06cc50 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7faa10001690 con 0x7faa1c06cc50 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.323+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 >> v1:172.21.15.172:6789/0 conn(0x7faa1c06cc50 legacy=0x7faa1c0c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa1959a640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c0cd290 0x7faa04003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa1959a640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa1959a640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa04004d00 0x7faa04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c15b470 con 0x7faa04003fa0 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0c2700 con 0x7faa04004d00 2024-03-16T19:29:23.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0400ba80 con 0x7faa1c0cd290 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa19d9b640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa1a59c640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c0cd290 0x7faa04003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa1ad9d640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa04004d00 0x7faa04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa19d9b640 1 -- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c0cd290 msgr2=0x7faa04003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa19d9b640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c0cd290 0x7faa04003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa19d9b640 1 -- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa04004d00 msgr2=0x7faa04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa19d9b640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa04004d00 0x7faa04005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.324+0000 7faa19d9b640 1 -- 172.21.15.172:0/3185764707 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa04003520 con 0x7faa04003fa0 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa19d9b640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa040047c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faa1c0b2750 tx=0x7faa040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123727917d3138e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10005ad0 con 0x7faa04003fa0 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa10005c50 con 0x7faa04003fa0 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa1959a640 1 -- 172.21.15.172:0/3185764707 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1002c8a0 con 0x7faa04003fa0 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa20e8f640 1 -- 172.21.15.172:0/3185764707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 msgr2=0x7faa040047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa20e8f640 1 --2- 172.21.15.172:0/3185764707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa040047c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faa1c0b2750 tx=0x7faa040057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa1ad9d640 1 -- 172.21.15.172:0/3185764707 reap_dead start 2024-03-16T19:29:23.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa20e8f640 1 -- 172.21.15.172:0/3185764707 shutdown_connections 2024-03-16T19:29:23.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa20e8f640 1 -- 172.21.15.172:0/3185764707 >> 172.21.15.172:0/3185764707 conn(0x7faa1c066c20 msgr2=0x7faa1c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:23.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa20e8f640 1 -- 172.21.15.172:0/3185764707 shutdown_connections 2024-03-16T19:29:23.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.325+0000 7faa20e8f640 1 -- 172.21.15.172:0/3185764707 wait complete. 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa20e8f640 1 Processor -- start 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa20e8f640 1 -- start start 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa20e8f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa1c0c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa20e8f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1c0c8aa0 0x7faa1c159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa20e8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c1f28e0 0x7faa1c1f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa20e8f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c1f1bf0 con 0x7faa04003fa0 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa20e8f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0c4c00 con 0x7faa1c0c8aa0 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa20e8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c1f32a0 con 0x7faa1c1f28e0 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa1a59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa1c0c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa1a59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa1c0c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52318/0 (socket says 172.21.15.172:52318) 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.327+0000 7faa1a59c640 1 -- 172.21.15.172:0/771139876 learned_addr learned my addr 172.21.15.172:0/771139876 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa1a59c640 1 -- 172.21.15.172:0/771139876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c1f28e0 msgr2=0x7faa1c1f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa1ad9d640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c1f28e0 0x7faa1c1f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa1a59c640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c1f28e0 0x7faa1c1f2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa19d9b640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1c0c8aa0 0x7faa1c159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa1a59c640 1 -- 172.21.15.172:0/771139876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1c0c8aa0 msgr2=0x7faa1c159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa1a59c640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1c0c8aa0 0x7faa1c159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa1a59c640 1 -- 172.21.15.172:0/771139876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c0c9b50 con 0x7faa04003fa0 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa1a59c640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa1c0c9510 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0049d0 tx=0x7faa0c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa0affd640 1 -- 172.21.15.172:0/771139876 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c015420 con 0x7faa04003fa0 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa0affd640 1 -- 172.21.15.172:0/771139876 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c015580 con 0x7faa04003fa0 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa0affd640 1 -- 172.21.15.172:0/771139876 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c015930 con 0x7faa04003fa0 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c0c9e30 con 0x7faa04003fa0 2024-03-16T19:29:23.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.328+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c1f20e0 con 0x7faa04003fa0 2024-03-16T19:29:23.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.329+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c0c2520 con 0x7faa04003fa0 2024-03-16T19:29:23.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.329+0000 7faa0affd640 1 -- 172.21.15.172:0/771139876 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7faa0c015a90 con 0x7faa04003fa0 2024-03-16T19:29:23.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.330+0000 7faa0affd640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9fc06b3b0 0x7fa9fc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.330+0000 7faa0affd640 1 -- 172.21.15.172:0/771139876 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faa0c08a6c0 con 0x7faa04003fa0 2024-03-16T19:29:23.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.330+0000 7faa19d9b640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9fc06b3b0 0x7fa9fc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.330+0000 7faa19d9b640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9fc06b3b0 0x7fa9fc06d870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa10034000 tx=0x7faa10007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.334+0000 7faa0affd640 1 -- 172.21.15.172:0/771139876 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa0c0556a0 con 0x7faa04003fa0 2024-03-16T19:29:23.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.505+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa1c0688d0 con 0x7faa04003fa0 2024-03-16T19:29:23.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.505+0000 7faa0affd640 1 -- 172.21.15.172:0/771139876 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa0c058d60 con 0x7faa04003fa0 2024-03-16T19:29:23.487 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 1 2024-03-16T19:29:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9fc06b3b0 msgr2=0x7fa9fc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9fc06b3b0 0x7fa9fc06d870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa10034000 tx=0x7faa10007000 comp rx=0 tx=0).stop 2024-03-16T19:29:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 msgr2=0x7faa1c0c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa1c0c9510 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0049d0 tx=0x7faa0c007700 comp rx=0 tx=0).stop 2024-03-16T19:29:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 shutdown_connections 2024-03-16T19:29:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9fc06b3b0 0x7fa9fc06d870 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c1f28e0 0x7faa1c1f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1c0c8aa0 0x7faa1c159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 --2- 172.21.15.172:0/771139876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa04003fa0 0x7faa1c0c9510 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 >> 172.21.15.172:0/771139876 conn(0x7faa1c066c20 msgr2=0x7faa1c0b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:23.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 shutdown_connections 2024-03-16T19:29:23.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.507+0000 7faa20e8f640 1 -- 172.21.15.172:0/771139876 wait complete. 2024-03-16T19:29:23.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-16T19:29:23.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.618+0000 7f8c0fc6f640 1 Processor -- start 2024-03-16T19:29:23.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.619+0000 7f8c0fc6f640 1 -- start start 2024-03-16T19:29:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.619+0000 7f8c0fc6f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8c080c51b0 0x7f8c081661d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.619+0000 7f8c0fc6f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0815b880 con 0x7f8c080cc780 2024-03-16T19:29:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.619+0000 7f8c0fc6f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c08155070 con 0x7f8c08154c90 2024-03-16T19:29:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.619+0000 7f8c0fc6f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c080c6ee0 con 0x7f8c080c51b0 2024-03-16T19:29:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.619+0000 7f8c0d20a640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8c080cc780 0x7f8c080cd1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40184/0 (socket says 172.21.15.172:40184) 2024-03-16T19:29:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.619+0000 7f8c0d20a640 1 -- 172.21.15.172:0/3193426283 learned_addr learned my addr 172.21.15.172:0/3193426283 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:23.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.620+0000 7f8c0e20c640 1 --2- 172.21.15.172:0/3193426283 >> v2:172.21.15.174:3300/0 conn(0x7f8c080c51b0 0x7f8c081661d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.620+0000 7f8c0ca09640 1 -- 172.21.15.172:0/3193426283 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791196966 0 0) 0x7f8bf8001110 con 0x7f8c080cc780 2024-03-16T19:29:23.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.620+0000 7f8c0ca09640 1 -- 172.21.15.172:0/3193426283 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bf4003520 con 0x7f8c080cc780 2024-03-16T19:29:23.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.620+0000 7f8c0e20c640 1 -- 172.21.15.172:0/3193426283 >> v1:172.21.15.174:6789/0 conn(0x7f8c08154c90 legacy=0x7f8c0815e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.620+0000 7f8c0ca09640 1 -- 172.21.15.172:0/3193426283 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1132151205 0 0) 0x7f8bf4003520 con 0x7f8c080cc780 2024-03-16T19:29:23.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.620+0000 7f8c0e20c640 1 -- 172.21.15.172:0/3193426283 >> v1:172.21.15.172:6789/0 conn(0x7f8c080cc780 legacy=0x7f8c080cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.620+0000 7f8c0e20c640 1 -- 172.21.15.172:0/3193426283 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0815ad00 con 0x7f8c080c51b0 2024-03-16T19:29:23.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.621+0000 7f8c0e20c640 1 --2- 172.21.15.172:0/3193426283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c080c51b0 0x7f8c081661d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009d90 tx=0x7f8c0002bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e657f214965035d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.621+0000 7f8c0ca09640 1 -- 172.21.15.172:0/3193426283 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8001800 con 0x7f8c080c51b0 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.621+0000 7f8c0ca09640 1 -- 172.21.15.172:0/3193426283 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0002c7a0 con 0x7f8c080c51b0 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.621+0000 7f8c0ca09640 1 -- 172.21.15.172:0/3193426283 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0002cb30 con 0x7f8c080c51b0 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.621+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/3193426283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c080c51b0 msgr2=0x7f8c081661d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.621+0000 7f8c0fc6f640 1 --2- 172.21.15.172:0/3193426283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c080c51b0 0x7f8c081661d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009d90 tx=0x7f8c0002bc70 comp rx=0 tx=0).stop 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.621+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/3193426283 shutdown_connections 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.621+0000 7f8c0fc6f640 1 --2- 172.21.15.172:0/3193426283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c080c51b0 0x7f8c081661d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.621+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/3193426283 >> 172.21.15.172:0/3193426283 conn(0x7f8c08067040 msgr2=0x7f8c08067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.622+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/3193426283 shutdown_connections 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.622+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/3193426283 wait complete. 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0fc6f640 1 Processor -- start 2024-03-16T19:29:23.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0fc6f640 1 -- start start 2024-03-16T19:29:23.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0fc6f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8c080c51b0 0x7f8c080cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0fc6f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c080cc780 0x7f8c08159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0fc6f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c08154c90 0x7f8c08159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0fc6f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c08158c50 con 0x7f8c08154c90 2024-03-16T19:29:23.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0fc6f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c08159f80 con 0x7f8c080cc780 2024-03-16T19:29:23.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0fc6f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c081f4670 con 0x7f8c080c51b0 2024-03-16T19:29:23.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0e20c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c08154c90 0x7f8c08159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0e20c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c08154c90 0x7f8c08159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52330/0 (socket says 172.21.15.172:52330) 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.623+0000 7f8c0e20c640 1 -- 172.21.15.172:0/1023745559 learned_addr learned my addr 172.21.15.172:0/1023745559 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0d20a640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c080cc780 0x7f8c08159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0da0b640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8c080c51b0 0x7f8c080cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0e20c640 1 -- 172.21.15.172:0/1023745559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8c080c51b0 msgr2=0x7f8c080cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0e20c640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8c080c51b0 0x7f8c080cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0e20c640 1 -- 172.21.15.172:0/1023745559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c080cc780 msgr2=0x7f8c08159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0e20c640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c080cc780 0x7f8c08159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0e20c640 1 -- 172.21.15.172:0/1023745559 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c00009a70 con 0x7f8c08154c90 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0e20c640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c08154c90 0x7f8c08159a40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009d90 tx=0x7f8c00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8bfe7fc640 1 -- 172.21.15.172:0/1023745559 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c00034c00 con 0x7f8c08154c90 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8bfe7fc640 1 -- 172.21.15.172:0/1023745559 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c00034d80 con 0x7f8c08154c90 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c080b5420 con 0x7f8c08154c90 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8bfe7fc640 1 -- 172.21.15.172:0/1023745559 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c00040690 con 0x7f8c08154c90 2024-03-16T19:29:23.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.624+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c081f3cd0 con 0x7f8c08154c90 2024-03-16T19:29:23.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.625+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0815dfb0 con 0x7f8c08154c90 2024-03-16T19:29:23.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.625+0000 7f8bfe7fc640 1 -- 172.21.15.172:0/1023745559 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8c000407f0 con 0x7f8c08154c90 2024-03-16T19:29:23.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.626+0000 7f8bfe7fc640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bec06b3b0 0x7f8bec06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.626+0000 7f8bfe7fc640 1 -- 172.21.15.172:0/1023745559 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8c000ac2c0 con 0x7f8c08154c90 2024-03-16T19:29:23.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.626+0000 7f8c0da0b640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bec06b3b0 0x7f8bec06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.626+0000 7f8c0da0b640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bec06b3b0 0x7f8bec06d870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8c04000f60 tx=0x7f8c04009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.630+0000 7f8bfe7fc640 1 -- 172.21.15.172:0/1023745559 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c0815dfb0 con 0x7f8c08154c90 2024-03-16T19:29:23.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.803+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8c081f47b0 con 0x7f8c08154c90 2024-03-16T19:29:23.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.803+0000 7f8bfe7fc640 1 -- 172.21.15.172:0/1023745559 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8c081f47b0 con 0x7f8c08154c90 2024-03-16T19:29:23.784 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:23.784 INFO:teuthology.orchestra.run.smithi172.stdout:[ 2024-03-16T19:29:23.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:23.785 INFO:teuthology.orchestra.run.smithi172.stdout: "poolnum": 1, 2024-03-16T19:29:23.785 INFO:teuthology.orchestra.run.smithi172.stdout: "poolname": ".mgr" 2024-03-16T19:29:23.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:23.785 INFO:teuthology.orchestra.run.smithi172.stdout:] 2024-03-16T19:29:23.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bec06b3b0 msgr2=0x7f8bec06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bec06b3b0 0x7f8bec06d870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8c04000f60 tx=0x7f8c04009040 comp rx=0 tx=0).stop 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c08154c90 msgr2=0x7f8c08159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c08154c90 0x7f8c08159a40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009d90 tx=0x7f8c00004a40 comp rx=0 tx=0).stop 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 shutdown_connections 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8c08154c90 0x7f8c08159a40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c080cc780 0x7f8c08159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bec06b3b0 0x7f8bec06d870 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 --2- 172.21.15.172:0/1023745559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8c080c51b0 0x7f8c080cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 >> 172.21.15.172:0/1023745559 conn(0x7f8c08067040 msgr2=0x7f8c080c60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 shutdown_connections 2024-03-16T19:29:23.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.805+0000 7f8c0fc6f640 1 -- 172.21.15.172:0/1023745559 wait complete. 2024-03-16T19:29:23.797 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-16T19:29:23.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-16T19:29:23.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.920+0000 7ffb1712c640 1 Processor -- start 2024-03-16T19:29:23.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.920+0000 7ffb1712c640 1 -- start start 2024-03-16T19:29:23.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb1712c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffb100710a0 0x7ffb101591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb1712c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb100b0ce0 0x7ffb100c3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb1712c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1006f640 con 0x7ffb10152ae0 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb1712c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb101596f0 con 0x7ffb100b0ce0 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb1712c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb10152ec0 con 0x7ffb100710a0 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb156c9640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb100b0ce0 0x7ffb100c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb156c9640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ffb100b0ce0 0x7ffb100c3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52340/0 (socket says 172.21.15.172:52340) 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb156c9640 1 -- 172.21.15.172:0/3305305160 learned_addr learned my addr 172.21.15.172:0/3305305160 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb0ffff640 1 --2- 172.21.15.172:0/3305305160 >> v2:172.21.15.174:3300/0 conn(0x7ffb100710a0 0x7ffb101591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb156c9640 1 -- 172.21.15.172:0/3305305160 >> v2:172.21.15.174:3300/0 conn(0x7ffb100710a0 msgr2=0x7ffb101591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb156c9640 1 --2- 172.21.15.172:0/3305305160 >> v2:172.21.15.174:3300/0 conn(0x7ffb100710a0 0x7ffb101591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb156c9640 1 -- 172.21.15.172:0/3305305160 >> v1:172.21.15.174:6789/0 conn(0x7ffb10152ae0 legacy=0x7ffb100c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.921+0000 7ffb156c9640 1 -- 172.21.15.172:0/3305305160 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb10071590 con 0x7ffb100b0ce0 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.922+0000 7ffb156c9640 1 --2- 172.21.15.172:0/3305305160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb100b0ce0 0x7ffb100c3460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffb100cc000 tx=0x7ffb0802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47642a0d647a603b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.922+0000 7ffb0f7fe640 1 -- 172.21.15.172:0/3305305160 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0802cb20 con 0x7ffb100b0ce0 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.922+0000 7ffb0f7fe640 1 -- 172.21.15.172:0/3305305160 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0802cc80 con 0x7ffb100b0ce0 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.922+0000 7ffb0f7fe640 1 -- 172.21.15.172:0/3305305160 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb080343f0 con 0x7ffb100b0ce0 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.922+0000 7ffb1712c640 1 -- 172.21.15.172:0/3305305160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb100b0ce0 msgr2=0x7ffb100c3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.922+0000 7ffb1712c640 1 --2- 172.21.15.172:0/3305305160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb100b0ce0 0x7ffb100c3460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffb100cc000 tx=0x7ffb0802bf80 comp rx=0 tx=0).stop 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.923+0000 7ffb1712c640 1 -- 172.21.15.172:0/3305305160 shutdown_connections 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.923+0000 7ffb1712c640 1 --2- 172.21.15.172:0/3305305160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb100b0ce0 0x7ffb100c3460 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.923+0000 7ffb1712c640 1 --2- 172.21.15.172:0/3305305160 >> v2:172.21.15.174:3300/0 conn(0x7ffb100710a0 0x7ffb101591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.923+0000 7ffb1712c640 1 -- 172.21.15.172:0/3305305160 >> 172.21.15.172:0/3305305160 conn(0x7ffb10066c20 msgr2=0x7ffb10067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.923+0000 7ffb1712c640 1 -- 172.21.15.172:0/3305305160 shutdown_connections 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.923+0000 7ffb1712c640 1 -- 172.21.15.172:0/3305305160 wait complete. 2024-03-16T19:29:23.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.924+0000 7ffb1712c640 1 Processor -- start 2024-03-16T19:29:23.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.924+0000 7ffb1712c640 1 -- start start 2024-03-16T19:29:23.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb1712c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb100710a0 0x7ffb100c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb1712c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb100b0ce0 0x7ffb101f1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb1712c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb10152ae0 0x7ffb10158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb1712c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100cdf30 con 0x7ffb10152ae0 2024-03-16T19:29:23.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb1712c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100c5d80 con 0x7ffb100b0ce0 2024-03-16T19:29:23.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb1712c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100ceb80 con 0x7ffb100710a0 2024-03-16T19:29:23.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb156c9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb10152ae0 0x7ffb10158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb156c9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb10152ae0 0x7ffb10158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52342/0 (socket says 172.21.15.172:52342) 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb156c9640 1 -- 172.21.15.172:0/612077289 learned_addr learned my addr 172.21.15.172:0/612077289 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb0ffff640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb100b0ce0 0x7ffb101f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb14ec8640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb100710a0 0x7ffb100c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb156c9640 1 -- 172.21.15.172:0/612077289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb100710a0 msgr2=0x7ffb100c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb156c9640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb100710a0 0x7ffb100c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb156c9640 1 -- 172.21.15.172:0/612077289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb100b0ce0 msgr2=0x7ffb101f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb156c9640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb100b0ce0 0x7ffb101f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb156c9640 1 -- 172.21.15.172:0/612077289 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb08009b30 con 0x7ffb10152ae0 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb0ffff640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb100b0ce0 0x7ffb101f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb156c9640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb10152ae0 0x7ffb10158ba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffb08036040 tx=0x7ffb08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb0d7fa640 1 -- 172.21.15.172:0/612077289 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08007e70 con 0x7ffb10152ae0 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb0d7fa640 1 -- 172.21.15.172:0/612077289 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb08009510 con 0x7ffb10152ae0 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb0d7fa640 1 -- 172.21.15.172:0/612077289 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0803e690 con 0x7ffb10152ae0 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb100cee60 con 0x7ffb10152ae0 2024-03-16T19:29:23.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.925+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb100cd870 con 0x7ffb10152ae0 2024-03-16T19:29:23.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.926+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb100ce070 con 0x7ffb10152ae0 2024-03-16T19:29:23.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.927+0000 7ffb0d7fa640 1 -- 172.21.15.172:0/612077289 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ffb0803e7f0 con 0x7ffb10152ae0 2024-03-16T19:29:23.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.927+0000 7ffb0d7fa640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffaf406b3b0 0x7ffaf406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:23.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.927+0000 7ffb0d7fa640 1 -- 172.21.15.172:0/612077289 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffb080ac430 con 0x7ffb10152ae0 2024-03-16T19:29:23.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.927+0000 7ffb14ec8640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffaf406b3b0 0x7ffaf406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:23.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.927+0000 7ffb14ec8640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffaf406b3b0 0x7ffaf406d870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffb040016e0 tx=0x7ffb04009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:23.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:23.931+0000 7ffb0d7fa640 1 -- 172.21.15.172:0/612077289 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb100ce070 con 0x7ffb10152ae0 2024-03-16T19:29:24.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.103+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ffb100ce070 con 0x7ffb10152ae0 2024-03-16T19:29:24.302 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.320+0000 7ffb0d7fa640 1 -- 172.21.15.172:0/612077289 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7ffb100ce070 con 0x7ffb10152ae0 2024-03-16T19:29:24.302 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'cephfs_metadata' created 2024-03-16T19:29:24.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffaf406b3b0 msgr2=0x7ffaf406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:24.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffaf406b3b0 0x7ffaf406d870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffb040016e0 tx=0x7ffb04009040 comp rx=0 tx=0).stop 2024-03-16T19:29:24.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb10152ae0 msgr2=0x7ffb10158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:24.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb10152ae0 0x7ffb10158ba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffb08036040 tx=0x7ffb08004a40 comp rx=0 tx=0).stop 2024-03-16T19:29:24.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 shutdown_connections 2024-03-16T19:29:24.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb10152ae0 0x7ffb10158ba0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb100b0ce0 0x7ffb101f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffaf406b3b0 0x7ffaf406d870 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 --2- 172.21.15.172:0/612077289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb100710a0 0x7ffb100c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 >> 172.21.15.172:0/612077289 conn(0x7ffb10066c20 msgr2=0x7ffb10157e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:24.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 shutdown_connections 2024-03-16T19:29:24.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.323+0000 7ffb1712c640 1 -- 172.21.15.172:0/612077289 wait complete. 2024-03-16T19:29:24.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-16T19:29:24.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.445+0000 7f77b47c7640 1 Processor -- start 2024-03-16T19:29:24.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.445+0000 7f77b47c7640 1 -- start start 2024-03-16T19:29:24.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.446+0000 7f77b47c7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f77ac1594c0 0x7f77ac0ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:24.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.446+0000 7f77b47c7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac0c3360 con 0x7f77ac0ce2b0 2024-03-16T19:29:24.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.446+0000 7f77b47c7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac152e40 con 0x7f77ac06ff30 2024-03-16T19:29:24.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.446+0000 7f77b47c7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac0c4330 con 0x7f77ac1594c0 2024-03-16T19:29:24.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.446+0000 7f77b2563640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f77ac0ce2b0 0x7f77ac158db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40186/0 (socket says 172.21.15.172:40186) 2024-03-16T19:29:24.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.446+0000 7f77b2563640 1 -- 172.21.15.172:0/116807692 learned_addr learned my addr 172.21.15.172:0/116807692 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:24.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1d62640 1 --2- 172.21.15.172:0/116807692 >> v2:172.21.15.174:3300/0 conn(0x7f77ac1594c0 0x7f77ac0ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020149012 0 0) 0x7f77ac0c3360 con 0x7f77ac0ce2b0 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7798003520 con 0x7f77ac0ce2b0 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131075575 0 0) 0x7f7798003520 con 0x7f77ac0ce2b0 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 >> v2:172.21.15.174:3300/0 conn(0x7f77ac1594c0 msgr2=0x7f77ac0ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 --2- 172.21.15.172:0/116807692 >> v2:172.21.15.174:3300/0 conn(0x7f77ac1594c0 0x7f77ac0ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 >> v1:172.21.15.174:6789/0 conn(0x7f77ac06ff30 legacy=0x7f77ac163100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac0c5520 con 0x7f77ac0ce2b0 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f77a8001670 con 0x7f77ac0ce2b0 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 >> v1:172.21.15.172:6789/0 conn(0x7f77ac0ce2b0 legacy=0x7f77ac158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7798003970 0x7f7798003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77980047c0 0x7f7798004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7798005180 0x7f779800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac0c4330 con 0x7f7798003970 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779800c120 con 0x7f7798005180 2024-03-16T19:29:24.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779800c2e0 con 0x7f77980047c0 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b2563640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7798003970 0x7f7798003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1d62640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77980047c0 0x7f7798004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b2d64640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7798005180 0x7f779800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b2563640 1 -- 172.21.15.172:0/116807692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77980047c0 msgr2=0x7f7798004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b2563640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77980047c0 0x7f7798004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b2563640 1 -- 172.21.15.172:0/116807692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7798005180 msgr2=0x7f779800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b2563640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7798005180 0x7f779800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b2d64640 1 -- 172.21.15.172:0/116807692 reap_dead start 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b2563640 1 -- 172.21.15.172:0/116807692 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798003520 con 0x7f7798003970 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.447+0000 7f77b1d62640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77980047c0 0x7f7798004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.448+0000 7f77b2563640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7798003970 0x7f7798003d90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f77a800c240 tx=0x7f77980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e18b8af19a7f43a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.448+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a8005ad0 con 0x7f7798003970 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.448+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a8005c50 con 0x7f7798003970 2024-03-16T19:29:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.448+0000 7f77b1561640 1 -- 172.21.15.172:0/116807692 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a802c8b0 con 0x7f7798003970 2024-03-16T19:29:24.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.449+0000 7f77b47c7640 1 -- 172.21.15.172:0/116807692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7798003970 msgr2=0x7f7798003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:24.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.449+0000 7f77b47c7640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7798003970 0x7f7798003d90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f77a800c240 tx=0x7f77980057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.449+0000 7f77b47c7640 1 -- 172.21.15.172:0/116807692 shutdown_connections 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.449+0000 7f77b47c7640 1 --2- 172.21.15.172:0/116807692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7798003970 0x7f7798003d90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.449+0000 7f77b47c7640 1 -- 172.21.15.172:0/116807692 >> 172.21.15.172:0/116807692 conn(0x7f77ac066c20 msgr2=0x7f77ac067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.449+0000 7f77b47c7640 1 -- 172.21.15.172:0/116807692 shutdown_connections 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.449+0000 7f77b47c7640 1 -- 172.21.15.172:0/116807692 wait complete. 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.450+0000 7f77b47c7640 1 Processor -- start 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.450+0000 7f77b47c7640 1 -- start start 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b47c7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7798003970 0x7f77ac0b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b47c7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77ac0b3560 0x7f77ac0ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b47c7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77ac0b0240 0x7f77ac06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b47c7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac1f0c60 con 0x7f77ac0b0240 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b47c7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac0afbe0 con 0x7f77ac0b3560 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b47c7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac1f0350 con 0x7f7798003970 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2d64640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77ac0b0240 0x7f77ac06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:24.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2d64640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77ac0b0240 0x7f77ac06d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52352/0 (socket says 172.21.15.172:52352) 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2d64640 1 -- 172.21.15.172:0/2531299365 learned_addr learned my addr 172.21.15.172:0/2531299365 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2563640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7798003970 0x7f77ac0b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b1d62640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77ac0b3560 0x7f77ac0ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2d64640 1 -- 172.21.15.172:0/2531299365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7798003970 msgr2=0x7f77ac0b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2d64640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7798003970 0x7f77ac0b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2d64640 1 -- 172.21.15.172:0/2531299365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77ac0b3560 msgr2=0x7f77ac0ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2d64640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77ac0b3560 0x7f77ac0ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2d64640 1 -- 172.21.15.172:0/2531299365 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac0afe80 con 0x7f77ac0b0240 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b2d64640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77ac0b0240 0x7f77ac06d3d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77a4007fd0 tx=0x7f77a400e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77a2ffd640 1 -- 172.21.15.172:0/2531299365 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4006d60 con 0x7f77ac0b0240 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77a2ffd640 1 -- 172.21.15.172:0/2531299365 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a4003970 con 0x7f77ac0b0240 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ac1f0610 con 0x7f77ac0b0240 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77a2ffd640 1 -- 172.21.15.172:0/2531299365 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4003d20 con 0x7f77ac0b0240 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.451+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac0aede0 con 0x7f77ac0b0240 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.452+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ac1f0dd0 con 0x7f77ac0b0240 2024-03-16T19:29:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.453+0000 7f77a2ffd640 1 -- 172.21.15.172:0/2531299365 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f77a4005110 con 0x7f77ac0b0240 2024-03-16T19:29:24.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.453+0000 7f77a2ffd640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f779006b3e0 0x7f779006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:24.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.453+0000 7f77a2ffd640 1 -- 172.21.15.172:0/2531299365 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f77a4089150 con 0x7f77ac0b0240 2024-03-16T19:29:24.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.453+0000 7f77b2563640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f779006b3e0 0x7f779006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:24.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.453+0000 7f77b2563640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f779006b3e0 0x7f779006d8a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f77a8034000 tx=0x7f77a8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:24.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.458+0000 7f77a2ffd640 1 -- 172.21.15.172:0/2531299365 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77a4054050 con 0x7f77ac0b0240 2024-03-16T19:29:24.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:24.632+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f77ac0688d0 con 0x7f77ac0b0240 2024-03-16T19:29:25.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.323+0000 7f77a2ffd640 1 -- 172.21.15.172:0/2531299365 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f77a4057710 con 0x7f77ac0b0240 2024-03-16T19:29:25.304 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'cephfs_data' created 2024-03-16T19:29:25.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f779006b3e0 msgr2=0x7f779006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:25.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f779006b3e0 0x7f779006d8a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f77a8034000 tx=0x7f77a8009030 comp rx=0 tx=0).stop 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77ac0b0240 msgr2=0x7f77ac06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77ac0b0240 0x7f77ac06d3d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77a4007fd0 tx=0x7f77a400e9e0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 shutdown_connections 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77ac0b0240 0x7f77ac06d3d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77ac0b3560 0x7f77ac0ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f779006b3e0 0x7f779006d8a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 --2- 172.21.15.172:0/2531299365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7798003970 0x7f77ac0b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 >> 172.21.15.172:0/2531299365 conn(0x7f77ac066c20 msgr2=0x7f77ac157c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 shutdown_connections 2024-03-16T19:29:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.325+0000 7f77b47c7640 1 -- 172.21.15.172:0/2531299365 wait complete. 2024-03-16T19:29:25.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-16T19:29:25.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.450+0000 7f517c79a640 1 Processor -- start 2024-03-16T19:29:25.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.450+0000 7f517c79a640 1 -- start start 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f517c79a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51740c5cf0 0x7f517406f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f517c79a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51740c8360 con 0x7f517415c860 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f517c79a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51740c85d0 con 0x7f51740c5910 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f517c79a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51740c88b0 con 0x7f51740c5cf0 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f517ad37640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f517415c860 0x7f51741585f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40198/0 (socket says 172.21.15.172:40198) 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f517ad37640 1 -- 172.21.15.172:0/3845577474 learned_addr learned my addr 172.21.15.172:0/3845577474 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f5179d35640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51740c5cf0 0x7f517406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516615619 0 0) 0x7f51740c8360 con 0x7f517415c860 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5160003520 con 0x7f517415c860 2024-03-16T19:29:25.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208222712 0 0) 0x7f51740c85d0 con 0x7f51740c5910 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.451+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51740c8360 con 0x7f51740c5910 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3672171262 0 0) 0x7f5160003520 con 0x7f517415c860 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51740c5cf0 msgr2=0x7f517406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51740c5cf0 0x7f517406f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 >> v1:172.21.15.174:6789/0 conn(0x7f51740c5910 legacy=0x7f51740cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517415bd80 con 0x7f517415c860 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f516c001670 con 0x7f517415c860 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 >> v1:172.21.15.172:6789/0 conn(0x7f517415c860 legacy=0x7f51741585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160003b30 0x7f5160003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51600047c0 0x7f5160004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5160005180 0x7f516000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51740c88b0 con 0x7f5160005180 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516000bef0 con 0x7f51600047c0 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516000c0b0 con 0x7f5160003b30 2024-03-16T19:29:25.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179d35640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51740c5cf0 0x7f517406f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f517ad37640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5160005180 0x7f516000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f517a536640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160003b30 0x7f5160003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179d35640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51600047c0 0x7f5160004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f517ad37640 1 -- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160003b30 msgr2=0x7f5160003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f517ad37640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160003b30 0x7f5160003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f517ad37640 1 -- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51600047c0 msgr2=0x7f5160004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f517ad37640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51600047c0 0x7f5160004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f517ad37640 1 -- 172.21.15.172:0/3845577474 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5160003520 con 0x7f5160005180 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.452+0000 7f5179d35640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51600047c0 0x7f5160004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f517ad37640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5160005180 0x7f516000b980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c160 tx=0x7f51600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94de87aae7f93f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c005ad0 con 0x7f5160005180 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c005c30 con 0x7f5160005180 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f5179534640 1 -- 172.21.15.172:0/3845577474 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c02c810 con 0x7f5160005180 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f517c79a640 1 -- 172.21.15.172:0/3845577474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5160005180 msgr2=0x7f516000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:25.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f517c79a640 1 --2- 172.21.15.172:0/3845577474 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5160005180 0x7f516000b980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c160 tx=0x7f51600057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f517ad37640 1 -- 172.21.15.172:0/3845577474 reap_dead start 2024-03-16T19:29:25.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f517c79a640 1 -- 172.21.15.172:0/3845577474 shutdown_connections 2024-03-16T19:29:25.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f517c79a640 1 -- 172.21.15.172:0/3845577474 >> 172.21.15.172:0/3845577474 conn(0x7f5174066c20 msgr2=0x7f5174067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:25.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.453+0000 7f517c79a640 1 -- 172.21.15.172:0/3845577474 shutdown_connections 2024-03-16T19:29:25.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.454+0000 7f517c79a640 1 -- 172.21.15.172:0/3845577474 wait complete. 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517c79a640 1 Processor -- start 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517c79a640 1 -- start start 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517c79a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160005180 0x7f5174156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517c79a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f517414db00 0x7f517414ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517c79a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51741ee720 0x7f51741eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517c79a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51741ee300 con 0x7f51741ee720 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517c79a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5174156980 con 0x7f517414db00 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517c79a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517414ecc0 con 0x7f5160005180 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517ad37640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51741ee720 0x7f51741eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517ad37640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51741ee720 0x7f51741eebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52370/0 (socket says 172.21.15.172:52370) 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.455+0000 7f517ad37640 1 -- 172.21.15.172:0/2005002791 learned_addr learned my addr 172.21.15.172:0/2005002791 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517ad37640 1 -- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160005180 msgr2=0x7f5174156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517a536640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160005180 0x7f5174156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f5179d35640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f517414db00 0x7f517414ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517ad37640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160005180 0x7f5174156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517ad37640 1 -- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f517414db00 msgr2=0x7f517414ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517ad37640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f517414db00 0x7f517414ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517ad37640 1 -- 172.21.15.172:0/2005002791 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517415b3d0 con 0x7f51741ee720 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f5179d35640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f517414db00 0x7f517414ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517a536640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160005180 0x7f5174156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517ad37640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51741ee720 0x7f51741eebd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f516c02c480 tx=0x7f516c001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f516affd640 1 -- 172.21.15.172:0/2005002791 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c03c040 con 0x7f51741ee720 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f516affd640 1 -- 172.21.15.172:0/2005002791 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c02cc20 con 0x7f51741ee720 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f516affd640 1 -- 172.21.15.172:0/2005002791 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c001bc0 con 0x7f51741ee720 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517415b630 con 0x7f51741ee720 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.456+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517414e700 con 0x7f51741ee720 2024-03-16T19:29:25.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.457+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51741ee060 con 0x7f51741ee720 2024-03-16T19:29:25.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.457+0000 7f516affd640 1 -- 172.21.15.172:0/2005002791 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f516c032070 con 0x7f51741ee720 2024-03-16T19:29:25.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.458+0000 7f516affd640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f515806b3b0 0x7f515806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:25.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.458+0000 7f516affd640 1 -- 172.21.15.172:0/2005002791 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f516c0aaa80 con 0x7f51741ee720 2024-03-16T19:29:25.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.458+0000 7f517a536640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f515806b3b0 0x7f515806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:25.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.458+0000 7f517a536640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f515806b3b0 0x7f515806d870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5170003ee0 tx=0x7f5170009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:25.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.462+0000 7f516affd640 1 -- 172.21.15.172:0/2005002791 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51741ee060 con 0x7f51741ee720 2024-03-16T19:29:25.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:25.673+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f51741ee440 con 0x7f51741ee720 2024-03-16T19:29:25.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:29:25.951 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:29:26.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.347+0000 7f516affd640 1 -- 172.21.15.172:0/2005002791 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f51741ee440 con 0x7f51741ee720 2024-03-16T19:29:26.329 INFO:teuthology.orchestra.run.smithi172.stderr:new fs with metadata pool 2 and data pool 3 2024-03-16T19:29:26.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.352+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f515806b3b0 msgr2=0x7f515806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.352+0000 7f517c79a640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f515806b3b0 0x7f515806d870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5170003ee0 tx=0x7f5170009040 comp rx=0 tx=0).stop 2024-03-16T19:29:26.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.352+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51741ee720 msgr2=0x7f51741eebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.352+0000 7f517c79a640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51741ee720 0x7f51741eebd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f516c02c480 tx=0x7f516c001980 comp rx=0 tx=0).stop 2024-03-16T19:29:26.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.353+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 shutdown_connections 2024-03-16T19:29:26.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.353+0000 7f517c79a640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51741ee720 0x7f51741eebd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.353+0000 7f517c79a640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f517414db00 0x7f517414ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.353+0000 7f517c79a640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f515806b3b0 0x7f515806d870 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.353+0000 7f517c79a640 1 --2- 172.21.15.172:0/2005002791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5160005180 0x7f5174156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.353+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 >> 172.21.15.172:0/2005002791 conn(0x7f5174066c20 msgr2=0x7f517414f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:26.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.353+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 shutdown_connections 2024-03-16T19:29:26.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.354+0000 7f517c79a640 1 -- 172.21.15.172:0/2005002791 wait complete. 2024-03-16T19:29:26.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:26.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.470+0000 7faa3e46b640 1 Processor -- start 2024-03-16T19:29:26.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.470+0000 7faa3e46b640 1 -- start start 2024-03-16T19:29:26.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa3e46b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faa380c2810 0x7faa38158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa3e46b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa38071ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa3e46b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3806aed0 con 0x7faa38151b80 2024-03-16T19:29:26.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa3e46b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa38070d80 con 0x7faa380c2810 2024-03-16T19:29:26.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa3e46b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa380c1670 con 0x7faa380c1880 2024-03-16T19:29:26.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa3ca08640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa38071ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa377fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faa380c2810 0x7faa38158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa3ca08640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa38071ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:39458/0 (socket says 172.21.15.172:39458) 2024-03-16T19:29:26.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa37fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7faa38151b80 0x7faa380c2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:46068/0 (socket says 172.21.15.172:46068) 2024-03-16T19:29:26.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.471+0000 7faa3ca08640 1 -- 172.21.15.172:0/2261823090 learned_addr learned my addr 172.21.15.172:0/2261823090 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:26.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.472+0000 7faa377fe640 1 -- 172.21.15.172:0/2261823090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 msgr2=0x7faa38071ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:26.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.472+0000 7faa377fe640 1 --2- 172.21.15.172:0/2261823090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa38071ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.472+0000 7faa377fe640 1 -- 172.21.15.172:0/2261823090 >> v1:172.21.15.174:6789/0 conn(0x7faa38151b80 legacy=0x7faa380c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.472+0000 7faa377fe640 1 -- 172.21.15.172:0/2261823090 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa380c10f0 con 0x7faa380c2810 2024-03-16T19:29:26.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.472+0000 7faa36ffd640 1 -- 172.21.15.172:0/2261823090 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040649697 0 0) 0x7faa3806aed0 con 0x7faa38151b80 2024-03-16T19:29:26.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.472+0000 7faa3ca08640 1 --2- 172.21.15.172:0/2261823090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa38071ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:26.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.472+0000 7faa377fe640 1 --2- 172.21.15.172:0/2261823090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380c2810 0x7faa38158230 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0097a0 tx=0x7faa2c02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=518b12333c84f7a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:26.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.472+0000 7faa36ffd640 1 -- 172.21.15.172:0/2261823090 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c02f050 con 0x7faa380c2810 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa36ffd640 1 -- 172.21.15.172:0/2261823090 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2c031700 con 0x7faa380c2810 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa36ffd640 1 -- 172.21.15.172:0/2261823090 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c031a90 con 0x7faa380c2810 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa3e46b640 1 -- 172.21.15.172:0/2261823090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380c2810 msgr2=0x7faa38158230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa3e46b640 1 --2- 172.21.15.172:0/2261823090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380c2810 0x7faa38158230 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0097a0 tx=0x7faa2c02b7a0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa3e46b640 1 -- 172.21.15.172:0/2261823090 shutdown_connections 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa3e46b640 1 --2- 172.21.15.172:0/2261823090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa38071ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa3e46b640 1 --2- 172.21.15.172:0/2261823090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380c2810 0x7faa38158230 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa3e46b640 1 -- 172.21.15.172:0/2261823090 >> 172.21.15.172:0/2261823090 conn(0x7faa380c8b30 msgr2=0x7faa380c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa3e46b640 1 -- 172.21.15.172:0/2261823090 shutdown_connections 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.473+0000 7faa3e46b640 1 -- 172.21.15.172:0/2261823090 wait complete. 2024-03-16T19:29:26.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.474+0000 7faa3e46b640 1 Processor -- start 2024-03-16T19:29:26.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.474+0000 7faa3e46b640 1 -- start start 2024-03-16T19:29:26.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3e46b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa381531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3e46b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380c2810 0x7faa3814d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3e46b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa38151b80 0x7faa38155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3e46b640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa38154b30 con 0x7faa38151b80 2024-03-16T19:29:26.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3e46b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa381f2470 con 0x7faa380c2810 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3e46b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa381f2900 con 0x7faa380c1880 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3ca08640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa38151b80 0x7faa38155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3ca08640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa38151b80 0x7faa38155ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52384/0 (socket says 172.21.15.172:52384) 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3ca08640 1 -- 172.21.15.172:0/3384551221 learned_addr learned my addr 172.21.15.172:0/3384551221 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa377fe640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380c2810 0x7faa3814d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa37fff640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa381531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3ca08640 1 -- 172.21.15.172:0/3384551221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 msgr2=0x7faa381531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3ca08640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa381531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3ca08640 1 -- 172.21.15.172:0/3384551221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380c2810 msgr2=0x7faa3814d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3ca08640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380c2810 0x7faa3814d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.475+0000 7faa3ca08640 1 -- 172.21.15.172:0/3384551221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa30009b30 con 0x7faa38151b80 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.476+0000 7faa3ca08640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa38151b80 0x7faa38155ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faa30009e70 tx=0x7faa3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.476+0000 7faa34ff9640 1 -- 172.21.15.172:0/3384551221 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30018070 con 0x7faa38151b80 2024-03-16T19:29:26.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.476+0000 7faa34ff9640 1 -- 172.21.15.172:0/3384551221 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3000f040 con 0x7faa38151b80 2024-03-16T19:29:26.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.476+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2c009400 con 0x7faa38151b80 2024-03-16T19:29:26.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.476+0000 7faa34ff9640 1 -- 172.21.15.172:0/3384551221 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3000ce00 con 0x7faa38151b80 2024-03-16T19:29:26.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.476+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa38154870 con 0x7faa38151b80 2024-03-16T19:29:26.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.477+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa381f2620 con 0x7faa38151b80 2024-03-16T19:29:26.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.477+0000 7faa34ff9640 1 -- 172.21.15.172:0/3384551221 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7faa30014480 con 0x7faa38151b80 2024-03-16T19:29:26.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.478+0000 7faa34ff9640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2406b3b0 0x7faa2406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.478+0000 7faa34ff9640 1 -- 172.21.15.172:0/3384551221 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa30089b50 con 0x7faa38151b80 2024-03-16T19:29:26.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.478+0000 7faa37fff640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2406b3b0 0x7faa2406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.478+0000 7faa37fff640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2406b3b0 0x7faa2406d870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faa28007fd0 tx=0x7faa28007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:26.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.482+0000 7faa34ff9640 1 -- 172.21.15.172:0/3384551221 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa381f2620 con 0x7faa38151b80 2024-03-16T19:29:26.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.653+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa38154870 con 0x7faa38151b80 2024-03-16T19:29:26.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.654+0000 7faa34ff9640 1 -- 172.21.15.172:0/3384551221 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16642 (secure 0 0 0) 0x7faa38154870 con 0x7faa38151b80 2024-03-16T19:29:26.636 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:26.636 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":20,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:25.674584+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2406b3b0 msgr2=0x7faa2406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2406b3b0 0x7faa2406d870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faa28007fd0 tx=0x7faa28007de0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa38151b80 msgr2=0x7faa38155ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa38151b80 0x7faa38155ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faa30009e70 tx=0x7faa3000c490 comp rx=0 tx=0).stop 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 shutdown_connections 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa38151b80 0x7faa38155ae0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380c2810 0x7faa3814d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2406b3b0 0x7faa2406d870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 --2- 172.21.15.172:0/3384551221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa380c1880 0x7faa381531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.656+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 >> 172.21.15.172:0/3384551221 conn(0x7faa380c8b30 msgr2=0x7faa38072a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:26.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.657+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 shutdown_connections 2024-03-16T19:29:26.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.657+0000 7faa3e46b640 1 -- 172.21.15.172:0/3384551221 wait complete. 2024-03-16T19:29:26.649 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:26.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.768+0000 7f1227a50640 1 Processor -- start 2024-03-16T19:29:26.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.768+0000 7f1227a50640 1 -- start start 2024-03-16T19:29:26.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1227a50640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1220151230 0x7f122006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1227a50640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12200c9650 0x7f1220157540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1227a50640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1220151610 con 0x7f122006c500 2024-03-16T19:29:26.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1227a50640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f122006bb30 con 0x7f12200c9650 2024-03-16T19:29:26.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1227a50640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12200c2440 con 0x7f1220151230 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1225fed640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12200c9650 0x7f1220157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1224feb640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f122006c500 0x7f1220156e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40212/0 (socket says 172.21.15.172:40212) 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1225fed640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f12200c9650 0x7f1220157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52400/0 (socket says 172.21.15.172:52400) 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1224feb640 1 -- 172.21.15.172:0/2417892600 learned_addr learned my addr 172.21.15.172:0/2417892600 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f12257ec640 1 --2- 172.21.15.172:0/2417892600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1220151230 0x7f122006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1225fed640 1 -- 172.21.15.172:0/2417892600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1220151230 msgr2=0x7f122006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1225fed640 1 --2- 172.21.15.172:0/2417892600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1220151230 0x7f122006bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1225fed640 1 -- 172.21.15.172:0/2417892600 >> v1:172.21.15.172:6789/0 conn(0x7f122006c500 legacy=0x7f1220156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1217fff640 1 -- 172.21.15.172:0/2417892600 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124944655 0 0) 0x7f1220151610 con 0x7f122006c500 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.769+0000 7f1225fed640 1 -- 172.21.15.172:0/2417892600 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12200c6a30 con 0x7f12200c9650 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.770+0000 7f1225fed640 1 --2- 172.21.15.172:0/2417892600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12200c9650 0x7f1220157540 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12180097a0 tx=0x7f121802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1990845d57827767 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.770+0000 7f1217fff640 1 -- 172.21.15.172:0/2417892600 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218030610 con 0x7f12200c9650 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.770+0000 7f1217fff640 1 -- 172.21.15.172:0/2417892600 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1218030770 con 0x7f12200c9650 2024-03-16T19:29:26.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.770+0000 7f1217fff640 1 -- 172.21.15.172:0/2417892600 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218030b00 con 0x7f12200c9650 2024-03-16T19:29:26.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.771+0000 7f1227a50640 1 -- 172.21.15.172:0/2417892600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12200c9650 msgr2=0x7f1220157540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.771+0000 7f1227a50640 1 --2- 172.21.15.172:0/2417892600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12200c9650 0x7f1220157540 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12180097a0 tx=0x7f121802b960 comp rx=0 tx=0).stop 2024-03-16T19:29:26.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.771+0000 7f1227a50640 1 -- 172.21.15.172:0/2417892600 shutdown_connections 2024-03-16T19:29:26.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.771+0000 7f1227a50640 1 --2- 172.21.15.172:0/2417892600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12200c9650 0x7f1220157540 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.771+0000 7f1227a50640 1 --2- 172.21.15.172:0/2417892600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1220151230 0x7f122006bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.771+0000 7f1227a50640 1 -- 172.21.15.172:0/2417892600 >> 172.21.15.172:0/2417892600 conn(0x7f12200c8b30 msgr2=0x7f12200c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:26.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.771+0000 7f1227a50640 1 -- 172.21.15.172:0/2417892600 shutdown_connections 2024-03-16T19:29:26.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.771+0000 7f1227a50640 1 -- 172.21.15.172:0/2417892600 wait complete. 2024-03-16T19:29:26.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.772+0000 7f1227a50640 1 Processor -- start 2024-03-16T19:29:26.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.772+0000 7f1227a50640 1 -- start start 2024-03-16T19:29:26.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1227a50640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f122006c500 0x7f12201ecba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1227a50640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12200c9650 0x7f12200cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1227a50640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1220151230 0x7f1220155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1227a50640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12200c4e10 con 0x7f1220151230 2024-03-16T19:29:26.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1227a50640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12201ed0e0 con 0x7f12200c9650 2024-03-16T19:29:26.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1227a50640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12200c4520 con 0x7f122006c500 2024-03-16T19:29:26.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1225fed640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1220151230 0x7f1220155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1225fed640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1220151230 0x7f1220155e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52406/0 (socket says 172.21.15.172:52406) 2024-03-16T19:29:26.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1225fed640 1 -- 172.21.15.172:0/3842214673 learned_addr learned my addr 172.21.15.172:0/3842214673 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:26.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1224feb640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12200c9650 0x7f12200cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f12257ec640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f122006c500 0x7f12201ecba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1225fed640 1 -- 172.21.15.172:0/3842214673 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f122006c500 msgr2=0x7f12201ecba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1225fed640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f122006c500 0x7f12201ecba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1225fed640 1 -- 172.21.15.172:0/3842214673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12200c9650 msgr2=0x7f12200cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1225fed640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12200c9650 0x7f12200cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1225fed640 1 -- 172.21.15.172:0/3842214673 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1218009400 con 0x7f1220151230 2024-03-16T19:29:26.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1225fed640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1220151230 0x7f1220155e20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12180098c0 tx=0x7f1218004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:26.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1215ffb640 1 -- 172.21.15.172:0/3842214673 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121803e040 con 0x7f1220151230 2024-03-16T19:29:26.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.773+0000 7f1215ffb640 1 -- 172.21.15.172:0/3842214673 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1218039c10 con 0x7f1220151230 2024-03-16T19:29:26.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.774+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12200c6000 con 0x7f1220151230 2024-03-16T19:29:26.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.774+0000 7f1215ffb640 1 -- 172.21.15.172:0/3842214673 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218006e10 con 0x7f1220151230 2024-03-16T19:29:26.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.774+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12200c4aa0 con 0x7f1220151230 2024-03-16T19:29:26.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.775+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12200c7bd0 con 0x7f1220151230 2024-03-16T19:29:26.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.775+0000 7f1215ffb640 1 -- 172.21.15.172:0/3842214673 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f121804d3b0 con 0x7f1220151230 2024-03-16T19:29:26.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.775+0000 7f1215ffb640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f120406b3b0 0x7f120406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:26.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.775+0000 7f1215ffb640 1 -- 172.21.15.172:0/3842214673 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12180ac090 con 0x7f1220151230 2024-03-16T19:29:26.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.775+0000 7f12257ec640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f120406b3b0 0x7f120406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:26.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.776+0000 7f12257ec640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f120406b3b0 0x7f120406d870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f121c003b80 tx=0x7f121c0098d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:26.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.780+0000 7f1215ffb640 1 -- 172.21.15.172:0/3842214673 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1218076e30 con 0x7f1220151230 2024-03-16T19:29:26.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.953+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12200681c0 con 0x7f1220151230 2024-03-16T19:29:26.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.954+0000 7f1215ffb640 1 -- 172.21.15.172:0/3842214673 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16642 (secure 0 0 0) 0x7f121807a4f0 con 0x7f1220151230 2024-03-16T19:29:26.936 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:26.936 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":20,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:25.674584+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f120406b3b0 msgr2=0x7f120406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f120406b3b0 0x7f120406d870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f121c003b80 tx=0x7f121c0098d0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1220151230 msgr2=0x7f1220155e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1220151230 0x7f1220155e20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12180098c0 tx=0x7f1218004730 comp rx=0 tx=0).stop 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 shutdown_connections 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1220151230 0x7f1220155e20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12200c9650 0x7f12200cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f120406b3b0 0x7f120406d870 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 --2- 172.21.15.172:0/3842214673 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f122006c500 0x7f12201ecba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 >> 172.21.15.172:0/3842214673 conn(0x7f12200c8b30 msgr2=0x7f122014e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:26.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.956+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 shutdown_connections 2024-03-16T19:29:26.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:26.957+0000 7f1227a50640 1 -- 172.21.15.172:0/3842214673 wait complete. 2024-03-16T19:29:26.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-16T19:29:27.053 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.071+0000 7f8aca90d640 1 Processor -- start 2024-03-16T19:29:27.053 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.071+0000 7f8aca90d640 1 -- start start 2024-03-16T19:29:27.053 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8aca90d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8ac4158150 0x7f8ac406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.053 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8aca90d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8ac406c500 0x7f8ac40ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.053 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8aca90d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac406e010 con 0x7f8ac406f4f0 2024-03-16T19:29:27.053 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8aca90d640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac40c8f20 con 0x7f8ac406c500 2024-03-16T19:29:27.053 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8aca90d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac415b580 con 0x7f8ac4158150 2024-03-16T19:29:27.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8ac37fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8ac406c500 0x7f8ac40ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8ac37fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8ac406c500 0x7f8ac40ce0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52422/0 (socket says 172.21.15.172:52422) 2024-03-16T19:29:27.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8ac37fe640 1 -- 172.21.15.172:0/2535134727 learned_addr learned my addr 172.21.15.172:0/2535134727 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:27.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8ac3fff640 1 --2- 172.21.15.172:0/2535134727 >> v2:172.21.15.174:3300/0 conn(0x7f8ac4158150 0x7f8ac406e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8ac37fe640 1 -- 172.21.15.172:0/2535134727 >> v2:172.21.15.174:3300/0 conn(0x7f8ac4158150 msgr2=0x7f8ac406e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:27.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8ac37fe640 1 --2- 172.21.15.172:0/2535134727 >> v2:172.21.15.174:3300/0 conn(0x7f8ac4158150 0x7f8ac406e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8ac37fe640 1 -- 172.21.15.172:0/2535134727 >> v1:172.21.15.174:6789/0 conn(0x7f8ac406f4f0 legacy=0x7f8ac415df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:27.054 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.072+0000 7f8ac37fe640 1 -- 172.21.15.172:0/2535134727 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac406d810 con 0x7f8ac406c500 2024-03-16T19:29:27.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8ac37fe640 1 --2- 172.21.15.172:0/2535134727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac406c500 0x7f8ac40ce0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40ced50 tx=0x7f8ab802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6110ebbbb7aec3b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8ac2ffd640 1 -- 172.21.15.172:0/2535134727 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab80317e0 con 0x7f8ac406c500 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8ac2ffd640 1 -- 172.21.15.172:0/2535134727 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab8031940 con 0x7f8ac406c500 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8ac2ffd640 1 -- 172.21.15.172:0/2535134727 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8031cd0 con 0x7f8ac406c500 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8aca90d640 1 -- 172.21.15.172:0/2535134727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac406c500 msgr2=0x7f8ac40ce0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8aca90d640 1 --2- 172.21.15.172:0/2535134727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac406c500 0x7f8ac40ce0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40ced50 tx=0x7f8ab802b880 comp rx=0 tx=0).stop 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8aca90d640 1 -- 172.21.15.172:0/2535134727 shutdown_connections 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8aca90d640 1 --2- 172.21.15.172:0/2535134727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac406c500 0x7f8ac40ce0c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8aca90d640 1 --2- 172.21.15.172:0/2535134727 >> v2:172.21.15.174:3300/0 conn(0x7f8ac4158150 0x7f8ac406e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8aca90d640 1 -- 172.21.15.172:0/2535134727 >> 172.21.15.172:0/2535134727 conn(0x7f8ac4066c20 msgr2=0x7f8ac4067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8aca90d640 1 -- 172.21.15.172:0/2535134727 shutdown_connections 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.073+0000 7f8aca90d640 1 -- 172.21.15.172:0/2535134727 wait complete. 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.074+0000 7f8aca90d640 1 Processor -- start 2024-03-16T19:29:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8aca90d640 1 -- start start 2024-03-16T19:29:27.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8aca90d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ac406c500 0x7f8ac4151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8aca90d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ac406f4f0 0x7f8ac4150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8aca90d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac4158150 0x7f8ac41559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8aca90d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac41f1890 con 0x7f8ac4158150 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8aca90d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac41f1dc0 con 0x7f8ac406f4f0 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8aca90d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac415a4b0 con 0x7f8ac406c500 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8ac8eaa640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac4158150 0x7f8ac41559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8ac8eaa640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac4158150 0x7f8ac41559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52424/0 (socket says 172.21.15.172:52424) 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.075+0000 7f8ac8eaa640 1 -- 172.21.15.172:0/1865245827 learned_addr learned my addr 172.21.15.172:0/1865245827 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac8eaa640 1 -- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ac406c500 msgr2=0x7f8ac4151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac3fff640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ac406c500 0x7f8ac4151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac37fe640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ac406f4f0 0x7f8ac4150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac8eaa640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ac406c500 0x7f8ac4151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac8eaa640 1 -- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ac406f4f0 msgr2=0x7f8ac4150a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac8eaa640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ac406f4f0 0x7f8ac4150a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac8eaa640 1 -- 172.21.15.172:0/1865245827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab80094d0 con 0x7f8ac4158150 2024-03-16T19:29:27.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac3fff640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ac406c500 0x7f8ac4151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac37fe640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ac406f4f0 0x7f8ac4150a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac8eaa640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac4158150 0x7f8ac41559b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00b9d0 tx=0x7f8abc00be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac0ff9640 1 -- 172.21.15.172:0/1865245827 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc00cbd0 con 0x7f8ac4158150 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac0ff9640 1 -- 172.21.15.172:0/1865245827 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc00cd30 con 0x7f8ac4158150 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8ac0ff9640 1 -- 172.21.15.172:0/1865245827 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc00cea0 con 0x7f8ac4158150 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac4151670 con 0x7f8ac4158150 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.076+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac41f1ae0 con 0x7f8ac4158150 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.077+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac406bf60 con 0x7f8ac4158150 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.077+0000 7f8ac0ff9640 1 -- 172.21.15.172:0/1865245827 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8abc01d3b0 con 0x7f8ac4158150 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.078+0000 7f8ac0ff9640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8ab006b3b0 0x7f8ab006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.078+0000 7f8ac0ff9640 1 -- 172.21.15.172:0/1865245827 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8abc089940 con 0x7f8ac4158150 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.078+0000 7f8ac3fff640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8ab006b3b0 0x7f8ab006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.078+0000 7f8ac3fff640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8ab006b3b0 0x7f8ab006d870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4008890 tx=0x7f8ab4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:27.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.082+0000 7f8ac0ff9640 1 -- 172.21.15.172:0/1865245827 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8abc054630 con 0x7f8ac4158150 2024-03-16T19:29:27.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.290+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8ac40c52b0 con 0x7f8ac4158150 2024-03-16T19:29:27.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.348+0000 7f8ac0ff9640 1 -- 172.21.15.172:0/1865245827 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8abc057cf0 con 0x7f8ac4158150 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8ab006b3b0 msgr2=0x7f8ab006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8ab006b3b0 0x7f8ab006d870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4008890 tx=0x7f8ab4008040 comp rx=0 tx=0).stop 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac4158150 msgr2=0x7f8ac41559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac4158150 0x7f8ac41559b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00b9d0 tx=0x7f8abc00be90 comp rx=0 tx=0).stop 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 shutdown_connections 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ac4158150 0x7f8ac41559b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ac406f4f0 0x7f8ac4150a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8ab006b3b0 0x7f8ab006d870 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 --2- 172.21.15.172:0/1865245827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ac406c500 0x7f8ac4151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 >> 172.21.15.172:0/1865245827 conn(0x7f8ac4066c20 msgr2=0x7f8ac4154ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 shutdown_connections 2024-03-16T19:29:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.350+0000 7f8aca90d640 1 -- 172.21.15.172:0/1865245827 wait complete. 2024-03-16T19:29:27.342 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-03-16T19:29:27.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-16T19:29:27.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.465+0000 7f3099ad4640 1 Processor -- start 2024-03-16T19:29:27.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.465+0000 7f3099ad4640 1 -- start start 2024-03-16T19:29:27.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3099ad4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f30940c7560 0x7f30940caae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3099ad4640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30940cb020 0x7f309415e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3099ad4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30940cba70 con 0x7f309415dfd0 2024-03-16T19:29:27.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3099ad4640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309406b750 con 0x7f30940cb020 2024-03-16T19:29:27.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3099ad4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309406b140 con 0x7f30940c7560 2024-03-16T19:29:27.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3093fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f30940cb020 0x7f309415e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f30937fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f309415dfd0 0x7f309406bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40214/0 (socket says 172.21.15.172:40214) 2024-03-16T19:29:27.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f30937fe640 1 -- 172.21.15.172:0/4171454483 learned_addr learned my addr 172.21.15.172:0/4171454483 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:27.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3093fff640 1 -- 172.21.15.172:0/4171454483 >> v2:172.21.15.174:3300/0 conn(0x7f30940c7560 msgr2=0x7f30940caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:27.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3093fff640 1 --2- 172.21.15.172:0/4171454483 >> v2:172.21.15.174:3300/0 conn(0x7f30940c7560 0x7f30940caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3093fff640 1 -- 172.21.15.172:0/4171454483 >> v1:172.21.15.172:6789/0 conn(0x7f309415dfd0 legacy=0x7f309406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:27.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f30927fc640 1 -- 172.21.15.172:0/4171454483 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274750751 0 0) 0x7f3084001110 con 0x7f309415dfd0 2024-03-16T19:29:27.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.466+0000 7f3093fff640 1 -- 172.21.15.172:0/4171454483 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30940cb4b0 con 0x7f30940cb020 2024-03-16T19:29:27.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.467+0000 7f3093fff640 1 --2- 172.21.15.172:0/4171454483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940cb020 0x7f309415e3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f309408d980 tx=0x7f308c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc54c58111bc876 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:27.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.467+0000 7f30927fc640 1 -- 172.21.15.172:0/4171454483 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c02cd30 con 0x7f30940cb020 2024-03-16T19:29:27.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.467+0000 7f30927fc640 1 -- 172.21.15.172:0/4171454483 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c02ce90 con 0x7f30940cb020 2024-03-16T19:29:27.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.467+0000 7f30927fc640 1 -- 172.21.15.172:0/4171454483 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c034660 con 0x7f30940cb020 2024-03-16T19:29:27.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.468+0000 7f3099ad4640 1 -- 172.21.15.172:0/4171454483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940cb020 msgr2=0x7f309415e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:27.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.468+0000 7f3099ad4640 1 --2- 172.21.15.172:0/4171454483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940cb020 0x7f309415e3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f309408d980 tx=0x7f308c02bf80 comp rx=0 tx=0).stop 2024-03-16T19:29:27.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.468+0000 7f3099ad4640 1 -- 172.21.15.172:0/4171454483 shutdown_connections 2024-03-16T19:29:27.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.468+0000 7f3099ad4640 1 --2- 172.21.15.172:0/4171454483 >> v2:172.21.15.174:3300/0 conn(0x7f30940c7560 0x7f30940caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.468+0000 7f3099ad4640 1 --2- 172.21.15.172:0/4171454483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940cb020 0x7f309415e3b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.468+0000 7f3099ad4640 1 -- 172.21.15.172:0/4171454483 >> 172.21.15.172:0/4171454483 conn(0x7f30940d0340 msgr2=0x7f30940d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:27.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.468+0000 7f3099ad4640 1 -- 172.21.15.172:0/4171454483 shutdown_connections 2024-03-16T19:29:27.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.468+0000 7f3099ad4640 1 -- 172.21.15.172:0/4171454483 wait complete. 2024-03-16T19:29:27.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.469+0000 7f3099ad4640 1 Processor -- start 2024-03-16T19:29:27.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.469+0000 7f3099ad4640 1 -- start start 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3099ad4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940c7560 0x7f30941f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3099ad4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f309415dfd0 0x7f3094157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3099ad4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3094152730 0x7f30940c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3099ad4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30941f4b70 con 0x7f30940c7560 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3099ad4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30941f10f0 con 0x7f309415dfd0 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3099ad4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30941f1580 con 0x7f3094152730 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f30937fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940c7560 0x7f30941f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f30937fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940c7560 0x7f30941f4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52444/0 (socket says 172.21.15.172:52444) 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f30937fe640 1 -- 172.21.15.172:0/561279972 learned_addr learned my addr 172.21.15.172:0/561279972 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3092ffd640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f309415dfd0 0x7f3094157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3093fff640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3094152730 0x7f30940c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f30937fe640 1 -- 172.21.15.172:0/561279972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3094152730 msgr2=0x7f30940c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f30937fe640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3094152730 0x7f30940c5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f30937fe640 1 -- 172.21.15.172:0/561279972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f309415dfd0 msgr2=0x7f3094157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f30937fe640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f309415dfd0 0x7f3094157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f30937fe640 1 -- 172.21.15.172:0/561279972 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c009a70 con 0x7f30940c7560 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3093fff640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3094152730 0x7f30940c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f30937fe640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940c7560 0x7f30941f4630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f308400c780 tx=0x7f308400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f307bfff640 1 -- 172.21.15.172:0/561279972 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840072d0 con 0x7f30940c7560 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f307bfff640 1 -- 172.21.15.172:0/561279972 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3084007450 con 0x7f30940c7560 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f307bfff640 1 -- 172.21.15.172:0/561279972 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840047e0 con 0x7f30940c7560 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30940c6af0 con 0x7f30940c7560 2024-03-16T19:29:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.470+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30941f07c0 con 0x7f30940c7560 2024-03-16T19:29:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.471+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30940681c0 con 0x7f30940c7560 2024-03-16T19:29:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.472+0000 7f307bfff640 1 -- 172.21.15.172:0/561279972 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3084004960 con 0x7f30940c7560 2024-03-16T19:29:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.472+0000 7f307bfff640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f308006b3e0 0x7f308006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.472+0000 7f3092ffd640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f308006b3e0 0x7f308006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.472+0000 7f307bfff640 1 -- 172.21.15.172:0/561279972 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3084088fa0 con 0x7f30940c7560 2024-03-16T19:29:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.473+0000 7f3092ffd640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f308006b3e0 0x7f308006d8a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f309414d430 tx=0x7f3088009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:27.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.477+0000 7f307bfff640 1 -- 172.21.15.172:0/561279972 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30840bce10 con 0x7f30940c7560 2024-03-16T19:29:27.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:27.688+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f30940cba70 con 0x7f30940c7560 2024-03-16T19:29:28.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.358+0000 7f307bfff640 1 -- 172.21.15.172:0/561279972 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f30940cba70 con 0x7f30940c7560 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f308006b3e0 msgr2=0x7f308006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f308006b3e0 0x7f308006d8a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f309414d430 tx=0x7f3088009060 comp rx=0 tx=0).stop 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940c7560 msgr2=0x7f30941f4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940c7560 0x7f30941f4630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f308400c780 tx=0x7f308400cc40 comp rx=0 tx=0).stop 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 shutdown_connections 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f308006b3e0 0x7f308006d8a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3094152730 0x7f30940c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f309415dfd0 0x7f3094157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 --2- 172.21.15.172:0/561279972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f30940c7560 0x7f30941f4630 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 >> 172.21.15.172:0/561279972 conn(0x7f30940d0340 msgr2=0x7f30940ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 shutdown_connections 2024-03-16T19:29:28.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.360+0000 7f3099ad4640 1 -- 172.21.15.172:0/561279972 wait complete. 2024-03-16T19:29:28.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-16T19:29:28.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.477+0000 7f39d43fd640 1 Processor -- start 2024-03-16T19:29:28.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.477+0000 7f39d43fd640 1 -- start start 2024-03-16T19:29:28.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d43fd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc15dfd0 0x7f39cc06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:28.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d43fd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f39cc0c7560 0x7f39cc0caae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:28.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d43fd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f39cc0cb020 0x7f39cc15e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:28.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d43fd640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39cc0cba70 con 0x7f39cc0cb020 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d43fd640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39cc06b750 con 0x7f39cc0c7560 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d43fd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc06b140 con 0x7f39cc15dfd0 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d299a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f39cc0cb020 0x7f39cc15e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d299a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f39cc0cb020 0x7f39cc15e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46034/0 (socket says 172.21.15.172:46034) 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d299a640 1 -- 172.21.15.172:0/1816843709 learned_addr learned my addr 172.21.15.172:0/1816843709 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d2199640 1 --2- 172.21.15.172:0/1816843709 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc15dfd0 0x7f39cc06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d1998640 1 --2- 172.21.15.172:0/1816843709 >> v2:172.21.15.174:3300/0 conn(0x7f39cc0c7560 0x7f39cc0caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d299a640 1 -- 172.21.15.172:0/1816843709 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc15dfd0 msgr2=0x7f39cc06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d299a640 1 --2- 172.21.15.172:0/1816843709 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc15dfd0 0x7f39cc06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d299a640 1 -- 172.21.15.172:0/1816843709 >> v2:172.21.15.174:3300/0 conn(0x7f39cc0c7560 msgr2=0x7f39cc0caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d299a640 1 --2- 172.21.15.172:0/1816843709 >> v2:172.21.15.174:3300/0 conn(0x7f39cc0c7560 0x7f39cc0caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d299a640 1 -- 172.21.15.172:0/1816843709 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc06c120 con 0x7f39cc0cb020 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.478+0000 7f39d2199640 1 --2- 172.21.15.172:0/1816843709 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc15dfd0 0x7f39cc06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:28.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.479+0000 7f39d299a640 1 --2- 172.21.15.172:0/1816843709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc0cb020 0x7f39cc15e3b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f39cc06d170 tx=0x7f39c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2940d3b899154a72 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:28.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.479+0000 7f39d1197640 1 -- 172.21.15.172:0/1816843709 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c402cb00 con 0x7f39cc0cb020 2024-03-16T19:29:28.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.479+0000 7f39d1197640 1 -- 172.21.15.172:0/1816843709 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c402cc60 con 0x7f39cc0cb020 2024-03-16T19:29:28.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.479+0000 7f39d1197640 1 -- 172.21.15.172:0/1816843709 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c40343f0 con 0x7f39cc0cb020 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.480+0000 7f39d43fd640 1 -- 172.21.15.172:0/1816843709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc0cb020 msgr2=0x7f39cc15e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.480+0000 7f39d43fd640 1 --2- 172.21.15.172:0/1816843709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc0cb020 0x7f39cc15e3b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f39cc06d170 tx=0x7f39c402bf60 comp rx=0 tx=0).stop 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.480+0000 7f39d43fd640 1 -- 172.21.15.172:0/1816843709 shutdown_connections 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.480+0000 7f39d43fd640 1 --2- 172.21.15.172:0/1816843709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc0cb020 0x7f39cc15e3b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.480+0000 7f39d43fd640 1 --2- 172.21.15.172:0/1816843709 >> v2:172.21.15.174:3300/0 conn(0x7f39cc0c7560 0x7f39cc0caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.480+0000 7f39d43fd640 1 --2- 172.21.15.172:0/1816843709 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc15dfd0 0x7f39cc06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.480+0000 7f39d43fd640 1 -- 172.21.15.172:0/1816843709 >> 172.21.15.172:0/1816843709 conn(0x7f39cc0d0340 msgr2=0x7f39cc0d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.480+0000 7f39d43fd640 1 -- 172.21.15.172:0/1816843709 shutdown_connections 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.480+0000 7f39d43fd640 1 -- 172.21.15.172:0/1816843709 wait complete. 2024-03-16T19:29:28.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.481+0000 7f39d43fd640 1 Processor -- start 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.481+0000 7f39d43fd640 1 -- start start 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d43fd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39cc0c7560 0x7f39cc14df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d43fd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc15dfd0 0x7f39cc153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d43fd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc154180 0x7f39cc156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d43fd640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc157420 con 0x7f39cc15dfd0 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d43fd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc155ce0 con 0x7f39cc0c7560 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d43fd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc154630 con 0x7f39cc154180 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d1998640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc15dfd0 0x7f39cc153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d1998640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc15dfd0 0x7f39cc153c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46050/0 (socket says 172.21.15.172:46050) 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d1998640 1 -- 172.21.15.172:0/3828715004 learned_addr learned my addr 172.21.15.172:0/3828715004 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d2199640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39cc0c7560 0x7f39cc14df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:28.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d299a640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc154180 0x7f39cc156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d1998640 1 -- 172.21.15.172:0/3828715004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc154180 msgr2=0x7f39cc156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d1998640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc154180 0x7f39cc156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d1998640 1 -- 172.21.15.172:0/3828715004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39cc0c7560 msgr2=0x7f39cc14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d1998640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39cc0c7560 0x7f39cc14df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d1998640 1 -- 172.21.15.172:0/3828715004 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c4009b30 con 0x7f39cc15dfd0 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d2199640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39cc0c7560 0x7f39cc14df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d1998640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc15dfd0 0x7f39cc153c40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00c820 tx=0x7f39bc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39c2ffd640 1 -- 172.21.15.172:0/3828715004 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc004020 con 0x7f39cc15dfd0 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39c2ffd640 1 -- 172.21.15.172:0/3828715004 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc004180 con 0x7f39cc15dfd0 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39c2ffd640 1 -- 172.21.15.172:0/3828715004 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc007830 con 0x7f39cc15dfd0 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.482+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39cc1553d0 con 0x7f39cc15dfd0 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.483+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39cc14ea20 con 0x7f39cc15dfd0 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.483+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39cc155e20 con 0x7f39cc15dfd0 2024-03-16T19:29:28.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.484+0000 7f39c2ffd640 1 -- 172.21.15.172:0/3828715004 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f39bc007990 con 0x7f39cc15dfd0 2024-03-16T19:29:28.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.484+0000 7f39c2ffd640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f39b006b3b0 0x7f39b006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:28.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.484+0000 7f39d2199640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f39b006b3b0 0x7f39b006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:28.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.484+0000 7f39c2ffd640 1 -- 172.21.15.172:0/3828715004 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39bc0897e0 con 0x7f39cc15dfd0 2024-03-16T19:29:28.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.485+0000 7f39d2199640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f39b006b3b0 0x7f39b006d870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f39c800a990 tx=0x7f39c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:28.469 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.489+0000 7f39c2ffd640 1 -- 172.21.15.172:0/3828715004 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39cc155e20 con 0x7f39cc15dfd0 2024-03-16T19:29:28.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:28.699+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f39cc155e20 con 0x7f39cc15dfd0 2024-03-16T19:29:29.345 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.364+0000 7f39c2ffd640 1 -- 172.21.15.172:0/3828715004 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v7) v1 ==== 115+0+0 (secure 0 0 0) 0x7f39cc155e20 con 0x7f39cc15dfd0 2024-03-16T19:29:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f39b006b3b0 msgr2=0x7f39b006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f39b006b3b0 0x7f39b006d870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f39c800a990 tx=0x7f39c8008040 comp rx=0 tx=0).stop 2024-03-16T19:29:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc15dfd0 msgr2=0x7f39cc153c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc15dfd0 0x7f39cc153c40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00c820 tx=0x7f39bc00cce0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 shutdown_connections 2024-03-16T19:29:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f39b006b3b0 0x7f39b006d870 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39cc154180 0x7f39cc156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc15dfd0 0x7f39cc153c40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 --2- 172.21.15.172:0/3828715004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39cc0c7560 0x7f39cc14df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 >> 172.21.15.172:0/3828715004 conn(0x7f39cc0d0340 msgr2=0x7f39cc0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 shutdown_connections 2024-03-16T19:29:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.367+0000 7f39d43fd640 1 -- 172.21.15.172:0/3828715004 wait complete. 2024-03-16T19:29:29.361 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-16T19:29:29.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:29:29.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.486+0000 7f346f71e640 1 Processor -- start 2024-03-16T19:29:29.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.486+0000 7f346f71e640 1 -- start start 2024-03-16T19:29:29.469 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.487+0000 7f346f71e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3468154cb0 0x7f346806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.469 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.487+0000 7f346f71e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f346806fe10 0x7f346815de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.469 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.487+0000 7f346f71e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346815e340 con 0x7f34680c8ba0 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.487+0000 7f346f71e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34680cbba0 con 0x7f346806fe10 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.487+0000 7f346f71e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34680c91f0 con 0x7f3468154cb0 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.487+0000 7f346ccb9640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f346806fe10 0x7f346815de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.487+0000 7f346ccb9640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f346806fe10 0x7f346815de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46064/0 (socket says 172.21.15.172:46064) 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.487+0000 7f346ccb9640 1 -- 172.21.15.172:0/233003873 learned_addr learned my addr 172.21.15.172:0/233003873 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.488+0000 7f346d4ba640 1 --2- 172.21.15.172:0/233003873 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3468154cb0 0x7f346806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.488+0000 7f346ccb9640 1 -- 172.21.15.172:0/233003873 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3468154cb0 msgr2=0x7f346806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.488+0000 7f346ccb9640 1 --2- 172.21.15.172:0/233003873 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3468154cb0 0x7f346806dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.488+0000 7f346ccb9640 1 -- 172.21.15.172:0/233003873 >> v1:172.21.15.174:6789/0 conn(0x7f34680c8ba0 legacy=0x7f34680cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.488+0000 7f346ccb9640 1 -- 172.21.15.172:0/233003873 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346806c180 con 0x7f346806fe10 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.488+0000 7f346ccb9640 1 --2- 172.21.15.172:0/233003873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f346806fe10 0x7f346815de00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3458009870 tx=0x7f345802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7828ce369d5342c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f345ffff640 1 -- 172.21.15.172:0/233003873 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458031740 con 0x7f346806fe10 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f345ffff640 1 -- 172.21.15.172:0/233003873 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34580318a0 con 0x7f346806fe10 2024-03-16T19:29:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f345ffff640 1 -- 172.21.15.172:0/233003873 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458031c30 con 0x7f346806fe10 2024-03-16T19:29:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f346f71e640 1 -- 172.21.15.172:0/233003873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f346806fe10 msgr2=0x7f346815de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f346f71e640 1 --2- 172.21.15.172:0/233003873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f346806fe10 0x7f346815de00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3458009870 tx=0x7f345802b7e0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f346f71e640 1 -- 172.21.15.172:0/233003873 shutdown_connections 2024-03-16T19:29:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f346f71e640 1 --2- 172.21.15.172:0/233003873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f346806fe10 0x7f346815de00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f346f71e640 1 --2- 172.21.15.172:0/233003873 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3468154cb0 0x7f346806dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f346f71e640 1 -- 172.21.15.172:0/233003873 >> 172.21.15.172:0/233003873 conn(0x7f3468070860 msgr2=0x7f3468070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f346f71e640 1 -- 172.21.15.172:0/233003873 shutdown_connections 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.489+0000 7f346f71e640 1 -- 172.21.15.172:0/233003873 wait complete. 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.490+0000 7f346f71e640 1 Processor -- start 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.490+0000 7f346f71e640 1 -- start start 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346f71e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f346806fe10 0x7f346814c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346f71e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34680c8ba0 0x7f34681f0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346f71e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3468154cb0 0x7f34681f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346f71e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34680cd7e0 con 0x7f34680c8ba0 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346f71e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34681e3c60 con 0x7f3468154cb0 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346f71e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34681e4190 con 0x7f346806fe10 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346ccb9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34680c8ba0 0x7f34681f0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346ccb9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34680c8ba0 0x7f34681f0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46068/0 (socket says 172.21.15.172:46068) 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346ccb9640 1 -- 172.21.15.172:0/1996388399 learned_addr learned my addr 172.21.15.172:0/1996388399 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346d4ba640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f346806fe10 0x7f346814c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346dcbb640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3468154cb0 0x7f34681f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346ccb9640 1 -- 172.21.15.172:0/1996388399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f346806fe10 msgr2=0x7f346814c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346ccb9640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f346806fe10 0x7f346814c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346ccb9640 1 -- 172.21.15.172:0/1996388399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3468154cb0 msgr2=0x7f34681f0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346ccb9640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3468154cb0 0x7f34681f0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.491+0000 7f346ccb9640 1 -- 172.21.15.172:0/1996388399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34580094d0 con 0x7f34680c8ba0 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.492+0000 7f346ccb9640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34680c8ba0 0x7f34681f0770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3458009870 tx=0x7f34580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.492+0000 7f345dffb640 1 -- 172.21.15.172:0/1996388399 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458033d60 con 0x7f34680c8ba0 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.492+0000 7f345dffb640 1 -- 172.21.15.172:0/1996388399 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345803c400 con 0x7f34680c8ba0 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.492+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34681e6970 con 0x7f34680c8ba0 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.492+0000 7f345dffb640 1 -- 172.21.15.172:0/1996388399 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345803c7b0 con 0x7f34680c8ba0 2024-03-16T19:29:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.492+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34681e3eb0 con 0x7f34680c8ba0 2024-03-16T19:29:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.493+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34680681c0 con 0x7f34680c8ba0 2024-03-16T19:29:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.493+0000 7f345dffb640 1 -- 172.21.15.172:0/1996388399 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f345803a040 con 0x7f34680c8ba0 2024-03-16T19:29:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.494+0000 7f345dffb640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f344c06b3b0 0x7f344c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.494+0000 7f345dffb640 1 -- 172.21.15.172:0/1996388399 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34580ab7a0 con 0x7f34680c8ba0 2024-03-16T19:29:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.494+0000 7f346d4ba640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f344c06b3b0 0x7f344c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.494+0000 7f346d4ba640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f344c06b3b0 0x7f344c06d870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3464008e50 tx=0x7f34640083a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:29.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.498+0000 7f345dffb640 1 -- 172.21.15.172:0/1996388399 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3458076490 con 0x7f34680c8ba0 2024-03-16T19:29:29.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.705+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f346806fbf0 con 0x7f34680c8ba0 2024-03-16T19:29:29.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.705+0000 7f345dffb640 1 -- 172.21.15.172:0/1996388399 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6556 (secure 0 0 0) 0x7f345802f090 con 0x7f34680c8ba0 2024-03-16T19:29:29.686 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:29.686 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1028682606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1028682606},{"type":"v1","addr":"172.21.15.174:6833","nonce":1028682606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6836/902696086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":902696086},{"type":"v1","addr":"172.21.15.174:6836","nonce":902696086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/864851052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":864851052},{"type":"v1","addr":"172.21.15.172:6836","nonce":864851052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/420882750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":420882750},{"type":"v1","addr":"172.21.15.172:6837","nonce":420882750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/2853574742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":2853574742},{"type":"v1","addr":"172.21.15.172:6839","nonce":2853574742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:29:25.674964+0000","modified":"2024-03-16T19:29:28.700853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.174:6837/2442775894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6835","nonce":2442775894},{"type":"v1","addr":"172.21.15.174:6837","nonce":2442775894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-16T19:29:29.687 INFO:teuthology.orchestra.run.smithi172.stderr:dumped fsmap epoch 7 2024-03-16T19:29:29.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f344c06b3b0 msgr2=0x7f344c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f344c06b3b0 0x7f344c06d870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3464008e50 tx=0x7f34640083a0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34680c8ba0 msgr2=0x7f34681f0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34680c8ba0 0x7f34681f0770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3458009870 tx=0x7f34580046c0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 shutdown_connections 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3468154cb0 0x7f34681f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34680c8ba0 0x7f34681f0770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f344c06b3b0 0x7f344c06d870 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 --2- 172.21.15.172:0/1996388399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f346806fe10 0x7f346814c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 >> 172.21.15.172:0/1996388399 conn(0x7f3468070860 msgr2=0x7f346815bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 shutdown_connections 2024-03-16T19:29:29.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.707+0000 7f346f71e640 1 -- 172.21.15.172:0/1996388399 wait complete. 2024-03-16T19:29:29.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:29.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.819+0000 7f9a1c6c3640 1 Processor -- start 2024-03-16T19:29:29.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.819+0000 7f9a1c6c3640 1 -- start start 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1c6c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a1415b4e0 0x7f9a14166310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1c6c3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a14158560 con 0x7f9a14158180 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1c6c3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a140ca0a0 con 0x7f9a1406fe50 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1c6c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1406cf90 con 0x7f9a1415b4e0 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1a45f640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9a14158180 0x7f9a140c9210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56082/0 (socket says 172.21.15.172:56082) 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1a45f640 1 -- 172.21.15.172:0/4001805165 learned_addr learned my addr 172.21.15.172:0/4001805165 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1ac60640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a1415b4e0 0x7f9a14166310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060195929 0 0) 0x7f9a14158560 con 0x7f9a14158180 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a00003520 con 0x7f9a14158180 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2167162858 0 0) 0x7f9a00003520 con 0x7f9a14158180 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a1415b4e0 msgr2=0x7f9a14166310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a1415b4e0 0x7f9a14166310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 >> v1:172.21.15.174:6789/0 conn(0x7f9a1406fe50 legacy=0x7f9a140ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a140c8ec0 con 0x7f9a14158180 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f9a10001670 con 0x7f9a14158180 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 >> v1:172.21.15.172:6789/0 conn(0x7f9a14158180 legacy=0x7f9a140c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a000039a0 0x7f9a00003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a1415b4e0 0x7f9a000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a00004d00 0x7f9a00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1406cf90 con 0x7f9a000039a0 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0000b980 con 0x7f9a00004d00 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0000bb40 con 0x7f9a1415b4e0 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.820+0000 7f9a1a45f640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a000039a0 0x7f9a00003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.821+0000 7f9a19c5e640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a1415b4e0 0x7f9a000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.821+0000 7f9a1ac60640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a00004d00 0x7f9a00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.821+0000 7f9a1a45f640 1 -- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a1415b4e0 msgr2=0x7f9a000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.821+0000 7f9a1a45f640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a1415b4e0 0x7f9a000047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.821+0000 7f9a1a45f640 1 -- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a00004d00 msgr2=0x7f9a00005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.821+0000 7f9a1a45f640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a00004d00 0x7f9a00005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.821+0000 7f9a1a45f640 1 -- 172.21.15.172:0/4001805165 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a00003520 con 0x7f9a000039a0 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.821+0000 7f9a1a45f640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a000039a0 0x7f9a00003d80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000c190 tx=0x7f9a000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4990d11261df63 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10005ad0 con 0x7f9a000039a0 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10005c50 con 0x7f9a000039a0 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1945d640 1 -- 172.21.15.172:0/4001805165 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1002c7e0 con 0x7f9a000039a0 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/4001805165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a000039a0 msgr2=0x7f9a00003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1c6c3640 1 --2- 172.21.15.172:0/4001805165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a000039a0 0x7f9a00003d80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000c190 tx=0x7f9a000057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1ac60640 1 -- 172.21.15.172:0/4001805165 reap_dead start 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/4001805165 shutdown_connections 2024-03-16T19:29:29.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/4001805165 >> 172.21.15.172:0/4001805165 conn(0x7f9a14066c20 msgr2=0x7f9a14067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:29.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/4001805165 shutdown_connections 2024-03-16T19:29:29.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.822+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/4001805165 wait complete. 2024-03-16T19:29:29.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.823+0000 7f9a1c6c3640 1 Processor -- start 2024-03-16T19:29:29.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.823+0000 7f9a1c6c3640 1 -- start start 2024-03-16T19:29:29.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1c6c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a000039a0 0x7f9a140ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1c6c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a1414f010 0x7f9a14150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1c6c3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a14150680 0x7f9a141f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1c6c3640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1406c5b0 con 0x7f9a14150680 2024-03-16T19:29:29.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1c6c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a141f40e0 con 0x7f9a1414f010 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1c6c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1414f7f0 con 0x7f9a000039a0 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1ac60640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a14150680 0x7f9a141f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1ac60640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a14150680 0x7f9a141f3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46078/0 (socket says 172.21.15.172:46078) 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1ac60640 1 -- 172.21.15.172:0/2087113495 learned_addr learned my addr 172.21.15.172:0/2087113495 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1a45f640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a000039a0 0x7f9a140ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a19c5e640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a1414f010 0x7f9a14150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1ac60640 1 -- 172.21.15.172:0/2087113495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a000039a0 msgr2=0x7f9a140ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1ac60640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a000039a0 0x7f9a140ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1ac60640 1 -- 172.21.15.172:0/2087113495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a1414f010 msgr2=0x7f9a14150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1ac60640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a1414f010 0x7f9a14150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.824+0000 7f9a1ac60640 1 -- 172.21.15.172:0/2087113495 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1414fad0 con 0x7f9a14150680 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.825+0000 7f9a1ac60640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a14150680 0x7f9a141f3670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9a140b36a0 tx=0x7f9a0c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.825+0000 7f9a0affd640 1 -- 172.21.15.172:0/2087113495 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c00e040 con 0x7f9a14150680 2024-03-16T19:29:29.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.825+0000 7f9a0affd640 1 -- 172.21.15.172:0/2087113495 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0c00f040 con 0x7f9a14150680 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.825+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1414d7e0 con 0x7f9a14150680 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.825+0000 7f9a0affd640 1 -- 172.21.15.172:0/2087113495 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c015430 con 0x7f9a14150680 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.825+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1414dfe0 con 0x7f9a14150680 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.826+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1414dd00 con 0x7f9a14150680 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.826+0000 7f9a0affd640 1 -- 172.21.15.172:0/2087113495 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9a0c015590 con 0x7f9a14150680 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.826+0000 7f9a0affd640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99f806b3b0 0x7f99f806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.827+0000 7f9a1a45f640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99f806b3b0 0x7f99f806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.827+0000 7f9a0affd640 1 -- 172.21.15.172:0/2087113495 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a0c08a5c0 con 0x7f9a14150680 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.827+0000 7f9a1a45f640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99f806b3b0 0x7f99f806d870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9a10034000 tx=0x7f9a10007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:29.831+0000 7f9a0affd640 1 -- 172.21.15.172:0/2087113495 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a0c0552b0 con 0x7f9a14150680 2024-03-16T19:29:29.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.003+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a140681c0 con 0x7f9a14150680 2024-03-16T19:29:29.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.004+0000 7f9a0affd640 1 -- 172.21.15.172:0/2087113495 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f9a0c058970 con 0x7f9a14150680 2024-03-16T19:29:29.985 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:29.985 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":21,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:27.336190+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:29.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99f806b3b0 msgr2=0x7f99f806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99f806b3b0 0x7f99f806d870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9a10034000 tx=0x7f9a10007000 comp rx=0 tx=0).stop 2024-03-16T19:29:29.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a14150680 msgr2=0x7f9a141f3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:29.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a14150680 0x7f9a141f3670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9a140b36a0 tx=0x7f9a0c00f580 comp rx=0 tx=0).stop 2024-03-16T19:29:29.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 shutdown_connections 2024-03-16T19:29:29.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a14150680 0x7f9a141f3670 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a1414f010 0x7f9a14150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99f806b3b0 0x7f99f806d870 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 --2- 172.21.15.172:0/2087113495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a000039a0 0x7f9a140ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:29.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 >> 172.21.15.172:0/2087113495 conn(0x7f9a14066c20 msgr2=0x7f9a141573b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:29.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 shutdown_connections 2024-03-16T19:29:29.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.006+0000 7f9a1c6c3640 1 -- 172.21.15.172:0/2087113495 wait complete. 2024-03-16T19:29:29.998 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-16T19:29:29.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:29:30.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.119+0000 7f3702223640 1 Processor -- start 2024-03-16T19:29:30.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.119+0000 7f3702223640 1 -- start start 2024-03-16T19:29:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f3702223640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f36fc0cc780 0x7f36fc0cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f3702223640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc15b880 con 0x7f36fc154c90 2024-03-16T19:29:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f3702223640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc155070 con 0x7f36fc0c51b0 2024-03-16T19:29:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f3702223640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36fc0c6ee0 con 0x7f36fc0cc780 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f36fb7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f36fc154c90 0x7f36fc15e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56096/0 (socket says 172.21.15.172:56096) 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f36fb7fe640 1 -- 172.21.15.172:0/2959502936 learned_addr learned my addr 172.21.15.172:0/2959502936 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f36faffd640 1 --2- 172.21.15.172:0/2959502936 >> v2:172.21.15.174:3300/0 conn(0x7f36fc0cc780 0x7f36fc0cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f36fa7fc640 1 -- 172.21.15.172:0/2959502936 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790187460 0 0) 0x7f36ec001110 con 0x7f36fc154c90 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f36fa7fc640 1 -- 172.21.15.172:0/2959502936 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e4003520 con 0x7f36fc154c90 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f36fa7fc640 1 -- 172.21.15.172:0/2959502936 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116035957 0 0) 0x7f36e4003520 con 0x7f36fc154c90 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f36faffd640 1 -- 172.21.15.172:0/2959502936 >> v1:172.21.15.174:6789/0 conn(0x7f36fc0c51b0 legacy=0x7f36fc1651c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.120+0000 7f36faffd640 1 -- 172.21.15.172:0/2959502936 >> v1:172.21.15.172:6789/0 conn(0x7f36fc154c90 legacy=0x7f36fc15e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.121+0000 7f36faffd640 1 -- 172.21.15.172:0/2959502936 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc15ad00 con 0x7f36fc0cc780 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.121+0000 7f36faffd640 1 --2- 172.21.15.172:0/2959502936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36fc0cc780 0x7f36fc0cd1c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36f00097a0 tx=0x7f36f002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c89bec54c704ec30 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.121+0000 7f36fa7fc640 1 -- 172.21.15.172:0/2959502936 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f002f050 con 0x7f36fc0cc780 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.121+0000 7f36fa7fc640 1 -- 172.21.15.172:0/2959502936 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f0031500 con 0x7f36fc0cc780 2024-03-16T19:29:30.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.121+0000 7f36fa7fc640 1 -- 172.21.15.172:0/2959502936 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ec001800 con 0x7f36fc0cc780 2024-03-16T19:29:30.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.122+0000 7f3702223640 1 -- 172.21.15.172:0/2959502936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36fc0cc780 msgr2=0x7f36fc0cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.122+0000 7f3702223640 1 --2- 172.21.15.172:0/2959502936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36fc0cc780 0x7f36fc0cd1c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36f00097a0 tx=0x7f36f002b820 comp rx=0 tx=0).stop 2024-03-16T19:29:30.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.122+0000 7f3702223640 1 -- 172.21.15.172:0/2959502936 shutdown_connections 2024-03-16T19:29:30.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.122+0000 7f3702223640 1 --2- 172.21.15.172:0/2959502936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36fc0cc780 0x7f36fc0cd1c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.122+0000 7f3702223640 1 -- 172.21.15.172:0/2959502936 >> 172.21.15.172:0/2959502936 conn(0x7f36fc067040 msgr2=0x7f36fc067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:30.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.122+0000 7f3702223640 1 -- 172.21.15.172:0/2959502936 shutdown_connections 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.122+0000 7f3702223640 1 -- 172.21.15.172:0/2959502936 wait complete. 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.123+0000 7f3702223640 1 Processor -- start 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.123+0000 7f3702223640 1 -- start start 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f3702223640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36fc0c51b0 0x7f36fc1f4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f3702223640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36fc0cc780 0x7f36fc0ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f3702223640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36fc154c90 0x7f36fc1524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f3702223640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc0d08c0 con 0x7f36fc0cc780 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f3702223640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc1f4a90 con 0x7f36fc0c51b0 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f3702223640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc1f0ee0 con 0x7f36fc154c90 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36faffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36fc0cc780 0x7f36fc0ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36faffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36fc0cc780 0x7f36fc0ce1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46094/0 (socket says 172.21.15.172:46094) 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36faffd640 1 -- 172.21.15.172:0/438493711 learned_addr learned my addr 172.21.15.172:0/438493711 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36fb7fe640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36fc0c51b0 0x7f36fc1f4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36fbfff640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36fc154c90 0x7f36fc1524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36faffd640 1 -- 172.21.15.172:0/438493711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36fc154c90 msgr2=0x7f36fc1524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36faffd640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36fc154c90 0x7f36fc1524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36faffd640 1 -- 172.21.15.172:0/438493711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36fc0c51b0 msgr2=0x7f36fc1f4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36faffd640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36fc0c51b0 0x7f36fc1f4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36faffd640 1 -- 172.21.15.172:0/438493711 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f0009400 con 0x7f36fc0cc780 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.124+0000 7f36faffd640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36fc0cc780 0x7f36fc0ce1e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f36f0000c00 tx=0x7f36f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.125+0000 7f36e3fff640 1 -- 172.21.15.172:0/438493711 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0033980 con 0x7f36fc0cc780 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.125+0000 7f36e3fff640 1 -- 172.21.15.172:0/438493711 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f0033b00 con 0x7f36fc0cc780 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.125+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36fc1f05d0 con 0x7f36fc0cc780 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.125+0000 7f36e3fff640 1 -- 172.21.15.172:0/438493711 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f003d470 con 0x7f36fc0cc780 2024-03-16T19:29:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.125+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36fc0d0da0 con 0x7f36fc0cc780 2024-03-16T19:29:30.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.126+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36fc15b560 con 0x7f36fc0cc780 2024-03-16T19:29:30.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.126+0000 7f36e3fff640 1 -- 172.21.15.172:0/438493711 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f36f003a040 con 0x7f36fc0cc780 2024-03-16T19:29:30.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.126+0000 7f36e3fff640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e806b3b0 0x7f36e806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.126+0000 7f36e3fff640 1 -- 172.21.15.172:0/438493711 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36f00ab6d0 con 0x7f36fc0cc780 2024-03-16T19:29:30.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.126+0000 7f36fb7fe640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e806b3b0 0x7f36e806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.127+0000 7f36fb7fe640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e806b3b0 0x7f36e806d870 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f36ec005880 tx=0x7f36ec0011d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:30.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.131+0000 7f36e3fff640 1 -- 172.21.15.172:0/438493711 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36f0076470 con 0x7f36fc0cc780 2024-03-16T19:29:30.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.300+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36fc15b2c0 con 0x7f36e806b3b0 2024-03-16T19:29:30.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.306+0000 7f36e3fff640 1 -- 172.21.15.172:0/438493711 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251847 (secure 0 0 0) 0x7f36ec00f880 con 0x7f36e806b3b0 2024-03-16T19:29:30.290 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:30.291 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:29:30.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e806b3b0 msgr2=0x7f36e806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e806b3b0 0x7f36e806d870 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f36ec005880 tx=0x7f36ec0011d0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36fc0cc780 msgr2=0x7f36fc0ce1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36fc0cc780 0x7f36fc0ce1e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f36f0000c00 tx=0x7f36f00046c0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 shutdown_connections 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e806b3b0 0x7f36e806d870 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36fc154c90 0x7f36fc1524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f36fc0cc780 0x7f36fc0ce1e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 --2- 172.21.15.172:0/438493711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36fc0c51b0 0x7f36fc1f4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 >> 172.21.15.172:0/438493711 conn(0x7f36fc067040 msgr2=0x7f36fc0cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 shutdown_connections 2024-03-16T19:29:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.309+0000 7f3702223640 1 -- 172.21.15.172:0/438493711 wait complete. 2024-03-16T19:29:30.301 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-16T19:29:30.200302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218976,"kb_used_data":2796,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755744,"statfs":{"total":773094113280,"available":772869881856,"internally_reserved":0,"allocated":2863104,"data_stored":1754076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.876050"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344501+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:29:27.344501+0000","last_peered":"2024-03-16T19:29:27.344501+0000","last_clean":"2024-03-16T19:29:27.344501+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:29:27.344501+0000","last_undegraded":"2024-03-16T19:29:27.344501+0000","last_fullsized":"2024-03-16T19:29:27.344501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339631+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:29:27.339631+0000","last_peered":"2024-03-16T19:29:27.339631+0000","last_clean":"2024-03-16T19:29:27.339631+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:29:27.339631+0000","last_undegraded":"2024-03-16T19:29:27.339631+0000","last_fullsized":"2024-03-16T19:29:27.339631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062822+0000","last_change":"2024-03-16T19:29:26.347979+0000","last_active":"2024-03-16T19:29:28.062822+0000","last_peered":"2024-03-16T19:29:28.062822+0000","last_clean":"2024-03-16T19:29:28.062822+0000","last_became_active":"2024-03-16T19:29:26.347711+0000","last_became_peered":"2024-03-16T19:29:26.347711+0000","last_unstale":"2024-03-16T19:29:28.062822+0000","last_undegraded":"2024-03-16T19:29:28.062822+0000","last_fullsized":"2024-03-16T19:29:28.062822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:08.017182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343558+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:29:27.343558+0000","last_peered":"2024-03-16T19:29:27.343558+0000","last_clean":"2024-03-16T19:29:27.343558+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:29:27.343558+0000","last_undegraded":"2024-03-16T19:29:27.343558+0000","last_fullsized":"2024-03-16T19:29:27.343558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344270+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:29:27.344270+0000","last_peered":"2024-03-16T19:29:27.344270+0000","last_clean":"2024-03-16T19:29:27.344270+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:29:27.344270+0000","last_undegraded":"2024-03-16T19:29:27.344270+0000","last_fullsized":"2024-03-16T19:29:27.344270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343104+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:29:27.343104+0000","last_peered":"2024-03-16T19:29:27.343104+0000","last_clean":"2024-03-16T19:29:27.343104+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:29:27.343104+0000","last_undegraded":"2024-03-16T19:29:27.343104+0000","last_fullsized":"2024-03-16T19:29:27.343104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344269+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:29:27.344269+0000","last_peered":"2024-03-16T19:29:27.344269+0000","last_clean":"2024-03-16T19:29:27.344269+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:29:27.344269+0000","last_undegraded":"2024-03-16T19:29:27.344269+0000","last_fullsized":"2024-03-16T19:29:27.344269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561737+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:29:27.561737+0000","last_peered":"2024-03-16T19:29:27.561737+0000","last_clean":"2024-03-16T19:29:27.561737+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:29:27.561737+0000","last_undegraded":"2024-03-16T19:29:27.561737+0000","last_fullsized":"2024-03-16T19:29:27.561737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344497+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:29:27.344497+0000","last_peered":"2024-03-16T19:29:27.344497+0000","last_clean":"2024-03-16T19:29:27.344497+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:29:27.344497+0000","last_undegraded":"2024-03-16T19:29:27.344497+0000","last_fullsized":"2024-03-16T19:29:27.344497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063286+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:29:28.063286+0000","last_peered":"2024-03-16T19:29:28.063286+0000","last_clean":"2024-03-16T19:29:28.063286+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:29:28.063286+0000","last_undegraded":"2024-03-16T19:29:28.063286+0000","last_fullsized":"2024-03-16T19:29:28.063286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062996+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:29:28.062996+0000","last_peered":"2024-03-16T19:29:28.062996+0000","last_clean":"2024-03-16T19:29:28.062996+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:29:28.062996+0000","last_undegraded":"2024-03-16T19:29:28.062996+0000","last_fullsized":"2024-03-16T19:29:28.062996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063177+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:29:28.063177+0000","last_peered":"2024-03-16T19:29:28.063177+0000","last_clean":"2024-03-16T19:29:28.063177+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:29:28.063177+0000","last_undegraded":"2024-03-16T19:29:28.063177+0000","last_fullsized":"2024-03-16T19:29:28.063177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561930+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:29:27.561930+0000","last_peered":"2024-03-16T19:29:27.561930+0000","last_clean":"2024-03-16T19:29:27.561930+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:29:27.561930+0000","last_undegraded":"2024-03-16T19:29:27.561930+0000","last_fullsized":"2024-03-16T19:29:27.561930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062462+0000","last_change":"2024-03-16T19:29:26.349644+0000","last_active":"2024-03-16T19:29:28.062462+0000","last_peered":"2024-03-16T19:29:28.062462+0000","last_clean":"2024-03-16T19:29:28.062462+0000","last_became_active":"2024-03-16T19:29:26.349397+0000","last_became_peered":"2024-03-16T19:29:26.349397+0000","last_unstale":"2024-03-16T19:29:28.062462+0000","last_undegraded":"2024-03-16T19:29:28.062462+0000","last_fullsized":"2024-03-16T19:29:28.062462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:47:20.566393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339397+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:29:27.339397+0000","last_peered":"2024-03-16T19:29:27.339397+0000","last_clean":"2024-03-16T19:29:27.339397+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:29:27.339397+0000","last_undegraded":"2024-03-16T19:29:27.339397+0000","last_fullsized":"2024-03-16T19:29:27.339397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344311+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:29:27.344311+0000","last_peered":"2024-03-16T19:29:27.344311+0000","last_clean":"2024-03-16T19:29:27.344311+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:29:27.344311+0000","last_undegraded":"2024-03-16T19:29:27.344311+0000","last_fullsized":"2024-03-16T19:29:27.344311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062569+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:29:28.062569+0000","last_peered":"2024-03-16T19:29:28.062569+0000","last_clean":"2024-03-16T19:29:28.062569+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:29:28.062569+0000","last_undegraded":"2024-03-16T19:29:28.062569+0000","last_fullsized":"2024-03-16T19:29:28.062569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344648+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:29:27.344648+0000","last_peered":"2024-03-16T19:29:27.344648+0000","last_clean":"2024-03-16T19:29:27.344648+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:29:27.344648+0000","last_undegraded":"2024-03-16T19:29:27.344648+0000","last_fullsized":"2024-03-16T19:29:27.344648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063112+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:29:28.063112+0000","last_peered":"2024-03-16T19:29:28.063112+0000","last_clean":"2024-03-16T19:29:28.063112+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:29:28.063112+0000","last_undegraded":"2024-03-16T19:29:28.063112+0000","last_fullsized":"2024-03-16T19:29:28.063112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062498+0000","last_change":"2024-03-16T19:29:26.357191+0000","last_active":"2024-03-16T19:29:28.062498+0000","last_peered":"2024-03-16T19:29:28.062498+0000","last_clean":"2024-03-16T19:29:28.062498+0000","last_became_active":"2024-03-16T19:29:26.356947+0000","last_became_peered":"2024-03-16T19:29:26.356947+0000","last_unstale":"2024-03-16T19:29:28.062498+0000","last_undegraded":"2024-03-16T19:29:28.062498+0000","last_fullsized":"2024-03-16T19:29:28.062498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:40.884164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343265+0000","last_change":"2024-03-16T19:29:26.358307+0000","last_active":"2024-03-16T19:29:27.343265+0000","last_peered":"2024-03-16T19:29:27.343265+0000","last_clean":"2024-03-16T19:29:27.343265+0000","last_became_active":"2024-03-16T19:29:26.357970+0000","last_became_peered":"2024-03-16T19:29:26.357970+0000","last_unstale":"2024-03-16T19:29:27.343265+0000","last_undegraded":"2024-03-16T19:29:27.343265+0000","last_fullsized":"2024-03-16T19:29:27.343265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:33.976868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561749+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:29:27.561749+0000","last_peered":"2024-03-16T19:29:27.561749+0000","last_clean":"2024-03-16T19:29:27.561749+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:29:27.561749+0000","last_undegraded":"2024-03-16T19:29:27.561749+0000","last_fullsized":"2024-03-16T19:29:27.561749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343286+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:29:27.343286+0000","last_peered":"2024-03-16T19:29:27.343286+0000","last_clean":"2024-03-16T19:29:27.343286+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:29:27.343286+0000","last_undegraded":"2024-03-16T19:29:27.343286+0000","last_fullsized":"2024-03-16T19:29:27.343286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562192+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:29:27.562192+0000","last_peered":"2024-03-16T19:29:27.562192+0000","last_clean":"2024-03-16T19:29:27.562192+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:29:27.562192+0000","last_undegraded":"2024-03-16T19:29:27.562192+0000","last_fullsized":"2024-03-16T19:29:27.562192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063443+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:29:28.063443+0000","last_peered":"2024-03-16T19:29:28.063443+0000","last_clean":"2024-03-16T19:29:28.063443+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:29:28.063443+0000","last_undegraded":"2024-03-16T19:29:28.063443+0000","last_fullsized":"2024-03-16T19:29:28.063443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339564+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:29:27.339564+0000","last_peered":"2024-03-16T19:29:27.339564+0000","last_clean":"2024-03-16T19:29:27.339564+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:29:27.339564+0000","last_undegraded":"2024-03-16T19:29:27.339564+0000","last_fullsized":"2024-03-16T19:29:27.339564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562214+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:29:27.562214+0000","last_peered":"2024-03-16T19:29:27.562214+0000","last_clean":"2024-03-16T19:29:27.562214+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:29:27.562214+0000","last_undegraded":"2024-03-16T19:29:27.562214+0000","last_fullsized":"2024-03-16T19:29:27.562214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062637+0000","last_change":"2024-03-16T19:29:26.349443+0000","last_active":"2024-03-16T19:29:28.062637+0000","last_peered":"2024-03-16T19:29:28.062637+0000","last_clean":"2024-03-16T19:29:28.062637+0000","last_became_active":"2024-03-16T19:29:26.349181+0000","last_became_peered":"2024-03-16T19:29:26.349181+0000","last_unstale":"2024-03-16T19:29:28.062637+0000","last_undegraded":"2024-03-16T19:29:28.062637+0000","last_fullsized":"2024-03-16T19:29:28.062637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:16.483346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343433+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:29:27.343433+0000","last_peered":"2024-03-16T19:29:27.343433+0000","last_clean":"2024-03-16T19:29:27.343433+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:29:27.343433+0000","last_undegraded":"2024-03-16T19:29:27.343433+0000","last_fullsized":"2024-03-16T19:29:27.343433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062168+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:29:28.062168+0000","last_peered":"2024-03-16T19:29:28.062168+0000","last_clean":"2024-03-16T19:29:28.062168+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:29:28.062168+0000","last_undegraded":"2024-03-16T19:29:28.062168+0000","last_fullsized":"2024-03-16T19:29:28.062168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062728+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:29:28.062728+0000","last_peered":"2024-03-16T19:29:28.062728+0000","last_clean":"2024-03-16T19:29:28.062728+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:29:28.062728+0000","last_undegraded":"2024-03-16T19:29:28.062728+0000","last_fullsized":"2024-03-16T19:29:28.062728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063484+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:29:28.063484+0000","last_peered":"2024-03-16T19:29:28.063484+0000","last_clean":"2024-03-16T19:29:28.063484+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:29:28.063484+0000","last_undegraded":"2024-03-16T19:29:28.063484+0000","last_fullsized":"2024-03-16T19:29:28.063484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562050+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:29:27.562050+0000","last_peered":"2024-03-16T19:29:27.562050+0000","last_clean":"2024-03-16T19:29:27.562050+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:29:27.562050+0000","last_undegraded":"2024-03-16T19:29:27.562050+0000","last_fullsized":"2024-03-16T19:29:27.562050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062765+0000","last_change":"2024-03-16T19:29:26.353288+0000","last_active":"2024-03-16T19:29:28.062765+0000","last_peered":"2024-03-16T19:29:28.062765+0000","last_clean":"2024-03-16T19:29:28.062765+0000","last_became_active":"2024-03-16T19:29:26.352973+0000","last_became_peered":"2024-03-16T19:29:26.352973+0000","last_unstale":"2024-03-16T19:29:28.062765+0000","last_undegraded":"2024-03-16T19:29:28.062765+0000","last_fullsized":"2024-03-16T19:29:28.062765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:22:41.505003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062951+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:29:28.062951+0000","last_peered":"2024-03-16T19:29:28.062951+0000","last_clean":"2024-03-16T19:29:28.062951+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:29:28.062951+0000","last_undegraded":"2024-03-16T19:29:28.062951+0000","last_fullsized":"2024-03-16T19:29:28.062951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561856+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:29:27.561856+0000","last_peered":"2024-03-16T19:29:27.561856+0000","last_clean":"2024-03-16T19:29:27.561856+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:29:27.561856+0000","last_undegraded":"2024-03-16T19:29:27.561856+0000","last_fullsized":"2024-03-16T19:29:27.561856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561778+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:29:27.561778+0000","last_peered":"2024-03-16T19:29:27.561778+0000","last_clean":"2024-03-16T19:29:27.561778+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:29:27.561778+0000","last_undegraded":"2024-03-16T19:29:27.561778+0000","last_fullsized":"2024-03-16T19:29:27.561778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343316+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:29:27.343316+0000","last_peered":"2024-03-16T19:29:27.343316+0000","last_clean":"2024-03-16T19:29:27.343316+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:29:27.343316+0000","last_undegraded":"2024-03-16T19:29:27.343316+0000","last_fullsized":"2024-03-16T19:29:27.343316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339401+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:29:27.339401+0000","last_peered":"2024-03-16T19:29:27.339401+0000","last_clean":"2024-03-16T19:29:27.339401+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:29:27.339401+0000","last_undegraded":"2024-03-16T19:29:27.339401+0000","last_fullsized":"2024-03-16T19:29:27.339401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063312+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:29:28.063312+0000","last_peered":"2024-03-16T19:29:28.063312+0000","last_clean":"2024-03-16T19:29:28.063312+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:29:28.063312+0000","last_undegraded":"2024-03-16T19:29:28.063312+0000","last_fullsized":"2024-03-16T19:29:28.063312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344324+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:29:27.344324+0000","last_peered":"2024-03-16T19:29:27.344324+0000","last_clean":"2024-03-16T19:29:27.344324+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:29:27.344324+0000","last_undegraded":"2024-03-16T19:29:27.344324+0000","last_fullsized":"2024-03-16T19:29:27.344324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343744+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:29:27.343744+0000","last_peered":"2024-03-16T19:29:27.343744+0000","last_clean":"2024-03-16T19:29:27.343744+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:29:27.343744+0000","last_undegraded":"2024-03-16T19:29:27.343744+0000","last_fullsized":"2024-03-16T19:29:27.343744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062922+0000","last_change":"2024-03-16T19:29:26.353918+0000","last_active":"2024-03-16T19:29:28.062922+0000","last_peered":"2024-03-16T19:29:28.062922+0000","last_clean":"2024-03-16T19:29:28.062922+0000","last_became_active":"2024-03-16T19:29:26.353463+0000","last_became_peered":"2024-03-16T19:29:26.353463+0000","last_unstale":"2024-03-16T19:29:28.062922+0000","last_undegraded":"2024-03-16T19:29:28.062922+0000","last_fullsized":"2024-03-16T19:29:28.062922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:36.035692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339586+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:29:27.339586+0000","last_peered":"2024-03-16T19:29:27.339586+0000","last_clean":"2024-03-16T19:29:27.339586+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:29:27.339586+0000","last_undegraded":"2024-03-16T19:29:27.339586+0000","last_fullsized":"2024-03-16T19:29:27.339586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063360+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:29:28.063360+0000","last_peered":"2024-03-16T19:29:28.063360+0000","last_clean":"2024-03-16T19:29:28.063360+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:29:28.063360+0000","last_undegraded":"2024-03-16T19:29:28.063360+0000","last_fullsized":"2024-03-16T19:29:28.063360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063209+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:29:28.063209+0000","last_peered":"2024-03-16T19:29:28.063209+0000","last_clean":"2024-03-16T19:29:28.063209+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:29:28.063209+0000","last_undegraded":"2024-03-16T19:29:28.063209+0000","last_fullsized":"2024-03-16T19:29:28.063209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344465+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:29:27.344465+0000","last_peered":"2024-03-16T19:29:27.344465+0000","last_clean":"2024-03-16T19:29:27.344465+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:29:27.344465+0000","last_undegraded":"2024-03-16T19:29:27.344465+0000","last_fullsized":"2024-03-16T19:29:27.344465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062833+0000","last_change":"2024-03-16T19:29:25.336296+0000","last_active":"2024-03-16T19:29:28.062833+0000","last_peered":"2024-03-16T19:29:28.062833+0000","last_clean":"2024-03-16T19:29:28.062833+0000","last_became_active":"2024-03-16T19:29:25.333046+0000","last_became_peered":"2024-03-16T19:29:25.333046+0000","last_unstale":"2024-03-16T19:29:28.062833+0000","last_undegraded":"2024-03-16T19:29:28.062833+0000","last_fullsized":"2024-03-16T19:29:28.062833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:33.538342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062507+0000","last_change":"2024-03-16T19:29:26.354709+0000","last_active":"2024-03-16T19:29:28.062507+0000","last_peered":"2024-03-16T19:29:28.062507+0000","last_clean":"2024-03-16T19:29:28.062507+0000","last_became_active":"2024-03-16T19:29:26.354377+0000","last_became_peered":"2024-03-16T19:29:26.354377+0000","last_unstale":"2024-03-16T19:29:28.062507+0000","last_undegraded":"2024-03-16T19:29:28.062507+0000","last_fullsized":"2024-03-16T19:29:28.062507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:41:56.277270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343553+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:29:27.343553+0000","last_peered":"2024-03-16T19:29:27.343553+0000","last_clean":"2024-03-16T19:29:27.343553+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:29:27.343553+0000","last_undegraded":"2024-03-16T19:29:27.343553+0000","last_fullsized":"2024-03-16T19:29:27.343553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344364+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:29:27.344364+0000","last_peered":"2024-03-16T19:29:27.344364+0000","last_clean":"2024-03-16T19:29:27.344364+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:29:27.344364+0000","last_undegraded":"2024-03-16T19:29:27.344364+0000","last_fullsized":"2024-03-16T19:29:27.344364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062737+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:29:28.062737+0000","last_peered":"2024-03-16T19:29:28.062737+0000","last_clean":"2024-03-16T19:29:28.062737+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:29:28.062737+0000","last_undegraded":"2024-03-16T19:29:28.062737+0000","last_fullsized":"2024-03-16T19:29:28.062737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062954+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:29:28.062954+0000","last_peered":"2024-03-16T19:29:28.062954+0000","last_clean":"2024-03-16T19:29:28.062954+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:29:28.062954+0000","last_undegraded":"2024-03-16T19:29:28.062954+0000","last_fullsized":"2024-03-16T19:29:28.062954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062109+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:29:28.062109+0000","last_peered":"2024-03-16T19:29:28.062109+0000","last_clean":"2024-03-16T19:29:28.062109+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:29:28.062109+0000","last_undegraded":"2024-03-16T19:29:28.062109+0000","last_fullsized":"2024-03-16T19:29:28.062109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344168+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:29:27.344168+0000","last_peered":"2024-03-16T19:29:27.344168+0000","last_clean":"2024-03-16T19:29:27.344168+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:29:27.344168+0000","last_undegraded":"2024-03-16T19:29:27.344168+0000","last_fullsized":"2024-03-16T19:29:27.344168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063139+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:29:28.063139+0000","last_peered":"2024-03-16T19:29:28.063139+0000","last_clean":"2024-03-16T19:29:28.063139+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:29:28.063139+0000","last_undegraded":"2024-03-16T19:29:28.063139+0000","last_fullsized":"2024-03-16T19:29:28.063139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344198+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:29:27.344198+0000","last_peered":"2024-03-16T19:29:27.344198+0000","last_clean":"2024-03-16T19:29:27.344198+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:29:27.344198+0000","last_undegraded":"2024-03-16T19:29:27.344198+0000","last_fullsized":"2024-03-16T19:29:27.344198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062656+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:29:28.062656+0000","last_peered":"2024-03-16T19:29:28.062656+0000","last_clean":"2024-03-16T19:29:28.062656+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:29:28.062656+0000","last_undegraded":"2024-03-16T19:29:28.062656+0000","last_fullsized":"2024-03-16T19:29:28.062656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344664+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:29:27.344664+0000","last_peered":"2024-03-16T19:29:27.344664+0000","last_clean":"2024-03-16T19:29:27.344664+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:29:27.344664+0000","last_undegraded":"2024-03-16T19:29:27.344664+0000","last_fullsized":"2024-03-16T19:29:27.344664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339668+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:29:27.339668+0000","last_peered":"2024-03-16T19:29:27.339668+0000","last_clean":"2024-03-16T19:29:27.339668+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:29:27.339668+0000","last_undegraded":"2024-03-16T19:29:27.339668+0000","last_fullsized":"2024-03-16T19:29:27.339668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344128+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:29:27.344128+0000","last_peered":"2024-03-16T19:29:27.344128+0000","last_clean":"2024-03-16T19:29:27.344128+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:29:27.344128+0000","last_undegraded":"2024-03-16T19:29:27.344128+0000","last_fullsized":"2024-03-16T19:29:27.344128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062516+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:29:28.062516+0000","last_peered":"2024-03-16T19:29:28.062516+0000","last_clean":"2024-03-16T19:29:28.062516+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:29:28.062516+0000","last_undegraded":"2024-03-16T19:29:28.062516+0000","last_fullsized":"2024-03-16T19:29:28.062516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344351+0000","last_change":"2024-03-16T19:29:26.364742+0000","last_active":"2024-03-16T19:29:27.344351+0000","last_peered":"2024-03-16T19:29:27.344351+0000","last_clean":"2024-03-16T19:29:27.344351+0000","last_became_active":"2024-03-16T19:29:26.364328+0000","last_became_peered":"2024-03-16T19:29:26.364328+0000","last_unstale":"2024-03-16T19:29:27.344351+0000","last_undegraded":"2024-03-16T19:29:27.344351+0000","last_fullsized":"2024-03-16T19:29:27.344351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:18.877774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343226+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:29:27.343226+0000","last_peered":"2024-03-16T19:29:27.343226+0000","last_clean":"2024-03-16T19:29:27.343226+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:29:27.343226+0000","last_undegraded":"2024-03-16T19:29:27.343226+0000","last_fullsized":"2024-03-16T19:29:27.343226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561997+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:29:27.561997+0000","last_peered":"2024-03-16T19:29:27.561997+0000","last_clean":"2024-03-16T19:29:27.561997+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:29:27.561997+0000","last_undegraded":"2024-03-16T19:29:27.561997+0000","last_fullsized":"2024-03-16T19:29:27.561997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343035+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:29:27.343035+0000","last_peered":"2024-03-16T19:29:27.343035+0000","last_clean":"2024-03-16T19:29:27.343035+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:29:27.343035+0000","last_undegraded":"2024-03-16T19:29:27.343035+0000","last_fullsized":"2024-03-16T19:29:27.343035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062305+0000","last_change":"2024-03-16T19:29:26.352692+0000","last_active":"2024-03-16T19:29:28.062305+0000","last_peered":"2024-03-16T19:29:28.062305+0000","last_clean":"2024-03-16T19:29:28.062305+0000","last_became_active":"2024-03-16T19:29:26.352419+0000","last_became_peered":"2024-03-16T19:29:26.352419+0000","last_unstale":"2024-03-16T19:29:28.062305+0000","last_undegraded":"2024-03-16T19:29:28.062305+0000","last_fullsized":"2024-03-16T19:29:28.062305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:33.056538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062749+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:29:28.062749+0000","last_peered":"2024-03-16T19:29:28.062749+0000","last_clean":"2024-03-16T19:29:28.062749+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:29:28.062749+0000","last_undegraded":"2024-03-16T19:29:28.062749+0000","last_fullsized":"2024-03-16T19:29:28.062749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339393+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:29:27.339393+0000","last_peered":"2024-03-16T19:29:27.339393+0000","last_clean":"2024-03-16T19:29:27.339393+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:29:27.339393+0000","last_undegraded":"2024-03-16T19:29:27.339393+0000","last_fullsized":"2024-03-16T19:29:27.339393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344073+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:29:27.344073+0000","last_peered":"2024-03-16T19:29:27.344073+0000","last_clean":"2024-03-16T19:29:27.344073+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:29:27.344073+0000","last_undegraded":"2024-03-16T19:29:27.344073+0000","last_fullsized":"2024-03-16T19:29:27.344073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063637+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:29:28.063637+0000","last_peered":"2024-03-16T19:29:28.063637+0000","last_clean":"2024-03-16T19:29:28.063637+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:29:28.063637+0000","last_undegraded":"2024-03-16T19:29:28.063637+0000","last_fullsized":"2024-03-16T19:29:28.063637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339499+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:29:27.339499+0000","last_peered":"2024-03-16T19:29:27.339499+0000","last_clean":"2024-03-16T19:29:27.339499+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:29:27.339499+0000","last_undegraded":"2024-03-16T19:29:27.339499+0000","last_fullsized":"2024-03-16T19:29:27.339499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343975+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:29:27.343975+0000","last_peered":"2024-03-16T19:29:27.343975+0000","last_clean":"2024-03-16T19:29:27.343975+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:29:27.343975+0000","last_undegraded":"2024-03-16T19:29:27.343975+0000","last_fullsized":"2024-03-16T19:29:27.343975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344573+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:29:27.344573+0000","last_peered":"2024-03-16T19:29:27.344573+0000","last_clean":"2024-03-16T19:29:27.344573+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:29:27.344573+0000","last_undegraded":"2024-03-16T19:29:27.344573+0000","last_fullsized":"2024-03-16T19:29:27.344573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339540+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:29:27.339540+0000","last_peered":"2024-03-16T19:29:27.339540+0000","last_clean":"2024-03-16T19:29:27.339540+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:29:27.339540+0000","last_undegraded":"2024-03-16T19:29:27.339540+0000","last_fullsized":"2024-03-16T19:29:27.339540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339712+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:29:27.339712+0000","last_peered":"2024-03-16T19:29:27.339712+0000","last_clean":"2024-03-16T19:29:27.339712+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:29:27.339712+0000","last_undegraded":"2024-03-16T19:29:27.339712+0000","last_fullsized":"2024-03-16T19:29:27.339712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562225+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:29:27.562225+0000","last_peered":"2024-03-16T19:29:27.562225+0000","last_clean":"2024-03-16T19:29:27.562225+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:29:27.562225+0000","last_undegraded":"2024-03-16T19:29:27.562225+0000","last_fullsized":"2024-03-16T19:29:27.562225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063405+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:29:28.063405+0000","last_peered":"2024-03-16T19:29:28.063405+0000","last_clean":"2024-03-16T19:29:28.063405+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:29:28.063405+0000","last_undegraded":"2024-03-16T19:29:28.063405+0000","last_fullsized":"2024-03-16T19:29:28.063405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062725+0000","last_change":"2024-03-16T19:29:26.350351+0000","last_active":"2024-03-16T19:29:28.062725+0000","last_peered":"2024-03-16T19:29:28.062725+0000","last_clean":"2024-03-16T19:29:28.062725+0000","last_became_active":"2024-03-16T19:29:26.350074+0000","last_became_peered":"2024-03-16T19:29:26.350074+0000","last_unstale":"2024-03-16T19:29:28.062725+0000","last_undegraded":"2024-03-16T19:29:28.062725+0000","last_fullsized":"2024-03-16T19:29:28.062725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:49.416395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.342931+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:29:27.342931+0000","last_peered":"2024-03-16T19:29:27.342931+0000","last_clean":"2024-03-16T19:29:27.342931+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:29:27.342931+0000","last_undegraded":"2024-03-16T19:29:27.342931+0000","last_fullsized":"2024-03-16T19:29:27.342931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063379+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:29:28.063379+0000","last_peered":"2024-03-16T19:29:28.063379+0000","last_clean":"2024-03-16T19:29:28.063379+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:29:28.063379+0000","last_undegraded":"2024-03-16T19:29:28.063379+0000","last_fullsized":"2024-03-16T19:29:28.063379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339690+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:29:27.339690+0000","last_peered":"2024-03-16T19:29:27.339690+0000","last_clean":"2024-03-16T19:29:27.339690+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:29:27.339690+0000","last_undegraded":"2024-03-16T19:29:27.339690+0000","last_fullsized":"2024-03-16T19:29:27.339690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561913+0000","last_change":"2024-03-16T19:29:26.347388+0000","last_active":"2024-03-16T19:29:27.561913+0000","last_peered":"2024-03-16T19:29:27.561913+0000","last_clean":"2024-03-16T19:29:27.561913+0000","last_became_active":"2024-03-16T19:29:26.346582+0000","last_became_peered":"2024-03-16T19:29:26.346582+0000","last_unstale":"2024-03-16T19:29:27.561913+0000","last_undegraded":"2024-03-16T19:29:27.561913+0000","last_fullsized":"2024-03-16T19:29:27.561913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:05.294895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339497+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:29:27.339497+0000","last_peered":"2024-03-16T19:29:27.339497+0000","last_clean":"2024-03-16T19:29:27.339497+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:29:27.339497+0000","last_undegraded":"2024-03-16T19:29:27.339497+0000","last_fullsized":"2024-03-16T19:29:27.339497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561968+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:29:27.561968+0000","last_peered":"2024-03-16T19:29:27.561968+0000","last_clean":"2024-03-16T19:29:27.561968+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:29:27.561968+0000","last_undegraded":"2024-03-16T19:29:27.561968+0000","last_fullsized":"2024-03-16T19:29:27.561968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344441+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:29:27.344441+0000","last_peered":"2024-03-16T19:29:27.344441+0000","last_clean":"2024-03-16T19:29:27.344441+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:29:27.344441+0000","last_undegraded":"2024-03-16T19:29:27.344441+0000","last_fullsized":"2024-03-16T19:29:27.344441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062837+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:29:28.062837+0000","last_peered":"2024-03-16T19:29:28.062837+0000","last_clean":"2024-03-16T19:29:28.062837+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:29:28.062837+0000","last_undegraded":"2024-03-16T19:29:28.062837+0000","last_fullsized":"2024-03-16T19:29:28.062837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343809+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:29:27.343809+0000","last_peered":"2024-03-16T19:29:27.343809+0000","last_clean":"2024-03-16T19:29:27.343809+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:29:27.343809+0000","last_undegraded":"2024-03-16T19:29:27.343809+0000","last_fullsized":"2024-03-16T19:29:27.343809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343792+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:29:27.343792+0000","last_peered":"2024-03-16T19:29:27.343792+0000","last_clean":"2024-03-16T19:29:27.343792+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:29:27.343792+0000","last_undegraded":"2024-03-16T19:29:27.343792+0000","last_fullsized":"2024-03-16T19:29:27.343792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344208+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:29:27.344208+0000","last_peered":"2024-03-16T19:29:27.344208+0000","last_clean":"2024-03-16T19:29:27.344208+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:29:27.344208+0000","last_undegraded":"2024-03-16T19:29:27.344208+0000","last_fullsized":"2024-03-16T19:29:27.344208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344457+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:29:27.344457+0000","last_peered":"2024-03-16T19:29:27.344457+0000","last_clean":"2024-03-16T19:29:27.344457+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:29:27.344457+0000","last_undegraded":"2024-03-16T19:29:27.344457+0000","last_fullsized":"2024-03-16T19:29:27.344457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062891+0000","last_change":"2024-03-16T19:29:26.354303+0000","last_active":"2024-03-16T19:29:28.062891+0000","last_peered":"2024-03-16T19:29:28.062891+0000","last_clean":"2024-03-16T19:29:28.062891+0000","last_became_active":"2024-03-16T19:29:26.353804+0000","last_became_peered":"2024-03-16T19:29:26.353804+0000","last_unstale":"2024-03-16T19:29:28.062891+0000","last_undegraded":"2024-03-16T19:29:28.062891+0000","last_fullsized":"2024-03-16T19:29:28.062891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:10:45.368536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562162+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:29:27.562162+0000","last_peered":"2024-03-16T19:29:27.562162+0000","last_clean":"2024-03-16T19:29:27.562162+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:29:27.562162+0000","last_undegraded":"2024-03-16T19:29:27.562162+0000","last_fullsized":"2024-03-16T19:29:27.562162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562176+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:29:27.562176+0000","last_peered":"2024-03-16T19:29:27.562176+0000","last_clean":"2024-03-16T19:29:27.562176+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:29:27.562176+0000","last_undegraded":"2024-03-16T19:29:27.562176+0000","last_fullsized":"2024-03-16T19:29:27.562176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062982+0000","last_change":"2024-03-16T19:29:25.342597+0000","last_active":"2024-03-16T19:29:28.062982+0000","last_peered":"2024-03-16T19:29:28.062982+0000","last_clean":"2024-03-16T19:29:28.062982+0000","last_became_active":"2024-03-16T19:29:25.342320+0000","last_became_peered":"2024-03-16T19:29:25.342320+0000","last_unstale":"2024-03-16T19:29:28.062982+0000","last_undegraded":"2024-03-16T19:29:28.062982+0000","last_fullsized":"2024-03-16T19:29:28.062982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:44:41.274396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":665973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":659673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:29:30.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:29:30.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.461+0000 7f65be8eb640 1 Processor -- start 2024-03-16T19:29:30.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.461+0000 7f65be8eb640 1 -- start start 2024-03-16T19:29:30.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65be8eb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65b8154c90 0x7f65b815e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65be8eb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f65b80cc780 0x7f65b80cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65be8eb640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b815b880 con 0x7f65b815ea30 2024-03-16T19:29:30.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65be8eb640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b80c6ee0 con 0x7f65b8154c90 2024-03-16T19:29:30.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65be8eb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b8155070 con 0x7f65b80cc780 2024-03-16T19:29:30.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65b7fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f65b8154c90 0x7f65b815e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65bce88640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f65b815ea30 0x7f65b806b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56098/0 (socket says 172.21.15.172:56098) 2024-03-16T19:29:30.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65bce88640 1 -- 172.21.15.172:0/1789789109 learned_addr learned my addr 172.21.15.172:0/1789789109 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65b77fe640 1 --2- 172.21.15.172:0/1789789109 >> v2:172.21.15.174:3300/0 conn(0x7f65b80cc780 0x7f65b80cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65b7fff640 1 -- 172.21.15.172:0/1789789109 >> v2:172.21.15.174:3300/0 conn(0x7f65b80cc780 msgr2=0x7f65b80cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65b7fff640 1 --2- 172.21.15.172:0/1789789109 >> v2:172.21.15.174:3300/0 conn(0x7f65b80cc780 0x7f65b80cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65b7fff640 1 -- 172.21.15.172:0/1789789109 >> v1:172.21.15.172:6789/0 conn(0x7f65b815ea30 legacy=0x7f65b806b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65b6ffd640 1 -- 172.21.15.172:0/1789789109 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693741009 0 0) 0x7f65b0001490 con 0x7f65b815ea30 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65b7fff640 1 -- 172.21.15.172:0/1789789109 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b815ad00 con 0x7f65b8154c90 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.462+0000 7f65b77fe640 1 --2- 172.21.15.172:0/1789789109 >> v2:172.21.15.174:3300/0 conn(0x7f65b80cc780 0x7f65b80cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.463+0000 7f65b7fff640 1 --2- 172.21.15.172:0/1789789109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 0x7f65b815e4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65b80b2c00 tx=0x7f65a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa2b159a5c91697 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.463+0000 7f65b6ffd640 1 -- 172.21.15.172:0/1789789109 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8031830 con 0x7f65b8154c90 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.463+0000 7f65b6ffd640 1 -- 172.21.15.172:0/1789789109 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a8031990 con 0x7f65b8154c90 2024-03-16T19:29:30.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.463+0000 7f65b6ffd640 1 -- 172.21.15.172:0/1789789109 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8031d20 con 0x7f65b8154c90 2024-03-16T19:29:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.464+0000 7f65be8eb640 1 -- 172.21.15.172:0/1789789109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 msgr2=0x7f65b815e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.464+0000 7f65be8eb640 1 --2- 172.21.15.172:0/1789789109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 0x7f65b815e4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65b80b2c00 tx=0x7f65a802b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.464+0000 7f65be8eb640 1 -- 172.21.15.172:0/1789789109 shutdown_connections 2024-03-16T19:29:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.464+0000 7f65be8eb640 1 --2- 172.21.15.172:0/1789789109 >> v2:172.21.15.174:3300/0 conn(0x7f65b80cc780 0x7f65b80cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.464+0000 7f65be8eb640 1 --2- 172.21.15.172:0/1789789109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 0x7f65b815e4f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.464+0000 7f65be8eb640 1 -- 172.21.15.172:0/1789789109 >> 172.21.15.172:0/1789789109 conn(0x7f65b8067040 msgr2=0x7f65b8067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.464+0000 7f65be8eb640 1 -- 172.21.15.172:0/1789789109 shutdown_connections 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.464+0000 7f65be8eb640 1 -- 172.21.15.172:0/1789789109 wait complete. 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.465+0000 7f65be8eb640 1 Processor -- start 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.465+0000 7f65be8eb640 1 -- start start 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65be8eb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b80cc780 0x7f65b81f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65be8eb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 0x7f65b8156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65be8eb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65b815ea30 0x7f65b8159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65be8eb640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b81f1720 con 0x7f65b8154c90 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65be8eb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b8156c50 con 0x7f65b80cc780 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65be8eb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b815ddd0 con 0x7f65b815ea30 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 0x7f65b8156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 0x7f65b8156710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46108/0 (socket says 172.21.15.172:46108) 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b77fe640 1 -- 172.21.15.172:0/908272101 learned_addr learned my addr 172.21.15.172:0/908272101 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b7fff640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b80cc780 0x7f65b81f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65bce88640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65b815ea30 0x7f65b8159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b77fe640 1 -- 172.21.15.172:0/908272101 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65b815ea30 msgr2=0x7f65b8159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b77fe640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65b815ea30 0x7f65b8159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b77fe640 1 -- 172.21.15.172:0/908272101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b80cc780 msgr2=0x7f65b81f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b77fe640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b80cc780 0x7f65b81f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b77fe640 1 -- 172.21.15.172:0/908272101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a8009400 con 0x7f65b8154c90 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65bce88640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65b815ea30 0x7f65b8159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b77fe640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 0x7f65b8156710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00c7f0 tx=0x7f65ac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b4ff9640 1 -- 172.21.15.172:0/908272101 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac007a30 con 0x7f65b8154c90 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b4ff9640 1 -- 172.21.15.172:0/908272101 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ac007b90 con 0x7f65b8154c90 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b815daf0 con 0x7f65b8154c90 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65b4ff9640 1 -- 172.21.15.172:0/908272101 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac00ce40 con 0x7f65b8154c90 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.466+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b815be10 con 0x7f65b8154c90 2024-03-16T19:29:30.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.467+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b815df90 con 0x7f65b8154c90 2024-03-16T19:29:30.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.468+0000 7f65b4ff9640 1 -- 172.21.15.172:0/908272101 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f65ac0113f0 con 0x7f65b8154c90 2024-03-16T19:29:30.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.468+0000 7f65b4ff9640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f65a406b3b0 0x7f65a406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.468+0000 7f65b4ff9640 1 -- 172.21.15.172:0/908272101 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65ac089d40 con 0x7f65b8154c90 2024-03-16T19:29:30.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.468+0000 7f65b7fff640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f65a406b3b0 0x7f65a406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.452 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.468+0000 7f65b7fff640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f65a406b3b0 0x7f65a406d870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f65a8036000 tx=0x7f65a803e040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:30.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.472+0000 7f65b4ff9640 1 -- 172.21.15.172:0/908272101 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65ac054a50 con 0x7f65b8154c90 2024-03-16T19:29:30.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.641+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65b8152b30 con 0x7f65a406b3b0 2024-03-16T19:29:30.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.647+0000 7f65b4ff9640 1 -- 172.21.15.172:0/908272101 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251847 (secure 0 0 0) 0x7f65b8152b30 con 0x7f65a406b3b0 2024-03-16T19:29:30.631 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:30.632 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:29:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f65a406b3b0 msgr2=0x7f65a406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f65a406b3b0 0x7f65a406d870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f65a8036000 tx=0x7f65a803e040 comp rx=0 tx=0).stop 2024-03-16T19:29:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 msgr2=0x7f65b8156710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 0x7f65b8156710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00c7f0 tx=0x7f65ac00cbf0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 shutdown_connections 2024-03-16T19:29:30.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f65a406b3b0 0x7f65a406d870 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65b815ea30 0x7f65b8159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65b8154c90 0x7f65b8156710 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 --2- 172.21.15.172:0/908272101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b80cc780 0x7f65b81f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 >> 172.21.15.172:0/908272101 conn(0x7f65b8067040 msgr2=0x7f65b80caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:30.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 shutdown_connections 2024-03-16T19:29:30.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.650+0000 7f65be8eb640 1 -- 172.21.15.172:0/908272101 wait complete. 2024-03-16T19:29:30.642 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-16T19:29:30.200302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218976,"kb_used_data":2796,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755744,"statfs":{"total":773094113280,"available":772869881856,"internally_reserved":0,"allocated":2863104,"data_stored":1754076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.876050"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344501+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:29:27.344501+0000","last_peered":"2024-03-16T19:29:27.344501+0000","last_clean":"2024-03-16T19:29:27.344501+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:29:27.344501+0000","last_undegraded":"2024-03-16T19:29:27.344501+0000","last_fullsized":"2024-03-16T19:29:27.344501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339631+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:29:27.339631+0000","last_peered":"2024-03-16T19:29:27.339631+0000","last_clean":"2024-03-16T19:29:27.339631+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:29:27.339631+0000","last_undegraded":"2024-03-16T19:29:27.339631+0000","last_fullsized":"2024-03-16T19:29:27.339631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062822+0000","last_change":"2024-03-16T19:29:26.347979+0000","last_active":"2024-03-16T19:29:28.062822+0000","last_peered":"2024-03-16T19:29:28.062822+0000","last_clean":"2024-03-16T19:29:28.062822+0000","last_became_active":"2024-03-16T19:29:26.347711+0000","last_became_peered":"2024-03-16T19:29:26.347711+0000","last_unstale":"2024-03-16T19:29:28.062822+0000","last_undegraded":"2024-03-16T19:29:28.062822+0000","last_fullsized":"2024-03-16T19:29:28.062822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:08.017182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343558+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:29:27.343558+0000","last_peered":"2024-03-16T19:29:27.343558+0000","last_clean":"2024-03-16T19:29:27.343558+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:29:27.343558+0000","last_undegraded":"2024-03-16T19:29:27.343558+0000","last_fullsized":"2024-03-16T19:29:27.343558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344270+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:29:27.344270+0000","last_peered":"2024-03-16T19:29:27.344270+0000","last_clean":"2024-03-16T19:29:27.344270+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:29:27.344270+0000","last_undegraded":"2024-03-16T19:29:27.344270+0000","last_fullsized":"2024-03-16T19:29:27.344270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343104+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:29:27.343104+0000","last_peered":"2024-03-16T19:29:27.343104+0000","last_clean":"2024-03-16T19:29:27.343104+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:29:27.343104+0000","last_undegraded":"2024-03-16T19:29:27.343104+0000","last_fullsized":"2024-03-16T19:29:27.343104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344269+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:29:27.344269+0000","last_peered":"2024-03-16T19:29:27.344269+0000","last_clean":"2024-03-16T19:29:27.344269+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:29:27.344269+0000","last_undegraded":"2024-03-16T19:29:27.344269+0000","last_fullsized":"2024-03-16T19:29:27.344269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561737+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:29:27.561737+0000","last_peered":"2024-03-16T19:29:27.561737+0000","last_clean":"2024-03-16T19:29:27.561737+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:29:27.561737+0000","last_undegraded":"2024-03-16T19:29:27.561737+0000","last_fullsized":"2024-03-16T19:29:27.561737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344497+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:29:27.344497+0000","last_peered":"2024-03-16T19:29:27.344497+0000","last_clean":"2024-03-16T19:29:27.344497+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:29:27.344497+0000","last_undegraded":"2024-03-16T19:29:27.344497+0000","last_fullsized":"2024-03-16T19:29:27.344497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063286+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:29:28.063286+0000","last_peered":"2024-03-16T19:29:28.063286+0000","last_clean":"2024-03-16T19:29:28.063286+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:29:28.063286+0000","last_undegraded":"2024-03-16T19:29:28.063286+0000","last_fullsized":"2024-03-16T19:29:28.063286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062996+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:29:28.062996+0000","last_peered":"2024-03-16T19:29:28.062996+0000","last_clean":"2024-03-16T19:29:28.062996+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:29:28.062996+0000","last_undegraded":"2024-03-16T19:29:28.062996+0000","last_fullsized":"2024-03-16T19:29:28.062996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063177+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:29:28.063177+0000","last_peered":"2024-03-16T19:29:28.063177+0000","last_clean":"2024-03-16T19:29:28.063177+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:29:28.063177+0000","last_undegraded":"2024-03-16T19:29:28.063177+0000","last_fullsized":"2024-03-16T19:29:28.063177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561930+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:29:27.561930+0000","last_peered":"2024-03-16T19:29:27.561930+0000","last_clean":"2024-03-16T19:29:27.561930+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:29:27.561930+0000","last_undegraded":"2024-03-16T19:29:27.561930+0000","last_fullsized":"2024-03-16T19:29:27.561930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062462+0000","last_change":"2024-03-16T19:29:26.349644+0000","last_active":"2024-03-16T19:29:28.062462+0000","last_peered":"2024-03-16T19:29:28.062462+0000","last_clean":"2024-03-16T19:29:28.062462+0000","last_became_active":"2024-03-16T19:29:26.349397+0000","last_became_peered":"2024-03-16T19:29:26.349397+0000","last_unstale":"2024-03-16T19:29:28.062462+0000","last_undegraded":"2024-03-16T19:29:28.062462+0000","last_fullsized":"2024-03-16T19:29:28.062462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:47:20.566393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339397+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:29:27.339397+0000","last_peered":"2024-03-16T19:29:27.339397+0000","last_clean":"2024-03-16T19:29:27.339397+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:29:27.339397+0000","last_undegraded":"2024-03-16T19:29:27.339397+0000","last_fullsized":"2024-03-16T19:29:27.339397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344311+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:29:27.344311+0000","last_peered":"2024-03-16T19:29:27.344311+0000","last_clean":"2024-03-16T19:29:27.344311+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:29:27.344311+0000","last_undegraded":"2024-03-16T19:29:27.344311+0000","last_fullsized":"2024-03-16T19:29:27.344311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062569+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:29:28.062569+0000","last_peered":"2024-03-16T19:29:28.062569+0000","last_clean":"2024-03-16T19:29:28.062569+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:29:28.062569+0000","last_undegraded":"2024-03-16T19:29:28.062569+0000","last_fullsized":"2024-03-16T19:29:28.062569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344648+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:29:27.344648+0000","last_peered":"2024-03-16T19:29:27.344648+0000","last_clean":"2024-03-16T19:29:27.344648+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:29:27.344648+0000","last_undegraded":"2024-03-16T19:29:27.344648+0000","last_fullsized":"2024-03-16T19:29:27.344648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063112+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:29:28.063112+0000","last_peered":"2024-03-16T19:29:28.063112+0000","last_clean":"2024-03-16T19:29:28.063112+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:29:28.063112+0000","last_undegraded":"2024-03-16T19:29:28.063112+0000","last_fullsized":"2024-03-16T19:29:28.063112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062498+0000","last_change":"2024-03-16T19:29:26.357191+0000","last_active":"2024-03-16T19:29:28.062498+0000","last_peered":"2024-03-16T19:29:28.062498+0000","last_clean":"2024-03-16T19:29:28.062498+0000","last_became_active":"2024-03-16T19:29:26.356947+0000","last_became_peered":"2024-03-16T19:29:26.356947+0000","last_unstale":"2024-03-16T19:29:28.062498+0000","last_undegraded":"2024-03-16T19:29:28.062498+0000","last_fullsized":"2024-03-16T19:29:28.062498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:40.884164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343265+0000","last_change":"2024-03-16T19:29:26.358307+0000","last_active":"2024-03-16T19:29:27.343265+0000","last_peered":"2024-03-16T19:29:27.343265+0000","last_clean":"2024-03-16T19:29:27.343265+0000","last_became_active":"2024-03-16T19:29:26.357970+0000","last_became_peered":"2024-03-16T19:29:26.357970+0000","last_unstale":"2024-03-16T19:29:27.343265+0000","last_undegraded":"2024-03-16T19:29:27.343265+0000","last_fullsized":"2024-03-16T19:29:27.343265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:33.976868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561749+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:29:27.561749+0000","last_peered":"2024-03-16T19:29:27.561749+0000","last_clean":"2024-03-16T19:29:27.561749+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:29:27.561749+0000","last_undegraded":"2024-03-16T19:29:27.561749+0000","last_fullsized":"2024-03-16T19:29:27.561749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343286+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:29:27.343286+0000","last_peered":"2024-03-16T19:29:27.343286+0000","last_clean":"2024-03-16T19:29:27.343286+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:29:27.343286+0000","last_undegraded":"2024-03-16T19:29:27.343286+0000","last_fullsized":"2024-03-16T19:29:27.343286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562192+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:29:27.562192+0000","last_peered":"2024-03-16T19:29:27.562192+0000","last_clean":"2024-03-16T19:29:27.562192+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:29:27.562192+0000","last_undegraded":"2024-03-16T19:29:27.562192+0000","last_fullsized":"2024-03-16T19:29:27.562192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063443+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:29:28.063443+0000","last_peered":"2024-03-16T19:29:28.063443+0000","last_clean":"2024-03-16T19:29:28.063443+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:29:28.063443+0000","last_undegraded":"2024-03-16T19:29:28.063443+0000","last_fullsized":"2024-03-16T19:29:28.063443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339564+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:29:27.339564+0000","last_peered":"2024-03-16T19:29:27.339564+0000","last_clean":"2024-03-16T19:29:27.339564+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:29:27.339564+0000","last_undegraded":"2024-03-16T19:29:27.339564+0000","last_fullsized":"2024-03-16T19:29:27.339564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562214+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:29:27.562214+0000","last_peered":"2024-03-16T19:29:27.562214+0000","last_clean":"2024-03-16T19:29:27.562214+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:29:27.562214+0000","last_undegraded":"2024-03-16T19:29:27.562214+0000","last_fullsized":"2024-03-16T19:29:27.562214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062637+0000","last_change":"2024-03-16T19:29:26.349443+0000","last_active":"2024-03-16T19:29:28.062637+0000","last_peered":"2024-03-16T19:29:28.062637+0000","last_clean":"2024-03-16T19:29:28.062637+0000","last_became_active":"2024-03-16T19:29:26.349181+0000","last_became_peered":"2024-03-16T19:29:26.349181+0000","last_unstale":"2024-03-16T19:29:28.062637+0000","last_undegraded":"2024-03-16T19:29:28.062637+0000","last_fullsized":"2024-03-16T19:29:28.062637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:16.483346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343433+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:29:27.343433+0000","last_peered":"2024-03-16T19:29:27.343433+0000","last_clean":"2024-03-16T19:29:27.343433+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:29:27.343433+0000","last_undegraded":"2024-03-16T19:29:27.343433+0000","last_fullsized":"2024-03-16T19:29:27.343433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062168+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:29:28.062168+0000","last_peered":"2024-03-16T19:29:28.062168+0000","last_clean":"2024-03-16T19:29:28.062168+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:29:28.062168+0000","last_undegraded":"2024-03-16T19:29:28.062168+0000","last_fullsized":"2024-03-16T19:29:28.062168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062728+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:29:28.062728+0000","last_peered":"2024-03-16T19:29:28.062728+0000","last_clean":"2024-03-16T19:29:28.062728+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:29:28.062728+0000","last_undegraded":"2024-03-16T19:29:28.062728+0000","last_fullsized":"2024-03-16T19:29:28.062728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063484+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:29:28.063484+0000","last_peered":"2024-03-16T19:29:28.063484+0000","last_clean":"2024-03-16T19:29:28.063484+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:29:28.063484+0000","last_undegraded":"2024-03-16T19:29:28.063484+0000","last_fullsized":"2024-03-16T19:29:28.063484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562050+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:29:27.562050+0000","last_peered":"2024-03-16T19:29:27.562050+0000","last_clean":"2024-03-16T19:29:27.562050+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:29:27.562050+0000","last_undegraded":"2024-03-16T19:29:27.562050+0000","last_fullsized":"2024-03-16T19:29:27.562050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062765+0000","last_change":"2024-03-16T19:29:26.353288+0000","last_active":"2024-03-16T19:29:28.062765+0000","last_peered":"2024-03-16T19:29:28.062765+0000","last_clean":"2024-03-16T19:29:28.062765+0000","last_became_active":"2024-03-16T19:29:26.352973+0000","last_became_peered":"2024-03-16T19:29:26.352973+0000","last_unstale":"2024-03-16T19:29:28.062765+0000","last_undegraded":"2024-03-16T19:29:28.062765+0000","last_fullsized":"2024-03-16T19:29:28.062765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:22:41.505003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062951+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:29:28.062951+0000","last_peered":"2024-03-16T19:29:28.062951+0000","last_clean":"2024-03-16T19:29:28.062951+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:29:28.062951+0000","last_undegraded":"2024-03-16T19:29:28.062951+0000","last_fullsized":"2024-03-16T19:29:28.062951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561856+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:29:27.561856+0000","last_peered":"2024-03-16T19:29:27.561856+0000","last_clean":"2024-03-16T19:29:27.561856+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:29:27.561856+0000","last_undegraded":"2024-03-16T19:29:27.561856+0000","last_fullsized":"2024-03-16T19:29:27.561856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561778+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:29:27.561778+0000","last_peered":"2024-03-16T19:29:27.561778+0000","last_clean":"2024-03-16T19:29:27.561778+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:29:27.561778+0000","last_undegraded":"2024-03-16T19:29:27.561778+0000","last_fullsized":"2024-03-16T19:29:27.561778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343316+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:29:27.343316+0000","last_peered":"2024-03-16T19:29:27.343316+0000","last_clean":"2024-03-16T19:29:27.343316+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:29:27.343316+0000","last_undegraded":"2024-03-16T19:29:27.343316+0000","last_fullsized":"2024-03-16T19:29:27.343316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339401+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:29:27.339401+0000","last_peered":"2024-03-16T19:29:27.339401+0000","last_clean":"2024-03-16T19:29:27.339401+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:29:27.339401+0000","last_undegraded":"2024-03-16T19:29:27.339401+0000","last_fullsized":"2024-03-16T19:29:27.339401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063312+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:29:28.063312+0000","last_peered":"2024-03-16T19:29:28.063312+0000","last_clean":"2024-03-16T19:29:28.063312+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:29:28.063312+0000","last_undegraded":"2024-03-16T19:29:28.063312+0000","last_fullsized":"2024-03-16T19:29:28.063312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344324+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:29:27.344324+0000","last_peered":"2024-03-16T19:29:27.344324+0000","last_clean":"2024-03-16T19:29:27.344324+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:29:27.344324+0000","last_undegraded":"2024-03-16T19:29:27.344324+0000","last_fullsized":"2024-03-16T19:29:27.344324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343744+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:29:27.343744+0000","last_peered":"2024-03-16T19:29:27.343744+0000","last_clean":"2024-03-16T19:29:27.343744+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:29:27.343744+0000","last_undegraded":"2024-03-16T19:29:27.343744+0000","last_fullsized":"2024-03-16T19:29:27.343744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062922+0000","last_change":"2024-03-16T19:29:26.353918+0000","last_active":"2024-03-16T19:29:28.062922+0000","last_peered":"2024-03-16T19:29:28.062922+0000","last_clean":"2024-03-16T19:29:28.062922+0000","last_became_active":"2024-03-16T19:29:26.353463+0000","last_became_peered":"2024-03-16T19:29:26.353463+0000","last_unstale":"2024-03-16T19:29:28.062922+0000","last_undegraded":"2024-03-16T19:29:28.062922+0000","last_fullsized":"2024-03-16T19:29:28.062922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:36.035692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339586+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:29:27.339586+0000","last_peered":"2024-03-16T19:29:27.339586+0000","last_clean":"2024-03-16T19:29:27.339586+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:29:27.339586+0000","last_undegraded":"2024-03-16T19:29:27.339586+0000","last_fullsized":"2024-03-16T19:29:27.339586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063360+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:29:28.063360+0000","last_peered":"2024-03-16T19:29:28.063360+0000","last_clean":"2024-03-16T19:29:28.063360+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:29:28.063360+0000","last_undegraded":"2024-03-16T19:29:28.063360+0000","last_fullsized":"2024-03-16T19:29:28.063360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063209+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:29:28.063209+0000","last_peered":"2024-03-16T19:29:28.063209+0000","last_clean":"2024-03-16T19:29:28.063209+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:29:28.063209+0000","last_undegraded":"2024-03-16T19:29:28.063209+0000","last_fullsized":"2024-03-16T19:29:28.063209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344465+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:29:27.344465+0000","last_peered":"2024-03-16T19:29:27.344465+0000","last_clean":"2024-03-16T19:29:27.344465+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:29:27.344465+0000","last_undegraded":"2024-03-16T19:29:27.344465+0000","last_fullsized":"2024-03-16T19:29:27.344465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062833+0000","last_change":"2024-03-16T19:29:25.336296+0000","last_active":"2024-03-16T19:29:28.062833+0000","last_peered":"2024-03-16T19:29:28.062833+0000","last_clean":"2024-03-16T19:29:28.062833+0000","last_became_active":"2024-03-16T19:29:25.333046+0000","last_became_peered":"2024-03-16T19:29:25.333046+0000","last_unstale":"2024-03-16T19:29:28.062833+0000","last_undegraded":"2024-03-16T19:29:28.062833+0000","last_fullsized":"2024-03-16T19:29:28.062833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:33.538342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062507+0000","last_change":"2024-03-16T19:29:26.354709+0000","last_active":"2024-03-16T19:29:28.062507+0000","last_peered":"2024-03-16T19:29:28.062507+0000","last_clean":"2024-03-16T19:29:28.062507+0000","last_became_active":"2024-03-16T19:29:26.354377+0000","last_became_peered":"2024-03-16T19:29:26.354377+0000","last_unstale":"2024-03-16T19:29:28.062507+0000","last_undegraded":"2024-03-16T19:29:28.062507+0000","last_fullsized":"2024-03-16T19:29:28.062507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:41:56.277270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343553+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:29:27.343553+0000","last_peered":"2024-03-16T19:29:27.343553+0000","last_clean":"2024-03-16T19:29:27.343553+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:29:27.343553+0000","last_undegraded":"2024-03-16T19:29:27.343553+0000","last_fullsized":"2024-03-16T19:29:27.343553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344364+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:29:27.344364+0000","last_peered":"2024-03-16T19:29:27.344364+0000","last_clean":"2024-03-16T19:29:27.344364+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:29:27.344364+0000","last_undegraded":"2024-03-16T19:29:27.344364+0000","last_fullsized":"2024-03-16T19:29:27.344364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062737+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:29:28.062737+0000","last_peered":"2024-03-16T19:29:28.062737+0000","last_clean":"2024-03-16T19:29:28.062737+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:29:28.062737+0000","last_undegraded":"2024-03-16T19:29:28.062737+0000","last_fullsized":"2024-03-16T19:29:28.062737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062954+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:29:28.062954+0000","last_peered":"2024-03-16T19:29:28.062954+0000","last_clean":"2024-03-16T19:29:28.062954+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:29:28.062954+0000","last_undegraded":"2024-03-16T19:29:28.062954+0000","last_fullsized":"2024-03-16T19:29:28.062954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062109+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:29:28.062109+0000","last_peered":"2024-03-16T19:29:28.062109+0000","last_clean":"2024-03-16T19:29:28.062109+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:29:28.062109+0000","last_undegraded":"2024-03-16T19:29:28.062109+0000","last_fullsized":"2024-03-16T19:29:28.062109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344168+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:29:27.344168+0000","last_peered":"2024-03-16T19:29:27.344168+0000","last_clean":"2024-03-16T19:29:27.344168+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:29:27.344168+0000","last_undegraded":"2024-03-16T19:29:27.344168+0000","last_fullsized":"2024-03-16T19:29:27.344168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063139+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:29:28.063139+0000","last_peered":"2024-03-16T19:29:28.063139+0000","last_clean":"2024-03-16T19:29:28.063139+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:29:28.063139+0000","last_undegraded":"2024-03-16T19:29:28.063139+0000","last_fullsized":"2024-03-16T19:29:28.063139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344198+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:29:27.344198+0000","last_peered":"2024-03-16T19:29:27.344198+0000","last_clean":"2024-03-16T19:29:27.344198+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:29:27.344198+0000","last_undegraded":"2024-03-16T19:29:27.344198+0000","last_fullsized":"2024-03-16T19:29:27.344198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062656+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:29:28.062656+0000","last_peered":"2024-03-16T19:29:28.062656+0000","last_clean":"2024-03-16T19:29:28.062656+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:29:28.062656+0000","last_undegraded":"2024-03-16T19:29:28.062656+0000","last_fullsized":"2024-03-16T19:29:28.062656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344664+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:29:27.344664+0000","last_peered":"2024-03-16T19:29:27.344664+0000","last_clean":"2024-03-16T19:29:27.344664+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:29:27.344664+0000","last_undegraded":"2024-03-16T19:29:27.344664+0000","last_fullsized":"2024-03-16T19:29:27.344664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339668+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:29:27.339668+0000","last_peered":"2024-03-16T19:29:27.339668+0000","last_clean":"2024-03-16T19:29:27.339668+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:29:27.339668+0000","last_undegraded":"2024-03-16T19:29:27.339668+0000","last_fullsized":"2024-03-16T19:29:27.339668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344128+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:29:27.344128+0000","last_peered":"2024-03-16T19:29:27.344128+0000","last_clean":"2024-03-16T19:29:27.344128+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:29:27.344128+0000","last_undegraded":"2024-03-16T19:29:27.344128+0000","last_fullsized":"2024-03-16T19:29:27.344128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062516+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:29:28.062516+0000","last_peered":"2024-03-16T19:29:28.062516+0000","last_clean":"2024-03-16T19:29:28.062516+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:29:28.062516+0000","last_undegraded":"2024-03-16T19:29:28.062516+0000","last_fullsized":"2024-03-16T19:29:28.062516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344351+0000","last_change":"2024-03-16T19:29:26.364742+0000","last_active":"2024-03-16T19:29:27.344351+0000","last_peered":"2024-03-16T19:29:27.344351+0000","last_clean":"2024-03-16T19:29:27.344351+0000","last_became_active":"2024-03-16T19:29:26.364328+0000","last_became_peered":"2024-03-16T19:29:26.364328+0000","last_unstale":"2024-03-16T19:29:27.344351+0000","last_undegraded":"2024-03-16T19:29:27.344351+0000","last_fullsized":"2024-03-16T19:29:27.344351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:18.877774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343226+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:29:27.343226+0000","last_peered":"2024-03-16T19:29:27.343226+0000","last_clean":"2024-03-16T19:29:27.343226+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:29:27.343226+0000","last_undegraded":"2024-03-16T19:29:27.343226+0000","last_fullsized":"2024-03-16T19:29:27.343226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561997+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:29:27.561997+0000","last_peered":"2024-03-16T19:29:27.561997+0000","last_clean":"2024-03-16T19:29:27.561997+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:29:27.561997+0000","last_undegraded":"2024-03-16T19:29:27.561997+0000","last_fullsized":"2024-03-16T19:29:27.561997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343035+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:29:27.343035+0000","last_peered":"2024-03-16T19:29:27.343035+0000","last_clean":"2024-03-16T19:29:27.343035+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:29:27.343035+0000","last_undegraded":"2024-03-16T19:29:27.343035+0000","last_fullsized":"2024-03-16T19:29:27.343035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062305+0000","last_change":"2024-03-16T19:29:26.352692+0000","last_active":"2024-03-16T19:29:28.062305+0000","last_peered":"2024-03-16T19:29:28.062305+0000","last_clean":"2024-03-16T19:29:28.062305+0000","last_became_active":"2024-03-16T19:29:26.352419+0000","last_became_peered":"2024-03-16T19:29:26.352419+0000","last_unstale":"2024-03-16T19:29:28.062305+0000","last_undegraded":"2024-03-16T19:29:28.062305+0000","last_fullsized":"2024-03-16T19:29:28.062305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:33.056538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062749+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:29:28.062749+0000","last_peered":"2024-03-16T19:29:28.062749+0000","last_clean":"2024-03-16T19:29:28.062749+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:29:28.062749+0000","last_undegraded":"2024-03-16T19:29:28.062749+0000","last_fullsized":"2024-03-16T19:29:28.062749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339393+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:29:27.339393+0000","last_peered":"2024-03-16T19:29:27.339393+0000","last_clean":"2024-03-16T19:29:27.339393+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:29:27.339393+0000","last_undegraded":"2024-03-16T19:29:27.339393+0000","last_fullsized":"2024-03-16T19:29:27.339393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344073+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:29:27.344073+0000","last_peered":"2024-03-16T19:29:27.344073+0000","last_clean":"2024-03-16T19:29:27.344073+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:29:27.344073+0000","last_undegraded":"2024-03-16T19:29:27.344073+0000","last_fullsized":"2024-03-16T19:29:27.344073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063637+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:29:28.063637+0000","last_peered":"2024-03-16T19:29:28.063637+0000","last_clean":"2024-03-16T19:29:28.063637+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:29:28.063637+0000","last_undegraded":"2024-03-16T19:29:28.063637+0000","last_fullsized":"2024-03-16T19:29:28.063637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339499+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:29:27.339499+0000","last_peered":"2024-03-16T19:29:27.339499+0000","last_clean":"2024-03-16T19:29:27.339499+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:29:27.339499+0000","last_undegraded":"2024-03-16T19:29:27.339499+0000","last_fullsized":"2024-03-16T19:29:27.339499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343975+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:29:27.343975+0000","last_peered":"2024-03-16T19:29:27.343975+0000","last_clean":"2024-03-16T19:29:27.343975+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:29:27.343975+0000","last_undegraded":"2024-03-16T19:29:27.343975+0000","last_fullsized":"2024-03-16T19:29:27.343975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344573+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:29:27.344573+0000","last_peered":"2024-03-16T19:29:27.344573+0000","last_clean":"2024-03-16T19:29:27.344573+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:29:27.344573+0000","last_undegraded":"2024-03-16T19:29:27.344573+0000","last_fullsized":"2024-03-16T19:29:27.344573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339540+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:29:27.339540+0000","last_peered":"2024-03-16T19:29:27.339540+0000","last_clean":"2024-03-16T19:29:27.339540+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:29:27.339540+0000","last_undegraded":"2024-03-16T19:29:27.339540+0000","last_fullsized":"2024-03-16T19:29:27.339540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339712+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:29:27.339712+0000","last_peered":"2024-03-16T19:29:27.339712+0000","last_clean":"2024-03-16T19:29:27.339712+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:29:27.339712+0000","last_undegraded":"2024-03-16T19:29:27.339712+0000","last_fullsized":"2024-03-16T19:29:27.339712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562225+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:29:27.562225+0000","last_peered":"2024-03-16T19:29:27.562225+0000","last_clean":"2024-03-16T19:29:27.562225+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:29:27.562225+0000","last_undegraded":"2024-03-16T19:29:27.562225+0000","last_fullsized":"2024-03-16T19:29:27.562225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063405+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:29:28.063405+0000","last_peered":"2024-03-16T19:29:28.063405+0000","last_clean":"2024-03-16T19:29:28.063405+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:29:28.063405+0000","last_undegraded":"2024-03-16T19:29:28.063405+0000","last_fullsized":"2024-03-16T19:29:28.063405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062725+0000","last_change":"2024-03-16T19:29:26.350351+0000","last_active":"2024-03-16T19:29:28.062725+0000","last_peered":"2024-03-16T19:29:28.062725+0000","last_clean":"2024-03-16T19:29:28.062725+0000","last_became_active":"2024-03-16T19:29:26.350074+0000","last_became_peered":"2024-03-16T19:29:26.350074+0000","last_unstale":"2024-03-16T19:29:28.062725+0000","last_undegraded":"2024-03-16T19:29:28.062725+0000","last_fullsized":"2024-03-16T19:29:28.062725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:49.416395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.342931+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:29:27.342931+0000","last_peered":"2024-03-16T19:29:27.342931+0000","last_clean":"2024-03-16T19:29:27.342931+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:29:27.342931+0000","last_undegraded":"2024-03-16T19:29:27.342931+0000","last_fullsized":"2024-03-16T19:29:27.342931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063379+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:29:28.063379+0000","last_peered":"2024-03-16T19:29:28.063379+0000","last_clean":"2024-03-16T19:29:28.063379+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:29:28.063379+0000","last_undegraded":"2024-03-16T19:29:28.063379+0000","last_fullsized":"2024-03-16T19:29:28.063379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339690+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:29:27.339690+0000","last_peered":"2024-03-16T19:29:27.339690+0000","last_clean":"2024-03-16T19:29:27.339690+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:29:27.339690+0000","last_undegraded":"2024-03-16T19:29:27.339690+0000","last_fullsized":"2024-03-16T19:29:27.339690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561913+0000","last_change":"2024-03-16T19:29:26.347388+0000","last_active":"2024-03-16T19:29:27.561913+0000","last_peered":"2024-03-16T19:29:27.561913+0000","last_clean":"2024-03-16T19:29:27.561913+0000","last_became_active":"2024-03-16T19:29:26.346582+0000","last_became_peered":"2024-03-16T19:29:26.346582+0000","last_unstale":"2024-03-16T19:29:27.561913+0000","last_undegraded":"2024-03-16T19:29:27.561913+0000","last_fullsized":"2024-03-16T19:29:27.561913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:05.294895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339497+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:29:27.339497+0000","last_peered":"2024-03-16T19:29:27.339497+0000","last_clean":"2024-03-16T19:29:27.339497+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:29:27.339497+0000","last_undegraded":"2024-03-16T19:29:27.339497+0000","last_fullsized":"2024-03-16T19:29:27.339497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561968+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:29:27.561968+0000","last_peered":"2024-03-16T19:29:27.561968+0000","last_clean":"2024-03-16T19:29:27.561968+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:29:27.561968+0000","last_undegraded":"2024-03-16T19:29:27.561968+0000","last_fullsized":"2024-03-16T19:29:27.561968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344441+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:29:27.344441+0000","last_peered":"2024-03-16T19:29:27.344441+0000","last_clean":"2024-03-16T19:29:27.344441+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:29:27.344441+0000","last_undegraded":"2024-03-16T19:29:27.344441+0000","last_fullsized":"2024-03-16T19:29:27.344441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062837+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:29:28.062837+0000","last_peered":"2024-03-16T19:29:28.062837+0000","last_clean":"2024-03-16T19:29:28.062837+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:29:28.062837+0000","last_undegraded":"2024-03-16T19:29:28.062837+0000","last_fullsized":"2024-03-16T19:29:28.062837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343809+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:29:27.343809+0000","last_peered":"2024-03-16T19:29:27.343809+0000","last_clean":"2024-03-16T19:29:27.343809+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:29:27.343809+0000","last_undegraded":"2024-03-16T19:29:27.343809+0000","last_fullsized":"2024-03-16T19:29:27.343809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343792+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:29:27.343792+0000","last_peered":"2024-03-16T19:29:27.343792+0000","last_clean":"2024-03-16T19:29:27.343792+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:29:27.343792+0000","last_undegraded":"2024-03-16T19:29:27.343792+0000","last_fullsized":"2024-03-16T19:29:27.343792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344208+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:29:27.344208+0000","last_peered":"2024-03-16T19:29:27.344208+0000","last_clean":"2024-03-16T19:29:27.344208+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:29:27.344208+0000","last_undegraded":"2024-03-16T19:29:27.344208+0000","last_fullsized":"2024-03-16T19:29:27.344208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344457+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:29:27.344457+0000","last_peered":"2024-03-16T19:29:27.344457+0000","last_clean":"2024-03-16T19:29:27.344457+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:29:27.344457+0000","last_undegraded":"2024-03-16T19:29:27.344457+0000","last_fullsized":"2024-03-16T19:29:27.344457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062891+0000","last_change":"2024-03-16T19:29:26.354303+0000","last_active":"2024-03-16T19:29:28.062891+0000","last_peered":"2024-03-16T19:29:28.062891+0000","last_clean":"2024-03-16T19:29:28.062891+0000","last_became_active":"2024-03-16T19:29:26.353804+0000","last_became_peered":"2024-03-16T19:29:26.353804+0000","last_unstale":"2024-03-16T19:29:28.062891+0000","last_undegraded":"2024-03-16T19:29:28.062891+0000","last_fullsized":"2024-03-16T19:29:28.062891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:10:45.368536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562162+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:29:27.562162+0000","last_peered":"2024-03-16T19:29:27.562162+0000","last_clean":"2024-03-16T19:29:27.562162+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:29:27.562162+0000","last_undegraded":"2024-03-16T19:29:27.562162+0000","last_fullsized":"2024-03-16T19:29:27.562162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562176+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:29:27.562176+0000","last_peered":"2024-03-16T19:29:27.562176+0000","last_clean":"2024-03-16T19:29:27.562176+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:29:27.562176+0000","last_undegraded":"2024-03-16T19:29:27.562176+0000","last_fullsized":"2024-03-16T19:29:27.562176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062982+0000","last_change":"2024-03-16T19:29:25.342597+0000","last_active":"2024-03-16T19:29:28.062982+0000","last_peered":"2024-03-16T19:29:28.062982+0000","last_clean":"2024-03-16T19:29:28.062982+0000","last_became_active":"2024-03-16T19:29:25.342320+0000","last_became_peered":"2024-03-16T19:29:25.342320+0000","last_unstale":"2024-03-16T19:29:28.062982+0000","last_undegraded":"2024-03-16T19:29:28.062982+0000","last_fullsized":"2024-03-16T19:29:28.062982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:44:41.274396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":665973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":659673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:29:30.645 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-16T19:29:30.646 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-16T19:29:30.646 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-16T19:29:30.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-16T19:29:30.646 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-16T19:29:30.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.802+0000 7fb15f664640 1 Processor -- start 2024-03-16T19:29:30.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.802+0000 7fb15f664640 1 -- start start 2024-03-16T19:29:30.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.802+0000 7fb15f664640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb158154bd0 0x7fb1580c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15f664640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15806ba50 con 0x7fb15806aed0 2024-03-16T19:29:30.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15f664640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15806bd30 con 0x7fb15815e520 2024-03-16T19:29:30.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15f664640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15815e0d0 con 0x7fb158154bd0 2024-03-16T19:29:30.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15cbff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb15806aed0 0x7fb15806dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56106/0 (socket says 172.21.15.172:56106) 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15cbff640 1 -- 172.21.15.172:0/972640068 learned_addr learned my addr 172.21.15.172:0/972640068 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15d400640 1 --2- 172.21.15.172:0/972640068 >> v2:172.21.15.174:3300/0 conn(0x7fb158154bd0 0x7fb1580c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115330501 0 0) 0x7fb15806ba50 con 0x7fb15806aed0 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb144003520 con 0x7fb15806aed0 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711136278 0 0) 0x7fb144003520 con 0x7fb15806aed0 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 >> v2:172.21.15.174:3300/0 conn(0x7fb158154bd0 msgr2=0x7fb1580c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 --2- 172.21.15.172:0/972640068 >> v2:172.21.15.174:3300/0 conn(0x7fb158154bd0 0x7fb1580c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 >> v1:172.21.15.174:6789/0 conn(0x7fb15815e520 legacy=0x7fb15815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15806c010 con 0x7fb15806aed0 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7fb148001670 con 0x7fb15806aed0 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15d400640 1 --2- 172.21.15.172:0/972640068 >> v2:172.21.15.174:3300/0 conn(0x7fb158154bd0 0x7fb1580c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 >> v1:172.21.15.172:6789/0 conn(0x7fb15806aed0 legacy=0x7fb15806dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb144003970 0x7fb144003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1440047c0 0x7fb144004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb144005180 0x7fb14400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15815e0d0 con 0x7fb1440047c0 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14400c030 con 0x7fb144003970 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14400c1f0 con 0x7fb144005180 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15cbff640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1440047c0 0x7fb144004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15d400640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb144003970 0x7fb144003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.803+0000 7fb15dc01640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb144005180 0x7fb14400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb15cbff640 1 -- 172.21.15.172:0/972640068 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb144005180 msgr2=0x7fb14400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb15cbff640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb144005180 0x7fb14400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb15cbff640 1 -- 172.21.15.172:0/972640068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb144003970 msgr2=0x7fb144003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb15cbff640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb144003970 0x7fb144003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb15dc01640 1 -- 172.21.15.172:0/972640068 reap_dead start 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb15cbff640 1 -- 172.21.15.172:0/972640068 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144003520 con 0x7fb1440047c0 2024-03-16T19:29:30.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb15cbff640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1440047c0 0x7fb144004c40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb14800c190 tx=0x7fb1440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39dfa8c32e0391ad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb148005ad0 con 0x7fb1440047c0 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb148005c30 con 0x7fb1440047c0 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.804+0000 7fb14ffff640 1 -- 172.21.15.172:0/972640068 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14802cab0 con 0x7fb1440047c0 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.805+0000 7fb15f664640 1 -- 172.21.15.172:0/972640068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1440047c0 msgr2=0x7fb144004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.805+0000 7fb15f664640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1440047c0 0x7fb144004c40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb14800c190 tx=0x7fb1440057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.805+0000 7fb15f664640 1 -- 172.21.15.172:0/972640068 shutdown_connections 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.805+0000 7fb15f664640 1 --2- 172.21.15.172:0/972640068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1440047c0 0x7fb144004c40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.805+0000 7fb15f664640 1 -- 172.21.15.172:0/972640068 >> 172.21.15.172:0/972640068 conn(0x7fb158071be0 msgr2=0x7fb158071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.805+0000 7fb15f664640 1 -- 172.21.15.172:0/972640068 shutdown_connections 2024-03-16T19:29:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.805+0000 7fb15f664640 1 -- 172.21.15.172:0/972640068 wait complete. 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.806+0000 7fb15f664640 1 Processor -- start 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.806+0000 7fb15f664640 1 -- start start 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15f664640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1440047c0 0x7fb158150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15f664640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1580c6de0 0x7fb1581e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15f664640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1581e7630 0x7fb158146b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15f664640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1581f3ad0 con 0x7fb1581e7630 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15f664640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15815bf20 con 0x7fb1440047c0 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15f664640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15815c200 con 0x7fb1580c6de0 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15dc01640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1581e7630 0x7fb158146b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15dc01640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1581e7630 0x7fb158146b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46122/0 (socket says 172.21.15.172:46122) 2024-03-16T19:29:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15dc01640 1 -- 172.21.15.172:0/3065118655 learned_addr learned my addr 172.21.15.172:0/3065118655 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15cbff640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1580c6de0 0x7fb1581e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15d400640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1440047c0 0x7fb158150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15dc01640 1 -- 172.21.15.172:0/3065118655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1580c6de0 msgr2=0x7fb1581e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15dc01640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1580c6de0 0x7fb1581e70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15dc01640 1 -- 172.21.15.172:0/3065118655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1440047c0 msgr2=0x7fb158150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15dc01640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1440047c0 0x7fb158150920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.807+0000 7fb15dc01640 1 -- 172.21.15.172:0/3065118655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb158155130 con 0x7fb1581e7630 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.808+0000 7fb15dc01640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1581e7630 0x7fb158146b60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb15806e8f0 tx=0x7fb15000fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.808+0000 7fb14dffb640 1 -- 172.21.15.172:0/3065118655 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15000fec0 con 0x7fb1581e7630 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.808+0000 7fb14dffb640 1 -- 172.21.15.172:0/3065118655 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb150003b80 con 0x7fb1581e7630 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.808+0000 7fb14dffb640 1 -- 172.21.15.172:0/3065118655 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500050d0 con 0x7fb1581e7630 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.808+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1580c6010 con 0x7fb1581e7630 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.808+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1580c65d0 con 0x7fb1581e7630 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.809+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1580681c0 con 0x7fb1581e7630 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.809+0000 7fb14dffb640 1 -- 172.21.15.172:0/3065118655 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb150005230 con 0x7fb1581e7630 2024-03-16T19:29:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.810+0000 7fb14dffb640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb13c06b3b0 0x7fb13c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:30.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.810+0000 7fb14dffb640 1 -- 172.21.15.172:0/3065118655 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb150089ab0 con 0x7fb1581e7630 2024-03-16T19:29:30.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.810+0000 7fb15d400640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb13c06b3b0 0x7fb13c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:30.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.810+0000 7fb15d400640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb13c06b3b0 0x7fb13c06d870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb1580698d0 tx=0x7fb154008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:30.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:30.814+0000 7fb14dffb640 1 -- 172.21.15.172:0/3065118655 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb150054710 con 0x7fb1581e7630 2024-03-16T19:29:31.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.031+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb15814ec50 con 0x7fb1581e7630 2024-03-16T19:29:31.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.034+0000 7fb14dffb640 1 -- 172.21.15.172:0/3065118655 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171252 (secure 0 0 0) 0x7fb150057dd0 con 0x7fb1581e7630 2024-03-16T19:29:31.016 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb13c06b3b0 msgr2=0x7fb13c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb13c06b3b0 0x7fb13c06d870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb1580698d0 tx=0x7fb154008040 comp rx=0 tx=0).stop 2024-03-16T19:29:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1581e7630 msgr2=0x7fb158146b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1581e7630 0x7fb158146b60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb15806e8f0 tx=0x7fb15000fc70 comp rx=0 tx=0).stop 2024-03-16T19:29:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 shutdown_connections 2024-03-16T19:29:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1581e7630 0x7fb158146b60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb13c06b3b0 0x7fb13c06d870 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1580c6de0 0x7fb1581e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 --2- 172.21.15.172:0/3065118655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1440047c0 0x7fb158150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.019 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 >> 172.21.15.172:0/3065118655 conn(0x7fb158071be0 msgr2=0x7fb158071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.019 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 shutdown_connections 2024-03-16T19:29:31.019 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.036+0000 7fb15f664640 1 -- 172.21.15.172:0/3065118655 wait complete. 2024-03-16T19:29:31.028 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":1161375053},{"type":"v1","addr":"172.21.15.172:6833","nonce":1161375053}]},"active_addr":"172.21.15.172:6833/1161375053","active_change":"2024-03-16T19:29:06.189534+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":14796479}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1142089520}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":3271244328}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1732045196}]}]} 2024-03-16T19:29:31.032 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-16T19:29:31.032 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-16T19:29:31.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:31.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.151+0000 7fee0a0df640 1 Processor -- start 2024-03-16T19:29:31.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.151+0000 7fee0a0df640 1 -- start start 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee0a0df640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040cc780 0x7fee040cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee0a0df640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fee040c51b0 0x7fee040cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee0a0df640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0415b880 con 0x7fee04154c90 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee0a0df640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee040c6ee0 con 0x7fee040c51b0 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee0a0df640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee04155070 con 0x7fee040cc780 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee02ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040cc780 0x7fee040cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee02ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040cc780 0x7fee040cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59706/0 (socket says 172.21.15.172:59706) 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee03fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fee040c51b0 0x7fee040cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee02ffd640 1 -- 172.21.15.172:0/904686424 learned_addr learned my addr 172.21.15.172:0/904686424 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.152+0000 7fee037fe640 1 --1- 172.21.15.172:0/904686424 >> v1:172.21.15.174:6789/0 conn(0x7fee04154c90 0x7fee0415e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:53272/0 (socket says 172.21.15.172:53272) 2024-03-16T19:29:31.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.153+0000 7fee02ffd640 1 -- 172.21.15.172:0/904686424 >> v2:172.21.15.174:3300/0 conn(0x7fee040c51b0 msgr2=0x7fee040cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.153+0000 7fee02ffd640 1 --2- 172.21.15.172:0/904686424 >> v2:172.21.15.174:3300/0 conn(0x7fee040c51b0 0x7fee040cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.153+0000 7fee02ffd640 1 -- 172.21.15.172:0/904686424 >> v1:172.21.15.174:6789/0 conn(0x7fee04154c90 legacy=0x7fee0415e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.153+0000 7fee02ffd640 1 -- 172.21.15.172:0/904686424 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0415ad00 con 0x7fee040cc780 2024-03-16T19:29:31.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.153+0000 7fee03fff640 1 --2- 172.21.15.172:0/904686424 >> v2:172.21.15.174:3300/0 conn(0x7fee040c51b0 0x7fee040cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:31.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.153+0000 7fee02ffd640 1 --2- 172.21.15.172:0/904686424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040cc780 0x7fee040cd1c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009870 tx=0x7fedf802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b666b77d4ae5a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee027fc640 1 -- 172.21.15.172:0/904686424 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8030840 con 0x7fee040cc780 2024-03-16T19:29:31.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee027fc640 1 -- 172.21.15.172:0/904686424 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf80309a0 con 0x7fee040cc780 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee027fc640 1 -- 172.21.15.172:0/904686424 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8030d30 con 0x7fee040cc780 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee0a0df640 1 -- 172.21.15.172:0/904686424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040cc780 msgr2=0x7fee040cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee0a0df640 1 --2- 172.21.15.172:0/904686424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040cc780 0x7fee040cd1c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009870 tx=0x7fedf802b980 comp rx=0 tx=0).stop 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee0a0df640 1 -- 172.21.15.172:0/904686424 shutdown_connections 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee0a0df640 1 --2- 172.21.15.172:0/904686424 >> v2:172.21.15.174:3300/0 conn(0x7fee040c51b0 0x7fee040cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee0a0df640 1 --2- 172.21.15.172:0/904686424 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040cc780 0x7fee040cd1c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee0a0df640 1 -- 172.21.15.172:0/904686424 >> 172.21.15.172:0/904686424 conn(0x7fee04067040 msgr2=0x7fee04067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee0a0df640 1 -- 172.21.15.172:0/904686424 shutdown_connections 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.154+0000 7fee0a0df640 1 -- 172.21.15.172:0/904686424 wait complete. 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.155+0000 7fee0a0df640 1 Processor -- start 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.155+0000 7fee0a0df640 1 -- start start 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee0a0df640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040c51b0 0x7fee041e4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee0a0df640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee040cc780 0x7fee041e6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee0a0df640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee04154c90 0x7fee041572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee0a0df640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041e5220 con 0x7fee040cc780 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee0a0df640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee040cef00 con 0x7fee04154c90 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee0a0df640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee040ce940 con 0x7fee040c51b0 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee02ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee040cc780 0x7fee041e6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee02ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee040cc780 0x7fee041e6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46138/0 (socket says 172.21.15.172:46138) 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee02ffd640 1 -- 172.21.15.172:0/2344074059 learned_addr learned my addr 172.21.15.172:0/2344074059 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee03fff640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee04154c90 0x7fee041572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.156+0000 7fee037fe640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040c51b0 0x7fee041e4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee02ffd640 1 -- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040c51b0 msgr2=0x7fee041e4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee02ffd640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040c51b0 0x7fee041e4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee02ffd640 1 -- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee04154c90 msgr2=0x7fee041572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee02ffd640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee04154c90 0x7fee041572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee02ffd640 1 -- 172.21.15.172:0/2344074059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc009a70 con 0x7fee040cc780 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee03fff640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee04154c90 0x7fee041572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee037fe640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040c51b0 0x7fee041e4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:31.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee02ffd640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee040cc780 0x7fee041e6440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fedf8000c00 tx=0x7fedf8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fedebfff640 1 -- 172.21.15.172:0/2344074059 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8039c90 con 0x7fee040cc780 2024-03-16T19:29:31.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fedebfff640 1 -- 172.21.15.172:0/2344074059 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8039df0 con 0x7fee040cc780 2024-03-16T19:29:31.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf80094d0 con 0x7fee040cc780 2024-03-16T19:29:31.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fedebfff640 1 -- 172.21.15.172:0/2344074059 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8043640 con 0x7fee040cc780 2024-03-16T19:29:31.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.157+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee041f0890 con 0x7fee040cc780 2024-03-16T19:29:31.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.158+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee041f06f0 con 0x7fee040cc780 2024-03-16T19:29:31.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.158+0000 7fedebfff640 1 -- 172.21.15.172:0/2344074059 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fedf80437a0 con 0x7fee040cc780 2024-03-16T19:29:31.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.159+0000 7fedebfff640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fedf006b3b0 0x7fedf006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.159+0000 7fee037fe640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fedf006b3b0 0x7fedf006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.159+0000 7fedebfff640 1 -- 172.21.15.172:0/2344074059 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedf803e070 con 0x7fee040cc780 2024-03-16T19:29:31.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.159+0000 7fee037fe640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fedf006b3b0 0x7fedf006d870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fedf4007fd0 tx=0x7fedf4007d60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.163+0000 7fedebfff640 1 -- 172.21.15.172:0/2344074059 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee041f06f0 con 0x7fee040cc780 2024-03-16T19:29:31.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.335+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee041f06f0 con 0x7fee040cc780 2024-03-16T19:29:31.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.336+0000 7fedebfff640 1 -- 172.21.15.172:0/2344074059 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16629 (secure 0 0 0) 0x7fee041f06f0 con 0x7fee040cc780 2024-03-16T19:29:31.318 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:31.318 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":21,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:27.336190+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fedf006b3b0 msgr2=0x7fedf006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fedf006b3b0 0x7fedf006d870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fedf4007fd0 tx=0x7fedf4007d60 comp rx=0 tx=0).stop 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee040cc780 msgr2=0x7fee041e6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee040cc780 0x7fee041e6440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fedf8000c00 tx=0x7fedf8004730 comp rx=0 tx=0).stop 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 shutdown_connections 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee04154c90 0x7fee041572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee040cc780 0x7fee041e6440 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fedf006b3b0 0x7fedf006d870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 --2- 172.21.15.172:0/2344074059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee040c51b0 0x7fee041e4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.338+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 >> 172.21.15.172:0/2344074059 conn(0x7fee04067040 msgr2=0x7fee0415e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.339+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 shutdown_connections 2024-03-16T19:29:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.339+0000 7fee0a0df640 1 -- 172.21.15.172:0/2344074059 wait complete. 2024-03-16T19:29:31.330 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-16T19:29:31.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:31.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.453+0000 7f597218c640 1 Processor -- start 2024-03-16T19:29:31.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.453+0000 7f597218c640 1 -- start start 2024-03-16T19:29:31.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.454+0000 7f597218c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f596c0c3610 0x7f596c0caba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.454+0000 7f597218c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c06fe10 0x7f596c0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.454+0000 7f597218c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c1590a0 con 0x7f596c152840 2024-03-16T19:29:31.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.454+0000 7f597218c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c0c4260 con 0x7f596c0c3610 2024-03-16T19:29:31.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.454+0000 7f597218c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c31c0 con 0x7f596c06fe10 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.454+0000 7f596b7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f596c152840 0x7f596c06e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56120/0 (socket says 172.21.15.172:56120) 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.454+0000 7f596b7fe640 1 -- 172.21.15.172:0/978301573 learned_addr learned my addr 172.21.15.172:0/978301573 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.454+0000 7f596bfff640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c06fe10 0x7f596c0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f5962ffd640 1 --2- 172.21.15.172:0/978301573 >> v2:172.21.15.174:3300/0 conn(0x7f596c0c3610 0x7f596c0caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405479059 0 0) 0x7f596c1590a0 con 0x7f596c152840 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5958003520 con 0x7f596c152840 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2530196871 0 0) 0x7f5958003520 con 0x7f596c152840 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c06fe10 msgr2=0x7f596c0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c06fe10 0x7f596c0cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 >> v2:172.21.15.174:3300/0 conn(0x7f596c0c3610 msgr2=0x7f596c0caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 --2- 172.21.15.172:0/978301573 >> v2:172.21.15.174:3300/0 conn(0x7f596c0c3610 0x7f596c0caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c1595d0 con 0x7f596c152840 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f595c001670 con 0x7f596c152840 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 >> v1:172.21.15.172:6789/0 conn(0x7f596c152840 legacy=0x7f596c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5958003970 0x7f5958003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59580047c0 0x7f5958004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c06fe10 0x7f5958005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c31c0 con 0x7f59580047c0 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595800b980 con 0x7f5958003970 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595800baf0 con 0x7f596c06fe10 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f5962ffd640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59580047c0 0x7f5958004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596b7fe640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5958003970 0x7f5958003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f596bfff640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c06fe10 0x7f5958005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f5962ffd640 1 -- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c06fe10 msgr2=0x7f5958005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f5962ffd640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c06fe10 0x7f5958005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f5962ffd640 1 -- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5958003970 msgr2=0x7f5958003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f5962ffd640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5958003970 0x7f5958003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.455+0000 7f5962ffd640 1 -- 172.21.15.172:0/978301573 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5958003520 con 0x7f59580047c0 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.456+0000 7f5962ffd640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59580047c0 0x7f5958004c40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5954009ac0 tx=0x7f59580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b01a122e3ee3cf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.456+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595402c6d0 con 0x7f59580047c0 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.456+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595402c830 con 0x7f59580047c0 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.456+0000 7f596affd640 1 -- 172.21.15.172:0/978301573 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595402cbc0 con 0x7f59580047c0 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.457+0000 7f597218c640 1 -- 172.21.15.172:0/978301573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59580047c0 msgr2=0x7f5958004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.457+0000 7f597218c640 1 --2- 172.21.15.172:0/978301573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59580047c0 0x7f5958004c40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5954009ac0 tx=0x7f59580057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.457+0000 7f596bfff640 1 -- 172.21.15.172:0/978301573 reap_dead start 2024-03-16T19:29:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.457+0000 7f597218c640 1 -- 172.21.15.172:0/978301573 shutdown_connections 2024-03-16T19:29:31.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.457+0000 7f597218c640 1 -- 172.21.15.172:0/978301573 >> 172.21.15.172:0/978301573 conn(0x7f596c070860 msgr2=0x7f596c070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.457+0000 7f597218c640 1 -- 172.21.15.172:0/978301573 shutdown_connections 2024-03-16T19:29:31.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.457+0000 7f597218c640 1 -- 172.21.15.172:0/978301573 wait complete. 2024-03-16T19:29:31.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.458+0000 7f597218c640 1 Processor -- start 2024-03-16T19:29:31.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.458+0000 7f597218c640 1 -- start start 2024-03-16T19:29:31.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f597218c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59580047c0 0x7f596c0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f597218c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c154c20 0x7f596c155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f597218c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596c1f1500 0x7f596c1f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f597218c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c14d650 con 0x7f596c1f1500 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f597218c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c154800 con 0x7f59580047c0 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f597218c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c154380 con 0x7f596c154c20 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596c1f1500 0x7f596c1f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596c1f1500 0x7f596c1f1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46156/0 (socket says 172.21.15.172:46156) 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596bfff640 1 -- 172.21.15.172:0/1486269814 learned_addr learned my addr 172.21.15.172:0/1486269814 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f5962ffd640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c154c20 0x7f596c155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596b7fe640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59580047c0 0x7f596c0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596bfff640 1 -- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c154c20 msgr2=0x7f596c155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596bfff640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c154c20 0x7f596c155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596bfff640 1 -- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59580047c0 msgr2=0x7f596c0ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596bfff640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59580047c0 0x7f596c0ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596bfff640 1 -- 172.21.15.172:0/1486269814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c1555c0 con 0x7f596c1f1500 2024-03-16T19:29:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f5962ffd640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c154c20 0x7f596c155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596b7fe640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59580047c0 0x7f596c0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f596bfff640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596c1f1500 0x7f596c1f1990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f596c0726d0 tx=0x7f596400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f5968ff9640 1 -- 172.21.15.172:0/1486269814 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596400e040 con 0x7f596c1f1500 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f5968ff9640 1 -- 172.21.15.172:0/1486269814 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596400f040 con 0x7f596c1f1500 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c14d370 con 0x7f596c1f1500 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f5968ff9640 1 -- 172.21.15.172:0/1486269814 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964015430 con 0x7f596c1f1500 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.459+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c14cfc0 con 0x7f596c1f1500 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.461+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c14cb40 con 0x7f596c1f1500 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.461+0000 7f5968ff9640 1 -- 172.21.15.172:0/1486269814 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5964015590 con 0x7f596c1f1500 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.461+0000 7f5968ff9640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f594c06b3b0 0x7f594c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.461+0000 7f5968ff9640 1 -- 172.21.15.172:0/1486269814 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f596408a370 con 0x7f596c1f1500 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.461+0000 7f596b7fe640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f594c06b3b0 0x7f594c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.462+0000 7f596b7fe640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f594c06b3b0 0x7f594c06d870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f595c001eb0 tx=0x7f595c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.466+0000 7f5968ff9640 1 -- 172.21.15.172:0/1486269814 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f596c14cb40 con 0x7f596c1f1500 2024-03-16T19:29:31.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.638+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f596c154940 con 0x7f596c1f1500 2024-03-16T19:29:31.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.639+0000 7f5968ff9640 1 -- 172.21.15.172:0/1486269814 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f596c154940 con 0x7f596c1f1500 2024-03-16T19:29:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:31.621 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":21,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:27.336190+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f594c06b3b0 msgr2=0x7f594c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f594c06b3b0 0x7f594c06d870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f595c001eb0 tx=0x7f595c009040 comp rx=0 tx=0).stop 2024-03-16T19:29:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596c1f1500 msgr2=0x7f596c1f1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596c1f1500 0x7f596c1f1990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f596c0726d0 tx=0x7f596400f580 comp rx=0 tx=0).stop 2024-03-16T19:29:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 shutdown_connections 2024-03-16T19:29:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f596c1f1500 0x7f596c1f1990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f594c06b3b0 0x7f594c06d870 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f596c154c20 0x7f596c155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 --2- 172.21.15.172:0/1486269814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59580047c0 0x7f596c0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.641+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 >> 172.21.15.172:0/1486269814 conn(0x7f596c070860 msgr2=0x7f596c158700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.642+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 shutdown_connections 2024-03-16T19:29:31.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.642+0000 7f597218c640 1 -- 172.21.15.172:0/1486269814 wait complete. 2024-03-16T19:29:31.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-16T19:29:31.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-16T19:29:31.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-16T19:29:31.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-16T19:29:31.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-16T19:29:31.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-16T19:29:31.637 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-16T19:29:31.637 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-16T19:29:31.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.826+0000 7ff60f209640 1 Processor -- start 2024-03-16T19:29:31.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.826+0000 7ff60f209640 1 -- start start 2024-03-16T19:29:31.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.828+0000 7ff60f209640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff6080c7a90 0x7ff60806c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.828+0000 7ff60f209640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff608159280 con 0x7ff6080c6c80 2024-03-16T19:29:31.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7f2e99df4640 1 Processor -- start 2024-03-16T19:29:31.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7f2e99df4640 1 -- start start 2024-03-16T19:29:31.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7f4faea88640 1 Processor -- start 2024-03-16T19:29:31.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7f19b131f640 1 Processor -- start 2024-03-16T19:29:31.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7f19b131f640 1 -- start start 2024-03-16T19:29:31.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7fdaed62a640 1 Processor -- start 2024-03-16T19:29:31.812 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.828+0000 7ff60f209640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff608071650 con 0x7ff60815f870 2024-03-16T19:29:31.812 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.828+0000 7ff60f209640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff608071910 con 0x7ff6080c7a90 2024-03-16T19:29:31.812 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.828+0000 7ff60ea08640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff6080c6c80 0x7ff6081694b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56130/0 (socket says 172.21.15.172:56130) 2024-03-16T19:29:31.812 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.828+0000 7ff60ea08640 1 -- 172.21.15.172:0/3965934477 learned_addr learned my addr 172.21.15.172:0/3965934477 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f2e99df4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e94157f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f2e937fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2e941516b0 0x7f2e9415b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56134/0 (socket says 172.21.15.172:56134) 2024-03-16T19:29:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f2e937fe640 1 -- 172.21.15.172:0/1787991012 learned_addr learned my addr 172.21.15.172:0/1787991012 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f2e99df4640 1 -- 172.21.15.172:0/1787991012 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e940c6990 con 0x7f2e941516b0 2024-03-16T19:29:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f2e99df4640 1 -- 172.21.15.172:0/1787991012 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e940c60f0 con 0x7f2e94151a90 2024-03-16T19:29:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f2e93fff640 1 --2- 172.21.15.172:0/1787991012 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e94157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f2e99df4640 1 -- 172.21.15.172:0/1787991012 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e940c63d0 con 0x7f2e940c39d0 2024-03-16T19:29:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f2e927fc640 1 -- 172.21.15.172:0/1787991012 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657188770 0 0) 0x7f2e940c60f0 con 0x7f2e94151a90 2024-03-16T19:29:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f2e927fc640 1 -- 172.21.15.172:0/1787991012 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e7c003520 con 0x7f2e94151a90 2024-03-16T19:29:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f2e93fff640 1 -- 172.21.15.172:0/1787991012 >> v1:172.21.15.174:6789/0 conn(0x7f2e94151a90 legacy=0x7f2e940c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f2e93fff640 1 -- 172.21.15.172:0/1787991012 >> v1:172.21.15.172:6789/0 conn(0x7f2e941516b0 legacy=0x7f2e9415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f2e93fff640 1 -- 172.21.15.172:0/1787991012 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e940c66b0 con 0x7f2e940c39d0 2024-03-16T19:29:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364844550 0 0) 0x7ff604001040 con 0x7ff6080c6c80 2024-03-16T19:29:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f4003520 con 0x7ff6080c6c80 2024-03-16T19:29:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599227838 0 0) 0x7ff608071650 con 0x7ff60815f870 2024-03-16T19:29:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff604001040 con 0x7ff60815f870 2024-03-16T19:29:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60da06640 1 --2- 172.21.15.172:0/3965934477 >> v2:172.21.15.172:3300/0 conn(0x7ff6080c7a90 0x7ff60806c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4004248400 0 0) 0x7ff5f4003520 con 0x7ff6080c6c80 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 >> v2:172.21.15.172:3300/0 conn(0x7ff6080c7a90 msgr2=0x7ff60806c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 --2- 172.21.15.172:0/3965934477 >> v2:172.21.15.172:3300/0 conn(0x7ff6080c7a90 0x7ff60806c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 >> v1:172.21.15.174:6789/0 conn(0x7ff60815f870 legacy=0x7ff608162920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff608071bf0 con 0x7ff6080c6c80 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7ff6040016b0 con 0x7ff6080c6c80 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60da06640 1 --2- 172.21.15.172:0/3965934477 >> v2:172.21.15.172:3300/0 conn(0x7ff6080c7a90 0x7ff60806c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 >> v1:172.21.15.172:6789/0 conn(0x7ff6080c6c80 legacy=0x7ff6081694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5f4003ba0 0x7ff5f4003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f40047c0 0x7ff5f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7ff60d205640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5f4005380 0x7ff5f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff608071910 con 0x7ff5f4003ba0 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f400c180 con 0x7ff5f40047c0 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f400c340 con 0x7ff5f4005380 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7ff60e207640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5f4003ba0 0x7ff5f4003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7ff60ea08640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5f4005380 0x7ff5f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7ff60da06640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f40047c0 0x7ff5f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7ff60e207640 1 -- 172.21.15.172:0/3965934477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5f4005380 msgr2=0x7ff5f400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7ff60e207640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5f4005380 0x7ff5f400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7ff60e207640 1 -- 172.21.15.172:0/3965934477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f40047c0 msgr2=0x7ff5f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7ff60e207640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f40047c0 0x7ff5f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7ff60e207640 1 -- 172.21.15.172:0/3965934477 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f4003520 con 0x7ff5f4003ba0 2024-03-16T19:29:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7ff60ea08640 1 -- 172.21.15.172:0/3965934477 reap_dead start 2024-03-16T19:29:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7fe349623640 1 Processor -- start 2024-03-16T19:29:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7fe349623640 1 -- start start 2024-03-16T19:29:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7fdaed62a640 1 -- start start 2024-03-16T19:29:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fdaed62a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae8158b30 0x7fdae80c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fdaed62a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdae80c8360 0x7fdae81627b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fdaed62a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae80c87b0 con 0x7fdae80730e0 2024-03-16T19:29:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fdaed62a640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae815ee40 con 0x7fdae80c8360 2024-03-16T19:29:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fdaed62a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae815f120 con 0x7fdae8158b30 2024-03-16T19:29:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.829+0000 7f4faea88640 1 -- start start 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f4faea88640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4fa8156dc0 0x7f4fa80c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f4faea88640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4fa806dbd0 0x7f4fa8150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f4faea88640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa80c2230 0x7f4fa80c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f4faea88640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa80c1da0 con 0x7f4fa806dbd0 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f4faea88640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa8151220 con 0x7f4fa8156dc0 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f4faea88640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa806e030 con 0x7f4fa80c2230 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f4fa7fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4fa806dbd0 0x7f4fa8150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f4fa7fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4fa806dbd0 0x7f4fa8150ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46174/0 (socket says 172.21.15.172:46174) 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f4fa7fff640 1 -- 172.21.15.172:0/3547990947 learned_addr learned my addr 172.21.15.172:0/3547990947 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f4fad025640 1 --2- 172.21.15.172:0/3547990947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa80c2230 0x7f4fa80c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f4fa7fff640 1 -- 172.21.15.172:0/3547990947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa80c2230 msgr2=0x7f4fa80c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f4fa7fff640 1 --2- 172.21.15.172:0/3547990947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa80c2230 0x7f4fa80c26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f4fa7fff640 1 -- 172.21.15.172:0/3547990947 >> v2:172.21.15.174:3300/0 conn(0x7f4fa8156dc0 msgr2=0x7f4fa80c84b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f4fa7fff640 1 --2- 172.21.15.172:0/3547990947 >> v2:172.21.15.174:3300/0 conn(0x7f4fa8156dc0 0x7f4fa80c84b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f4fa7fff640 1 -- 172.21.15.172:0/3547990947 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa8157370 con 0x7f4fa806dbd0 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f19b131f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f19a40db210 0x7f19a40d5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f19b131f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f19a40dce10 0x7f19a404be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f19abfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f19a40db210 0x7f19a40d5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.830+0000 7f19abfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f19a40db210 0x7f19a40d5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42502/0 (socket says 172.21.15.172:42502) 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19b131f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a40080e0 con 0x7f19a404b9b0 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19abfff640 1 -- 172.21.15.172:0/3740853815 learned_addr learned my addr 172.21.15.172:0/3740853815 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19b131f640 1 -- 172.21.15.172:0/3740853815 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a4047c50 con 0x7f19a40dce10 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19b131f640 1 -- 172.21.15.172:0/3740853815 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a40db5f0 con 0x7f19a40db210 2024-03-16T19:29:31.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19aaffd640 1 -- 172.21.15.172:0/3740853815 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968889577 0 0) 0x7f19a40080e0 con 0x7f19a404b9b0 2024-03-16T19:29:31.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19aaffd640 1 -- 172.21.15.172:0/3740853815 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199c003520 con 0x7f19a404b9b0 2024-03-16T19:29:31.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19abfff640 1 -- 172.21.15.172:0/3740853815 >> v2:172.21.15.172:3300/0 conn(0x7f19a40dce10 msgr2=0x7f19a404be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19abfff640 1 --2- 172.21.15.172:0/3740853815 >> v2:172.21.15.172:3300/0 conn(0x7f19a40dce10 0x7f19a404be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19abfff640 1 -- 172.21.15.172:0/3740853815 >> v1:172.21.15.174:6789/0 conn(0x7f19a404b9b0 legacy=0x7f19a40dc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f19abfff640 1 -- 172.21.15.172:0/3740853815 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a40d56b0 con 0x7f19a40db210 2024-03-16T19:29:31.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fe349623640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fe34415ece0 0x7fe3440c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fe349623640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344070650 0x7fe3440c9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7f3a90455640 1 Processor -- start 2024-03-16T19:29:31.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fe349623640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34415f190 con 0x7fe344159020 2024-03-16T19:29:31.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fe349623640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34415f470 con 0x7fe34415ece0 2024-03-16T19:29:31.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.831+0000 7fe349623640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34415f750 con 0x7fe344070650 2024-03-16T19:29:31.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fe343fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344070650 0x7fe3440c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fe343fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344070650 0x7fe3440c9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59786/0 (socket says 172.21.15.172:59786) 2024-03-16T19:29:31.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fe343fff640 1 -- 172.21.15.172:0/2873628884 learned_addr learned my addr 172.21.15.172:0/2873628884 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fe3427fc640 1 --2- 172.21.15.172:0/2873628884 >> v2:172.21.15.172:3300/0 conn(0x7fe34415ece0 0x7fe3440c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7f3a90455640 1 -- start start 2024-03-16T19:29:31.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fdae6ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae8158b30 0x7fdae80c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fe3427fc640 1 -- 172.21.15.172:0/2873628884 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344070650 msgr2=0x7fe3440c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fe3427fc640 1 --2- 172.21.15.172:0/2873628884 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344070650 0x7fe3440c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fe3427fc640 1 -- 172.21.15.172:0/2873628884 >> v1:172.21.15.172:6789/0 conn(0x7fe344159020 legacy=0x7fe34406ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fe3427fc640 1 -- 172.21.15.172:0/2873628884 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34415fa30 con 0x7fe34415ece0 2024-03-16T19:29:31.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7f19abfff640 1 --2- 172.21.15.172:0/3740853815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a40db210 0x7f19a40d5170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f19a4008d40 tx=0x7f19a002ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=968969b654d35836 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7f2e93fff640 1 --2- 172.21.15.172:0/1787991012 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e94157f60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2e880097d0 tx=0x7f2e8802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9404a7bd2b309701 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fdae6ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae8158b30 0x7fdae80c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59782/0 (socket says 172.21.15.172:59782) 2024-03-16T19:29:31.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7f4fa7fff640 1 --2- 172.21.15.172:0/3547990947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa806dbd0 0x7f4fa8150ce0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4fa80c8b00 tx=0x7f4f9402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad97cc18db67838 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fae63a6b640 1 Processor -- start 2024-03-16T19:29:31.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7f19aaffd640 1 -- 172.21.15.172:0/3740853815 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a002f870 con 0x7f19a40db210 2024-03-16T19:29:31.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f19aaffd640 1 -- 172.21.15.172:0/3740853815 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a002f9d0 con 0x7f19a40db210 2024-03-16T19:29:31.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f19aaffd640 1 -- 172.21.15.172:0/3740853815 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a002fd60 con 0x7f19a40db210 2024-03-16T19:29:31.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fdae6ffd640 1 -- 172.21.15.172:0/4286099564 learned_addr learned my addr 172.21.15.172:0/4286099564 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f2e927fc640 1 -- 172.21.15.172:0/1787991012 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88031890 con 0x7f2e940c39d0 2024-03-16T19:29:31.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.832+0000 7fdae7fff640 1 --2- 172.21.15.172:0/4286099564 >> v2:172.21.15.172:3300/0 conn(0x7fdae80c8360 0x7fdae81627b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f2e927fc640 1 -- 172.21.15.172:0/1787991012 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e880319f0 con 0x7f2e940c39d0 2024-03-16T19:29:31.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7fdae7fff640 1 -- 172.21.15.172:0/4286099564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae8158b30 msgr2=0x7fdae80c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f2e927fc640 1 -- 172.21.15.172:0/1787991012 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88031d80 con 0x7f2e940c39d0 2024-03-16T19:29:31.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7fdae7fff640 1 --2- 172.21.15.172:0/4286099564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae8158b30 0x7fdae80c7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f4fa77fe640 1 -- 172.21.15.172:0/3547990947 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f94030870 con 0x7f4fa806dbd0 2024-03-16T19:29:31.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f4fa77fe640 1 -- 172.21.15.172:0/3547990947 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f940309d0 con 0x7f4fa806dbd0 2024-03-16T19:29:31.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f4fa77fe640 1 -- 172.21.15.172:0/3547990947 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f94030d60 con 0x7f4fa806dbd0 2024-03-16T19:29:31.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7fdae7fff640 1 -- 172.21.15.172:0/4286099564 >> v1:172.21.15.172:6789/0 conn(0x7fdae80730e0 legacy=0x7fdae81620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7fdae7fff640 1 -- 172.21.15.172:0/4286099564 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae815f400 con 0x7fdae80c8360 2024-03-16T19:29:31.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f3a90455640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3a88154c70 0x7f3a880cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7fae63a6b640 1 -- start start 2024-03-16T19:29:31.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f3a90455640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3a880caee0 0x7f3a880cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7ff60e207640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5f4003ba0 0x7ff5f4003fa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff600001fd0 tx=0x7ff5f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9682bef26ceb95 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f3a8e1f1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3a88154c70 0x7f3a880cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f3a8e1f1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3a88154c70 0x7f3a880cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42518/0 (socket says 172.21.15.172:42518) 2024-03-16T19:29:31.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7f3a8e1f1640 1 -- 172.21.15.172:0/788529572 learned_addr learned my addr 172.21.15.172:0/788529572 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600003b70 con 0x7ff5f4003ba0 2024-03-16T19:29:31.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff600003cd0 con 0x7ff5f4003ba0 2024-03-16T19:29:31.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f3a8d9f0640 1 --2- 172.21.15.172:0/788529572 >> v2:172.21.15.172:3300/0 conn(0x7f3a880caee0 0x7f3a880cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.833+0000 7fe3427fc640 1 --2- 172.21.15.172:0/2873628884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34415ece0 0x7fe3440c91e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe3380097a0 tx=0x7fe33802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430644df950f9274 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f3a90455640 1 -- 172.21.15.172:0/788529572 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8815ded0 con 0x7f3a8806b2b0 2024-03-16T19:29:31.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f3a90455640 1 -- 172.21.15.172:0/788529572 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a8815ed30 con 0x7f3a880caee0 2024-03-16T19:29:31.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f3a90455640 1 -- 172.21.15.172:0/788529572 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a8815e1b0 con 0x7f3a88154c70 2024-03-16T19:29:31.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f19b131f640 1 -- 172.21.15.172:0/3740853815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a40db210 msgr2=0x7f19a40d5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f19b131f640 1 --2- 172.21.15.172:0/3740853815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a40db210 0x7f19a40d5170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f19a4008d40 tx=0x7f19a002ba50 comp rx=0 tx=0).stop 2024-03-16T19:29:31.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f19b131f640 1 -- 172.21.15.172:0/3740853815 shutdown_connections 2024-03-16T19:29:31.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fae63a6b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fae5c0b5c20 0x7fae5c15ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fae63a6b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fae5c0cd0d0 0x7fae5c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fae63a6b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c162c80 con 0x7fae5c072ba0 2024-03-16T19:29:31.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fae63a6b640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c0b53a0 con 0x7fae5c0cd0d0 2024-03-16T19:29:31.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fae63a6b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c0b3fd0 con 0x7fae5c0b5c20 2024-03-16T19:29:31.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fe341ffb640 1 -- 172.21.15.172:0/2873628884 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33802e070 con 0x7fe34415ece0 2024-03-16T19:29:31.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fe341ffb640 1 -- 172.21.15.172:0/2873628884 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3380315a0 con 0x7fe34415ece0 2024-03-16T19:29:31.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f19b131f640 1 --2- 172.21.15.172:0/3740853815 >> v2:172.21.15.172:3300/0 conn(0x7f19a40dce10 0x7f19a404be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f19b131f640 1 --2- 172.21.15.172:0/3740853815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a40db210 0x7f19a40d5170 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f19b131f640 1 -- 172.21.15.172:0/3740853815 >> 172.21.15.172:0/3740853815 conn(0x7f19a404d4b0 msgr2=0x7f19a404d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fe341ffb640 1 -- 172.21.15.172:0/2873628884 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338031930 con 0x7fe34415ece0 2024-03-16T19:29:31.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f4faea88640 1 -- 172.21.15.172:0/3547990947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa806dbd0 msgr2=0x7f4fa8150ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f4faea88640 1 --2- 172.21.15.172:0/3547990947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa806dbd0 0x7f4fa8150ce0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4fa80c8b00 tx=0x7f4f9402b9b0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7ff60d205640 1 -- 172.21.15.172:0/3965934477 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000052f0 con 0x7ff5f4003ba0 2024-03-16T19:29:31.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fae61807640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fae5c072ba0 0x7fae5c0b38c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56160/0 (socket says 172.21.15.172:56160) 2024-03-16T19:29:31.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fae61807640 1 -- 172.21.15.172:0/3216270985 learned_addr learned my addr 172.21.15.172:0/3216270985 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:31.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fae62008640 1 --2- 172.21.15.172:0/3216270985 >> v2:172.21.15.172:3300/0 conn(0x7fae5c0cd0d0 0x7fae5c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fae61006640 1 --2- 172.21.15.172:0/3216270985 >> v2:172.21.15.174:3300/0 conn(0x7fae5c0b5c20 0x7fae5c15ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fae62008640 1 -- 172.21.15.172:0/3216270985 >> v2:172.21.15.174:3300/0 conn(0x7fae5c0b5c20 msgr2=0x7fae5c15ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fae62008640 1 --2- 172.21.15.172:0/3216270985 >> v2:172.21.15.174:3300/0 conn(0x7fae5c0b5c20 0x7fae5c15ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fae62008640 1 -- 172.21.15.172:0/3216270985 >> v1:172.21.15.172:6789/0 conn(0x7fae5c072ba0 legacy=0x7fae5c0b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fae62008640 1 -- 172.21.15.172:0/3216270985 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c072f80 con 0x7fae5c0cd0d0 2024-03-16T19:29:31.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fae61006640 1 --2- 172.21.15.172:0/3216270985 >> v2:172.21.15.174:3300/0 conn(0x7fae5c0b5c20 0x7fae5c15ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:31.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f2e99df4640 1 -- 172.21.15.172:0/1787991012 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 msgr2=0x7f2e94157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f2e99df4640 1 --2- 172.21.15.172:0/1787991012 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e94157f60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2e880097d0 tx=0x7f2e8802b930 comp rx=0 tx=0).stop 2024-03-16T19:29:31.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f2e99df4640 1 -- 172.21.15.172:0/1787991012 shutdown_connections 2024-03-16T19:29:31.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f2e99df4640 1 --2- 172.21.15.172:0/1787991012 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e94157f60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f2e99df4640 1 -- 172.21.15.172:0/1787991012 >> 172.21.15.172:0/1787991012 conn(0x7f2e940cc4d0 msgr2=0x7f2e940cc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7ff60f209640 1 -- 172.21.15.172:0/3965934477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5f4003ba0 msgr2=0x7ff5f4003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7ff60f209640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5f4003ba0 0x7ff5f4003fa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff600001fd0 tx=0x7ff5f40057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7ff60f209640 1 -- 172.21.15.172:0/3965934477 shutdown_connections 2024-03-16T19:29:31.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7ff60f209640 1 --2- 172.21.15.172:0/3965934477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5f4003ba0 0x7ff5f4003fa0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7ff60f209640 1 -- 172.21.15.172:0/3965934477 >> 172.21.15.172:0/3965934477 conn(0x7ff6080cc280 msgr2=0x7ff6080cc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fe349623640 1 -- 172.21.15.172:0/2873628884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34415ece0 msgr2=0x7fe3440c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fe349623640 1 --2- 172.21.15.172:0/2873628884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34415ece0 0x7fe3440c91e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe3380097a0 tx=0x7fe33802b840 comp rx=0 tx=0).stop 2024-03-16T19:29:31.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fe349623640 1 -- 172.21.15.172:0/2873628884 shutdown_connections 2024-03-16T19:29:31.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fe349623640 1 --2- 172.21.15.172:0/2873628884 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344070650 0x7fe3440c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fe349623640 1 --2- 172.21.15.172:0/2873628884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe34415ece0 0x7fe3440c91e0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe3380097a0 tx=0x7fe33802b840 comp rx=0 tx=0).stop 2024-03-16T19:29:31.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fe349623640 1 -- 172.21.15.172:0/2873628884 >> 172.21.15.172:0/2873628884 conn(0x7fe3440ca0c0 msgr2=0x7fe3440ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fdae7fff640 1 --2- 172.21.15.172:0/4286099564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae80c8360 0x7fdae81627b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009d90 tx=0x7fdae002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8907252a443fd71 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7fdae5ffb640 1 -- 172.21.15.172:0/4286099564 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae002cce0 con 0x7fdae80c8360 2024-03-16T19:29:31.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fdae5ffb640 1 -- 172.21.15.172:0/4286099564 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae002ce40 con 0x7fdae80c8360 2024-03-16T19:29:31.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fdae5ffb640 1 -- 172.21.15.172:0/4286099564 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0034660 con 0x7fdae80c8360 2024-03-16T19:29:31.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f3a8d1ef640 1 -- 172.21.15.172:0/788529572 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816975032 0 0) 0x7f3a8815ded0 con 0x7f3a8806b2b0 2024-03-16T19:29:31.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f3a8d1ef640 1 -- 172.21.15.172:0/788529572 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a74003520 con 0x7f3a8806b2b0 2024-03-16T19:29:31.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f3a8d9f0640 1 -- 172.21.15.172:0/788529572 >> v2:172.21.15.174:3300/0 conn(0x7f3a88154c70 msgr2=0x7f3a880cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:31.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f3a8d9f0640 1 --2- 172.21.15.172:0/788529572 >> v2:172.21.15.174:3300/0 conn(0x7f3a88154c70 0x7f3a880cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f3a8d9f0640 1 -- 172.21.15.172:0/788529572 >> v1:172.21.15.174:6789/0 conn(0x7f3a8806b2b0 legacy=0x7f3a8806b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f3a8d9f0640 1 -- 172.21.15.172:0/788529572 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8815e490 con 0x7f3a880caee0 2024-03-16T19:29:31.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f4faea88640 1 -- 172.21.15.172:0/3547990947 shutdown_connections 2024-03-16T19:29:31.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f4faea88640 1 --2- 172.21.15.172:0/3547990947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa80c2230 0x7f4fa80c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f4faea88640 1 --2- 172.21.15.172:0/3547990947 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa806dbd0 0x7f4fa8150ce0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f4faea88640 1 --2- 172.21.15.172:0/3547990947 >> v2:172.21.15.174:3300/0 conn(0x7f4fa8156dc0 0x7f4fa80c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7f4faea88640 1 -- 172.21.15.172:0/3547990947 >> 172.21.15.172:0/3547990947 conn(0x7f4fa80cc030 msgr2=0x7f4fa80cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f19b131f640 1 -- 172.21.15.172:0/3740853815 shutdown_connections 2024-03-16T19:29:31.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fe349623640 1 -- 172.21.15.172:0/2873628884 shutdown_connections 2024-03-16T19:29:31.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.834+0000 7f19b131f640 1 -- 172.21.15.172:0/3740853815 wait complete. 2024-03-16T19:29:31.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fdaed62a640 1 -- 172.21.15.172:0/4286099564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae80c8360 msgr2=0x7fdae81627b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fdaed62a640 1 --2- 172.21.15.172:0/4286099564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae80c8360 0x7fdae81627b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009d90 tx=0x7fdae002bf30 comp rx=0 tx=0).stop 2024-03-16T19:29:31.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7f4faea88640 1 -- 172.21.15.172:0/3547990947 shutdown_connections 2024-03-16T19:29:31.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7ff60f209640 1 -- 172.21.15.172:0/3965934477 shutdown_connections 2024-03-16T19:29:31.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7ff60f209640 1 -- 172.21.15.172:0/3965934477 wait complete. 2024-03-16T19:29:31.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7f4faea88640 1 -- 172.21.15.172:0/3547990947 wait complete. 2024-03-16T19:29:31.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7fdaed62a640 1 -- 172.21.15.172:0/4286099564 shutdown_connections 2024-03-16T19:29:31.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7fdaed62a640 1 --2- 172.21.15.172:0/4286099564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae80c8360 0x7fdae81627b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7fdaed62a640 1 --2- 172.21.15.172:0/4286099564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae8158b30 0x7fdae80c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7fdaed62a640 1 -- 172.21.15.172:0/4286099564 >> 172.21.15.172:0/4286099564 conn(0x7fdae80c8b30 msgr2=0x7fdae80c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7f2e99df4640 1 -- 172.21.15.172:0/1787991012 shutdown_connections 2024-03-16T19:29:31.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7fdaed62a640 1 -- 172.21.15.172:0/4286099564 shutdown_connections 2024-03-16T19:29:31.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7f3a8d9f0640 1 --2- 172.21.15.172:0/788529572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a880caee0 0x7f3a880cc160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3a780097a0 tx=0x7f3a7802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c762da353b273c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7f19b131f640 1 Processor -- start 2024-03-16T19:29:31.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.835+0000 7fe349623640 1 -- 172.21.15.172:0/2873628884 wait complete. 2024-03-16T19:29:31.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7f3a8d1ef640 1 -- 172.21.15.172:0/788529572 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78030610 con 0x7f3a880caee0 2024-03-16T19:29:31.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7fdaed62a640 1 -- 172.21.15.172:0/4286099564 wait complete. 2024-03-16T19:29:31.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7f3a8d1ef640 1 -- 172.21.15.172:0/788529572 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a78030770 con 0x7f3a880caee0 2024-03-16T19:29:31.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7f3a8d1ef640 1 -- 172.21.15.172:0/788529572 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78030b00 con 0x7f3a880caee0 2024-03-16T19:29:31.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7fae62008640 1 --2- 172.21.15.172:0/3216270985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae5c0cd0d0 0x7fae5c0cd560 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0b52f0 tx=0x7fae5402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233b667d149e3e92 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.837+0000 7fae60805640 1 -- 172.21.15.172:0/3216270985 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5402cd10 con 0x7fae5c0cd0d0 2024-03-16T19:29:31.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.837+0000 7fae60805640 1 -- 172.21.15.172:0/3216270985 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5402ce70 con 0x7fae5c0cd0d0 2024-03-16T19:29:31.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.837+0000 7fae60805640 1 -- 172.21.15.172:0/3216270985 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54034640 con 0x7fae5c0cd0d0 2024-03-16T19:29:31.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.837+0000 7f2e99df4640 1 -- 172.21.15.172:0/1787991012 wait complete. 2024-03-16T19:29:31.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.837+0000 7ff60f209640 1 Processor -- start 2024-03-16T19:29:31.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.837+0000 7fe349623640 1 Processor -- start 2024-03-16T19:29:31.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.836+0000 7f19b131f640 1 -- start start 2024-03-16T19:29:31.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.837+0000 7f19b131f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a404b9b0 0x7f19a4036450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fae63a6b640 1 -- 172.21.15.172:0/3216270985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae5c0cd0d0 msgr2=0x7fae5c0cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fae63a6b640 1 --2- 172.21.15.172:0/3216270985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae5c0cd0d0 0x7fae5c0cd560 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0b52f0 tx=0x7fae5402bf60 comp rx=0 tx=0).stop 2024-03-16T19:29:31.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7f4faea88640 1 Processor -- start 2024-03-16T19:29:31.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7f4faea88640 1 -- start start 2024-03-16T19:29:31.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7f19b131f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a40db210 0x7f19a4050720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fae63a6b640 1 -- 172.21.15.172:0/3216270985 shutdown_connections 2024-03-16T19:29:31.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fae63a6b640 1 --2- 172.21.15.172:0/3216270985 >> v2:172.21.15.174:3300/0 conn(0x7fae5c0b5c20 0x7fae5c15ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fae63a6b640 1 --2- 172.21.15.172:0/3216270985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae5c0cd0d0 0x7fae5c0cd560 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fae63a6b640 1 -- 172.21.15.172:0/3216270985 >> 172.21.15.172:0/3216270985 conn(0x7fae5c067040 msgr2=0x7fae5c067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fdaed62a640 1 Processor -- start 2024-03-16T19:29:31.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fdaed62a640 1 -- start start 2024-03-16T19:29:31.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7f3a90455640 1 -- 172.21.15.172:0/788529572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a880caee0 msgr2=0x7f3a880cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7f3a90455640 1 --2- 172.21.15.172:0/788529572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a880caee0 0x7f3a880cc160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3a780097a0 tx=0x7f3a7802b960 comp rx=0 tx=0).stop 2024-03-16T19:29:31.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.837+0000 7ff60f209640 1 -- start start 2024-03-16T19:29:31.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7ff60f209640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5f4003ba0 0x7ff60815c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7ff60f209640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff608137830 0x7ff60815a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fae63a6b640 1 -- 172.21.15.172:0/3216270985 shutdown_connections 2024-03-16T19:29:31.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fae63a6b640 1 -- 172.21.15.172:0/3216270985 wait complete. 2024-03-16T19:29:31.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7ff60f209640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6080c58d0 0x7ff608147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7ff60f209640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff608137c70 con 0x7ff608137830 2024-03-16T19:29:31.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7ff60f209640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff608132f00 con 0x7ff6080c58d0 2024-03-16T19:29:31.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7ff60f209640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff608137ee0 con 0x7ff5f4003ba0 2024-03-16T19:29:31.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7ff60da06640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff608137830 0x7ff60815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7ff60da06640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff608137830 0x7ff60815a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46222/0 (socket says 172.21.15.172:46222) 2024-03-16T19:29:31.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7ff60da06640 1 -- 172.21.15.172:0/2166249101 learned_addr learned my addr 172.21.15.172:0/2166249101 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.837+0000 7fe349623640 1 -- start start 2024-03-16T19:29:31.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7f3a90455640 1 -- 172.21.15.172:0/788529572 shutdown_connections 2024-03-16T19:29:31.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7f3a90455640 1 --2- 172.21.15.172:0/788529572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a880caee0 0x7f3a880cc160 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7f3a90455640 1 --2- 172.21.15.172:0/788529572 >> v2:172.21.15.174:3300/0 conn(0x7f3a88154c70 0x7f3a880cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7f3a90455640 1 -- 172.21.15.172:0/788529572 >> 172.21.15.172:0/788529572 conn(0x7f3a88067040 msgr2=0x7f3a88067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.838+0000 7fe349623640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe344070650 0x7fe3441f8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19abfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a404b9b0 0x7f19a4036450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7fe349623640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe344159020 0x7fe3440cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7ff60e207640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5f4003ba0 0x7ff60815c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19abfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a404b9b0 0x7f19a4036450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42540/0 (socket says 172.21.15.172:42540) 2024-03-16T19:29:31.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19b131f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19a40dce10 0x7f19a4050c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19b131f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a40d3910 con 0x7f19a40db210 2024-03-16T19:29:31.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19ab7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a40db210 0x7f19a4050720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19b131f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a404ffb0 con 0x7f19a404b9b0 2024-03-16T19:29:31.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19b131f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4181bb0 con 0x7f19a40dce10 2024-03-16T19:29:31.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19ab7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a40db210 0x7f19a4050720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46218/0 (socket says 172.21.15.172:46218) 2024-03-16T19:29:31.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19ab7fe640 1 -- 172.21.15.172:0/3979271778 learned_addr learned my addr 172.21.15.172:0/3979271778 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7fe349623640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344154570 0x7fe3441f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7fe342ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe344070650 0x7fe3441f8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7fe342ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe344070650 0x7fe3441f8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46230/0 (socket says 172.21.15.172:46230) 2024-03-16T19:29:31.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7fe3427fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe344159020 0x7fe3440cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7fe3427fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe344159020 0x7fe3440cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42548/0 (socket says 172.21.15.172:42548) 2024-03-16T19:29:31.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7fdaed62a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae80730e0 0x7fdae81f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f4faea88640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fa806dbd0 0x7f4fa80c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19ab7fe640 1 -- 172.21.15.172:0/3979271778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19a40dce10 msgr2=0x7f19a4050c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T19:29:31.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19ab7fe640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19a40dce10 0x7f19a4050c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19ab7fe640 1 -- 172.21.15.172:0/3979271778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a404b9b0 msgr2=0x7f19a4036450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19ab7fe640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a404b9b0 0x7f19a4036450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7fdaed62a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae80c8360 0x7fdae815e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f19ab7fe640 1 -- 172.21.15.172:0/3979271778 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00094d0 con 0x7f19a40db210 2024-03-16T19:29:31.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7fdaed62a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae8158b30 0x7fdae80cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f4faea88640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa80c2230 0x7f4fa813bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f19ab7fe640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a40db210 0x7f19a4050720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f199800c530 tx=0x7f199800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f2e99df4640 1 Processor -- start 2024-03-16T19:29:31.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7f2e99df4640 1 -- start start 2024-03-16T19:29:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7ff60ea08640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6080c58d0 0x7ff608147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7ff60da06640 1 -- 172.21.15.172:0/2166249101 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5f4003ba0 msgr2=0x7ff60815c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7ff60da06640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5f4003ba0 0x7ff60815c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7ff60da06640 1 -- 172.21.15.172:0/2166249101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6080c58d0 msgr2=0x7ff608147c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7ff60da06640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6080c58d0 0x7ff608147c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7ff60da06640 1 -- 172.21.15.172:0/2166249101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6081381c0 con 0x7ff608137830 2024-03-16T19:29:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.839+0000 7ff60e207640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5f4003ba0 0x7ff60815c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4fa7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa80c2230 0x7f4fa813bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4fa7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa80c2230 0x7f4fa813bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46236/0 (socket says 172.21.15.172:46236) 2024-03-16T19:29:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f3a90455640 1 -- 172.21.15.172:0/788529572 shutdown_connections 2024-03-16T19:29:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdaed62a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae80c5d20 con 0x7fdae8158b30 2024-03-16T19:29:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdaed62a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae81f5820 con 0x7fdae80730e0 2024-03-16T19:29:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdaed62a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae815eaf0 con 0x7fdae80c8360 2024-03-16T19:29:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae67fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae80c8360 0x7fdae815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae8158b30 0x7fdae80cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae67fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae80c8360 0x7fdae815e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59804/0 (socket says 172.21.15.172:59804) 2024-03-16T19:29:31.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae8158b30 0x7fdae80cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46232/0 (socket says 172.21.15.172:46232) 2024-03-16T19:29:31.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae67fc640 1 -- 172.21.15.172:0/1609314534 learned_addr learned my addr 172.21.15.172:0/1609314534 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae80730e0 0x7fdae81f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4fac824640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fa806dbd0 0x7f4fa80c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7ff60da06640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff608137830 0x7ff60815a470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800e7f0 tx=0x7ff5f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4fac824640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fa806dbd0 0x7f4fa80c4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42566/0 (socket says 172.21.15.172:42566) 2024-03-16T19:29:31.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4fa7fff640 1 -- 172.21.15.172:0/3150799701 learned_addr learned my addr 172.21.15.172:0/3150799701 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4faea88640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa8156dc0 0x7f4fa8140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fe349623640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe344153dc0 con 0x7fe344070650 2024-03-16T19:29:31.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae6ffd640 1 -- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae80c8360 msgr2=0x7fdae815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa80c5190 con 0x7f4fa80c2230 2024-03-16T19:29:31.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f19a8ff9640 1 -- 172.21.15.172:0/3979271778 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19980052d0 con 0x7f19a40db210 2024-03-16T19:29:31.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f19a8ff9640 1 -- 172.21.15.172:0/3979271778 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1998005430 con 0x7f19a40db210 2024-03-16T19:29:31.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f19a8ff9640 1 -- 172.21.15.172:0/3979271778 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998012870 con 0x7f19a40db210 2024-03-16T19:29:31.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae6ffd640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae80c8360 0x7fdae815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae6ffd640 1 -- 172.21.15.172:0/1609314534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae8158b30 msgr2=0x7fdae80cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f19b131f640 1 -- 172.21.15.172:0/3979271778 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a4181e90 con 0x7f19a40db210 2024-03-16T19:29:31.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae6ffd640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdae8158b30 0x7fdae80cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4fad025640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa8156dc0 0x7f4fa8140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa80c64b0 con 0x7f4fa806dbd0 2024-03-16T19:29:31.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa80c47a0 con 0x7f4fa8156dc0 2024-03-16T19:29:31.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fdae6ffd640 1 -- 172.21.15.172:0/1609314534 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae0009a70 con 0x7fdae80730e0 2024-03-16T19:29:31.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f19b131f640 1 -- 172.21.15.172:0/3979271778 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a4182450 con 0x7f19a40db210 2024-03-16T19:29:31.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fe349623640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3441f8ec0 con 0x7fe344159020 2024-03-16T19:29:31.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fe349623640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe344153240 con 0x7fe344154570 2024-03-16T19:29:31.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fe343fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344154570 0x7fe3441f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fe343fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344154570 0x7fe3441f50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59806/0 (socket says 172.21.15.172:59806) 2024-03-16T19:29:31.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fe343fff640 1 -- 172.21.15.172:0/2593560127 learned_addr learned my addr 172.21.15.172:0/2593560127 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fae63a6b640 1 Processor -- start 2024-03-16T19:29:31.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7fae63a6b640 1 -- start start 2024-03-16T19:29:31.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e99df4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e941457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fe343fff640 1 -- 172.21.15.172:0/2593560127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe344159020 msgr2=0x7fe3440cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e99df4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e941516b0 0x7f2e941f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fe343fff640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe344159020 0x7fe3440cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e99df4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e941f8640 0x7f2e941f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fdae6ffd640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae80730e0 0x7fdae81f52e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdad800e870 tx=0x7fdad800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e99df4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9413fd50 con 0x7f2e941516b0 2024-03-16T19:29:31.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e99df4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e941595e0 con 0x7f2e941f8640 2024-03-16T19:29:31.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e99df4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e941f7c70 con 0x7f2e940c39d0 2024-03-16T19:29:31.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e937fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e941457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e937fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e941457c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59820/0 (socket says 172.21.15.172:59820) 2024-03-16T19:29:31.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e937fe640 1 -- 172.21.15.172:0/2990209872 learned_addr learned my addr 172.21.15.172:0/2990209872 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fe343fff640 1 -- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe344070650 msgr2=0x7fe3441f8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fe343fff640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe344070650 0x7fe3441f8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fdacf7fe640 1 -- 172.21.15.172:0/1609314534 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad800ee60 con 0x7fdae80730e0 2024-03-16T19:29:31.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fdaed62a640 1 -- 172.21.15.172:0/1609314534 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae815c940 con 0x7fdae80730e0 2024-03-16T19:29:31.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f4fad025640 1 -- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fa806dbd0 msgr2=0x7f4fa80c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fdaed62a640 1 -- 172.21.15.172:0/1609314534 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae815cf00 con 0x7fdae80730e0 2024-03-16T19:29:31.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f4fad025640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fa806dbd0 0x7f4fa80c4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.840+0000 7f3a90455640 1 -- 172.21.15.172:0/788529572 wait complete. 2024-03-16T19:29:31.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f19967fc640 1 -- 172.21.15.172:0/3979271778 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f19a40519f0 con 0x7f19a40db210 2024-03-16T19:29:31.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7ff5feffd640 1 -- 172.21.15.172:0/2166249101 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f800a710 con 0x7ff608137830 2024-03-16T19:29:31.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f4fad025640 1 -- 172.21.15.172:0/3150799701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa80c2230 msgr2=0x7f4fa813bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f4fad025640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa80c2230 0x7f4fa813bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f4fad025640 1 -- 172.21.15.172:0/3150799701 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f940094d0 con 0x7f4fa8156dc0 2024-03-16T19:29:31.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e937fe640 1 -- 172.21.15.172:0/2990209872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e941f8640 msgr2=0x7f2e941f6f40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T19:29:31.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e937fe640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e941f8640 0x7f2e941f6f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e937fe640 1 -- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e941516b0 msgr2=0x7f2e941f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e937fe640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e941516b0 0x7f2e941f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f2e937fe640 1 -- 172.21.15.172:0/2990209872 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e88009400 con 0x7f2e940c39d0 2024-03-16T19:29:31.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7ff5feffd640 1 -- 172.21.15.172:0/2166249101 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f800a870 con 0x7ff608137830 2024-03-16T19:29:31.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7ff60f209640 1 -- 172.21.15.172:0/2166249101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6081331c0 con 0x7ff608137830 2024-03-16T19:29:31.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7f3a90455640 1 Processor -- start 2024-03-16T19:29:31.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f3a90455640 1 -- start start 2024-03-16T19:29:31.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fae63a6b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae5c072ba0 0x7fae5c1f82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fae63a6b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fae5c0b5c20 0x7fae5c1f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fae63a6b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae5c0cd0d0 0x7fae5c0cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fae63a6b640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c06d430 con 0x7fae5c0cd0d0 2024-03-16T19:29:31.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fae63a6b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c1f4330 con 0x7fae5c072ba0 2024-03-16T19:29:31.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fae63a6b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c1f8830 con 0x7fae5c0b5c20 2024-03-16T19:29:31.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fae61006640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fae5c0b5c20 0x7fae5c1f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fae61807640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae5c072ba0 0x7fae5c1f82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f2e937fe640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e941457c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c00cbf0 tx=0x7f2e8c00a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7ff60f209640 1 -- 172.21.15.172:0/2166249101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff608133780 con 0x7ff608137830 2024-03-16T19:29:31.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f4fad025640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa8156dc0 0x7f4fa8140040 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0007d40 tx=0x7f4fa000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fae61006640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fae5c0b5c20 0x7fae5c1f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59834/0 (socket says 172.21.15.172:59834) 2024-03-16T19:29:31.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fae61006640 1 -- 172.21.15.172:0/4005144185 learned_addr learned my addr 172.21.15.172:0/4005144185 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f2e7bfff640 1 -- 172.21.15.172:0/2990209872 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00aca0 con 0x7f2e940c39d0 2024-03-16T19:29:31.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f4fa57fa640 1 -- 172.21.15.172:0/3150799701 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0018070 con 0x7f4fa8156dc0 2024-03-16T19:29:31.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f4fa57fa640 1 -- 172.21.15.172:0/3150799701 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa000edd0 con 0x7f4fa8156dc0 2024-03-16T19:29:31.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f2e7bfff640 1 -- 172.21.15.172:0/2990209872 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8c00ae00 con 0x7f2e940c39d0 2024-03-16T19:29:31.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa80c5ef0 con 0x7f4fa8156dc0 2024-03-16T19:29:31.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9413e910 con 0x7f2e940c39d0 2024-03-16T19:29:31.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fdacf7fe640 1 -- 172.21.15.172:0/1609314534 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad800c940 con 0x7fdae80730e0 2024-03-16T19:29:31.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa813f840 con 0x7f4fa8156dc0 2024-03-16T19:29:31.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fdacf7fe640 1 -- 172.21.15.172:0/1609314534 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad800ccd0 con 0x7fdae80730e0 2024-03-16T19:29:31.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fae61807640 1 -- 172.21.15.172:0/4005144185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fae5c0b5c20 msgr2=0x7fae5c1f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fae61807640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fae5c0b5c20 0x7fae5c1f4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fae61807640 1 -- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae5c0cd0d0 msgr2=0x7fae5c0cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fae61807640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fae5c0cd0d0 0x7fae5c0cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7fae61807640 1 -- 172.21.15.172:0/4005144185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae54009a70 con 0x7fae5c072ba0 2024-03-16T19:29:31.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7fae61807640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae5c072ba0 0x7fae5c1f82f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fae5800c440 tx=0x7fae58001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9413f790 con 0x7f2e940c39d0 2024-03-16T19:29:31.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7fdaed62a640 1 -- 172.21.15.172:0/1609314534 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdac8001030 con 0x7fdae80730e0 2024-03-16T19:29:31.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7f3a90455640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a8806b2b0 0x7f3a880cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7f3a90455640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a880caee0 0x7f3a881481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7f3a90455640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a88154c70 0x7f3a88148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7f3a90455640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8815d230 con 0x7f3a8806b2b0 2024-03-16T19:29:31.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7f3a90455640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8815d480 con 0x7f3a88154c70 2024-03-16T19:29:31.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7f3a90455640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a881f3700 con 0x7f3a880caee0 2024-03-16T19:29:31.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.842+0000 7f4fa57fa640 1 -- 172.21.15.172:0/3150799701 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0003bb0 con 0x7f4fa8156dc0 2024-03-16T19:29:31.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7f19a8ff9640 1 -- 172.21.15.172:0/3979271778 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f19980129d0 con 0x7f19a40db210 2024-03-16T19:29:31.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fe343fff640 1 -- 172.21.15.172:0/2593560127 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe338009400 con 0x7fe344154570 2024-03-16T19:29:31.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fe343fff640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344154570 0x7fe3441f50e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe33400e840 tx=0x7fe33400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.841+0000 7fe342ffd640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe344070650 0x7fe3441f8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-16T19:29:31.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f19a8ff9640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f199006b4b0 0x7f199006d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.843+0000 7fe3427fc640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe344159020 0x7fe3440cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-16T19:29:31.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4fa80c1a30 con 0x7f4fa8156dc0 2024-03-16T19:29:31.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f2e7bfff640 1 -- 172.21.15.172:0/2990209872 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c012600 con 0x7f2e940c39d0 2024-03-16T19:29:31.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f19a8ff9640 1 -- 172.21.15.172:0/3979271778 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19980897a0 con 0x7f19a40db210 2024-03-16T19:29:31.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f19a8ff9640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1990073650 0x7f1990075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f19a8ff9640 1 -- 172.21.15.172:0/3979271778 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1990076120 con 0x7f1990073650 2024-03-16T19:29:31.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7ff5feffd640 1 -- 172.21.15.172:0/2166249101 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f800ac00 con 0x7ff608137830 2024-03-16T19:29:31.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7fae527fc640 1 -- 172.21.15.172:0/4005144185 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58005ce0 con 0x7fae5c072ba0 2024-03-16T19:29:31.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f19abfff640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f199006b4b0 0x7f199006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7fae527fc640 1 -- 172.21.15.172:0/4005144185 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae58005e40 con 0x7fae5c072ba0 2024-03-16T19:29:31.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5c1e8a90 con 0x7fae5c072ba0 2024-03-16T19:29:31.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f3a8e9f2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a88154c70 0x7f3a88148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f3a8e9f2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a88154c70 0x7f3a88148700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42602/0 (socket says 172.21.15.172:42602) 2024-03-16T19:29:31.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f3a8e9f2640 1 -- 172.21.15.172:0/3817666976 learned_addr learned my addr 172.21.15.172:0/3817666976 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:31.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7f3a8d9f0640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a880caee0 0x7f3a881481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5c1e84d0 con 0x7fae5c072ba0 2024-03-16T19:29:31.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7fe32b7fe640 1 -- 172.21.15.172:0/2593560127 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33400c940 con 0x7fe344154570 2024-03-16T19:29:31.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7fe32b7fe640 1 -- 172.21.15.172:0/2593560127 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33400caa0 con 0x7fe344154570 2024-03-16T19:29:31.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7fe32b7fe640 1 -- 172.21.15.172:0/2593560127 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33400ce30 con 0x7fe344154570 2024-03-16T19:29:31.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe344153520 con 0x7fe344154570 2024-03-16T19:29:31.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f3a8e9f2640 1 -- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a880caee0 msgr2=0x7f3a881481c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f3a8e9f2640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a880caee0 0x7f3a881481c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f3a8e9f2640 1 -- 172.21.15.172:0/3817666976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a8806b2b0 msgr2=0x7f3a880cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f3a8e9f2640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a8806b2b0 0x7f3a880cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f3a8e9f2640 1 -- 172.21.15.172:0/3817666976 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a78009400 con 0x7f3a88154c70 2024-03-16T19:29:31.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f4fa57fa640 1 -- 172.21.15.172:0/3150799701 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4fa0014420 con 0x7f4fa8156dc0 2024-03-16T19:29:31.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f19abfff640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f199006b4b0 0x7f199006d970 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f19a0036000 tx=0x7f19a0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7fae527fc640 1 -- 172.21.15.172:0/4005144185 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae580072d0 con 0x7fae5c072ba0 2024-03-16T19:29:31.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe344153ae0 con 0x7fe344154570 2024-03-16T19:29:31.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.844+0000 7ff5feffd640 1 -- 172.21.15.172:0/2166249101 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff5f8015480 con 0x7ff608137830 2024-03-16T19:29:31.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae44001030 con 0x7fae5c072ba0 2024-03-16T19:29:31.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7ff5feffd640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff5ec06b480 0x7ff5ec06d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7ff5feffd640 1 -- 172.21.15.172:0/2166249101 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5f808a570 con 0x7ff608137830 2024-03-16T19:29:31.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7ff60e207640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff5ec06b480 0x7ff5ec06d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7ff5fcff9640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ff5f0001650 0x7ff5f0003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7ff60ea08640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ff5f0001650 0x7ff5f0003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7ff5fcff9640 1 -- 172.21.15.172:0/2166249101 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5f0006bb0 con 0x7ff5f0001650 2024-03-16T19:29:31.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7fdacf7fe640 1 -- 172.21.15.172:0/1609314534 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fdad8013470 con 0x7fdae80730e0 2024-03-16T19:29:31.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7fdacf7fe640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdad406b480 0x7fdad406d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7fdacf7fe640 1 -- 172.21.15.172:0/1609314534 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdad8089f20 con 0x7fdae80730e0 2024-03-16T19:29:31.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7fdae67fc640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdad406b480 0x7fdad406d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f4fa57fa640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8c06b480 0x7f4f8c06d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f4fac824640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8c06b480 0x7f4f8c06d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.845+0000 7f19b0b1e640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1990073650 0x7f1990075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7f19a8ff9640 1 -- 172.21.15.172:0/3979271778 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19980543f0 con 0x7f19a40db210 2024-03-16T19:29:31.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7fdacf7fe640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdad4073700 0x7fdad4075b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7fdacf7fe640 1 -- 172.21.15.172:0/1609314534 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdad40761d0 con 0x7fdad4073700 2024-03-16T19:29:31.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7fdacf7fe640 1 -- 172.21.15.172:0/1609314534 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdad808c540 con 0x7fdae80730e0 2024-03-16T19:29:31.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7f3a8e9f2640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a88154c70 0x7f3a88148700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400c400 tx=0x7f3a8400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe344159400 con 0x7fe344154570 2024-03-16T19:29:31.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7f3a7effd640 1 -- 172.21.15.172:0/3817666976 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a840052d0 con 0x7f3a88154c70 2024-03-16T19:29:31.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7f3a7effd640 1 -- 172.21.15.172:0/3817666976 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a84005430 con 0x7f3a88154c70 2024-03-16T19:29:31.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7f3a7effd640 1 -- 172.21.15.172:0/3817666976 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a840037f0 con 0x7f3a88154c70 2024-03-16T19:29:31.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7fdae7fff640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdad4073700 0x7fdad4075b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e74001030 con 0x7f2e940c39d0 2024-03-16T19:29:31.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a881f39c0 con 0x7f3a88154c70 2024-03-16T19:29:31.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a881f3f80 con 0x7f3a88154c70 2024-03-16T19:29:31.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7ff60e207640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff5ec06b480 0x7ff5ec06d940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff600034000 tx=0x7ff600009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7fae527fc640 1 -- 172.21.15.172:0/4005144185 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fae58007430 con 0x7fae5c072ba0 2024-03-16T19:29:31.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7fae527fc640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fae4006b480 0x7fae4006d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7f2e7bfff640 1 -- 172.21.15.172:0/2990209872 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f2e8c007070 con 0x7f2e940c39d0 2024-03-16T19:29:31.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7f2e7bfff640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2e8006b480 0x7f2e8006d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7ff60ea08640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ff5f0001650 0x7ff5f0003b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7fe32b7fe640 1 -- 172.21.15.172:0/2593560127 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fe334013470 con 0x7fe344154570 2024-03-16T19:29:31.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7fdae67fc640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdad406b480 0x7fdad406d940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdae80c16e0 tx=0x7fdadc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.846+0000 7f4fa57fa640 1 -- 172.21.15.172:0/3150799701 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fa0089cb0 con 0x7f4fa8156dc0 2024-03-16T19:29:31.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7f4fa57fa640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4f8c073620 0x7f4f8c075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7f4fa57fa640 1 -- 172.21.15.172:0/3150799701 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f8c0760f0 con 0x7f4f8c073620 2024-03-16T19:29:31.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7f4fa57fa640 1 -- 172.21.15.172:0/3150799701 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fa0054720 con 0x7f4fa8156dc0 2024-03-16T19:29:31.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7f4fac824640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8c06b480 0x7f4f8c06d940 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c006fd0 tx=0x7f4f9c009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7f4fa7fff640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4f8c073620 0x7f4f8c075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fae527fc640 1 -- 172.21.15.172:0/4005144185 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae58089060 con 0x7fae5c072ba0 2024-03-16T19:29:31.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7f2e92ffd640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2e8006b480 0x7f2e8006d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fae527fc640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fae400736a0 0x7fae40075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fae61006640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fae4006b480 0x7fae4006d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fae527fc640 1 -- 172.21.15.172:0/4005144185 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae40076170 con 0x7fae400736a0 2024-03-16T19:29:31.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fae527fc640 1 -- 172.21.15.172:0/4005144185 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae5808b680 con 0x7fae5c072ba0 2024-03-16T19:29:31.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fe32b7fe640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe33006b480 0x7fe33006d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a70001030 con 0x7f3a88154c70 2024-03-16T19:29:31.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7f19b0b1e640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1990073650 0x7f1990075a70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7f2e7bfff640 1 -- 172.21.15.172:0/2990209872 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e8c089ae0 con 0x7f2e940c39d0 2024-03-16T19:29:31.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.847+0000 7fdae7fff640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdad4073700 0x7fdad4075b20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7ff5feffd640 1 -- 172.21.15.172:0/2166249101 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff5f0006bb0 con 0x7ff5f0001650 2024-03-16T19:29:31.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fdacf7fe640 1 -- 172.21.15.172:0/1609314534 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdad40761d0 con 0x7fdad4073700 2024-03-16T19:29:31.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7f2e7bfff640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f2e800736a0 0x7f2e80075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7f2e7bfff640 1 -- 172.21.15.172:0/2990209872 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e80076170 con 0x7f2e800736a0 2024-03-16T19:29:31.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7f2e7bfff640 1 -- 172.21.15.172:0/2990209872 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e8c08e2a0 con 0x7f2e940c39d0 2024-03-16T19:29:31.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7f2e92ffd640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2e8006b480 0x7f2e8006d940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e84000ff0 tx=0x7f2e840079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fae62008640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fae400736a0 0x7fae40075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fae61006640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fae4006b480 0x7fae4006d940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fae5c162210 tx=0x7fae4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7f19a8ff9640 1 -- 172.21.15.172:0/3979271778 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1990076120 con 0x7f1990073650 2024-03-16T19:29:31.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7f2e93fff640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f2e800736a0 0x7f2e80075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7fae62008640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fae400736a0 0x7fae40075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fe32b7fe640 1 -- 172.21.15.172:0/2593560127 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe334089f20 con 0x7fe344154570 2024-03-16T19:29:31.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7f3a7effd640 1 -- 172.21.15.172:0/3817666976 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3a84003950 con 0x7f3a88154c70 2024-03-16T19:29:31.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7f4fa7fff640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4f8c073620 0x7f4f8c075a40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.848+0000 7fe32b7fe640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fe330073590 0x7fe3300759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7fe342ffd640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe33006b480 0x7fe33006d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7fe3427fc640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fe330073590 0x7fe3300759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7fe32b7fe640 1 -- 172.21.15.172:0/2593560127 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe330076060 con 0x7fe330073590 2024-03-16T19:29:31.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7fe32b7fe640 1 -- 172.21.15.172:0/2593560127 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe33408c540 con 0x7fe344154570 2024-03-16T19:29:31.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7fe342ffd640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe33006b480 0x7fe33006d940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe33c007ee0 tx=0x7fe33c0073a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.849+0000 7fe3427fc640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fe330073590 0x7fe3300759b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7f3a7effd640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a6c06b480 0x7f3a6c06d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7f3a8e1f1640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a6c06b480 0x7f3a6c06d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7f3a7effd640 1 -- 172.21.15.172:0/3817666976 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a840891d0 con 0x7f3a88154c70 2024-03-16T19:29:31.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7fae527fc640 1 -- 172.21.15.172:0/4005144185 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fae40076170 con 0x7fae400736a0 2024-03-16T19:29:31.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7fe32b7fe640 1 -- 172.21.15.172:0/2593560127 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe330076060 con 0x7fe330073590 2024-03-16T19:29:31.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7f3a7effd640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3a6c073700 0x7f3a6c075b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:31.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7f3a7effd640 1 -- 172.21.15.172:0/3817666976 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a6c0761d0 con 0x7f3a6c073700 2024-03-16T19:29:31.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7f3a7effd640 1 -- 172.21.15.172:0/3817666976 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a8408b7f0 con 0x7f3a88154c70 2024-03-16T19:29:31.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7f4fa57fa640 1 -- 172.21.15.172:0/3150799701 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4f8c0760f0 con 0x7f4f8c073620 2024-03-16T19:29:31.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7f3a8d9f0640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3a6c073700 0x7f3a6c075b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:31.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.850+0000 7f2e93fff640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f2e800736a0 0x7f2e80075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.851+0000 7f3a8d9f0640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3a6c073700 0x7f3a6c075b20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.852+0000 7f2e7bfff640 1 -- 172.21.15.172:0/2990209872 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2e80076170 con 0x7f2e800736a0 2024-03-16T19:29:31.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.852+0000 7f3a8e1f1640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a6c06b480 0x7f3a6c06d940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3a80000b60 tx=0x7f3a8000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:31.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.852+0000 7f3a7effd640 1 -- 172.21.15.172:0/3817666976 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a6c0761d0 con 0x7f3a6c073700 2024-03-16T19:29:31.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.880+0000 7fdaed62a640 1 -- 172.21.15.172:0/1609314534 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdac8002d30 con 0x7fdad4073700 2024-03-16T19:29:31.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.880+0000 7fdacf7fe640 1 -- 172.21.15.172:0/1609314534 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdac8002d30 con 0x7fdad4073700 2024-03-16T19:29:31.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.881+0000 7ff5fcff9640 1 -- 172.21.15.172:0/2166249101 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff5f0005c40 con 0x7ff5f0001650 2024-03-16T19:29:31.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.881+0000 7fdacd7fa640 1 -- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdad4073700 msgr2=0x7fdad4075b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.881+0000 7fdacd7fa640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdad4073700 0x7fdad4075b20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.881+0000 7f19967fc640 1 -- 172.21.15.172:0/3979271778 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f19a40077b0 con 0x7f1990073650 2024-03-16T19:29:31.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.881+0000 7fdacd7fa640 1 -- 172.21.15.172:0/1609314534 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdad406b480 msgr2=0x7fdad406d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.881+0000 7fdacd7fa640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdad406b480 0x7fdad406d940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdae80c16e0 tx=0x7fdadc009040 comp rx=0 tx=0).stop 2024-03-16T19:29:31.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7fdacd7fa640 1 -- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae80730e0 msgr2=0x7fdae81f52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.881+0000 7ff5feffd640 1 -- 172.21.15.172:0/2166249101 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff5f0005c40 con 0x7ff5f0001650 2024-03-16T19:29:31.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7fdacd7fa640 1 --2- 172.21.15.172:0/1609314534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae80730e0 0x7fdae81f52e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdad800e870 tx=0x7fdad800ed30 comp rx=0 tx=0).stop 2024-03-16T19:29:31.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7fdae7fff640 1 -- 172.21.15.172:0/1609314534 reap_dead start 2024-03-16T19:29:31.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7f19a8ff9640 1 -- 172.21.15.172:0/3979271778 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f19ac04f7c0 con 0x7f1990073650 2024-03-16T19:29:31.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 -- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ff5f0001650 msgr2=0x7ff5f0003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ff5f0001650 0x7ff5f0003b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7fdacd7fa640 1 -- 172.21.15.172:0/1609314534 shutdown_connections 2024-03-16T19:29:31.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7fdacd7fa640 1 -- 172.21.15.172:0/1609314534 >> 172.21.15.172:0/1609314534 conn(0x7fdae80c8b30 msgr2=0x7fdae8155790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 -- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff5ec06b480 msgr2=0x7ff5ec06d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff5ec06b480 0x7ff5ec06d940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff600034000 tx=0x7ff600009030 comp rx=0 tx=0).stop 2024-03-16T19:29:31.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 -- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff608137830 msgr2=0x7ff60815a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 --2- 172.21.15.172:0/2166249101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff608137830 0x7ff60815a470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800e7f0 tx=0x7ff5f800ecb0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff60ea08640 1 -- 172.21.15.172:0/2166249101 reap_dead start 2024-03-16T19:29:31.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4fa8157930 con 0x7f4f8c073620 2024-03-16T19:29:31.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 -- 172.21.15.172:0/2166249101 shutdown_connections 2024-03-16T19:29:31.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 -- 172.21.15.172:0/2166249101 >> 172.21.15.172:0/2166249101 conn(0x7ff6080cc280 msgr2=0x7ff60806c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7f19967fc640 1 -- 172.21.15.172:0/3979271778 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1990073650 msgr2=0x7f1990075a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7f19967fc640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1990073650 0x7f1990075a70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 -- 172.21.15.172:0/2166249101 shutdown_connections 2024-03-16T19:29:31.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7ff5fcff9640 1 -- 172.21.15.172:0/2166249101 wait complete. 2024-03-16T19:29:31.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.882+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae44002d30 con 0x7fae400736a0 2024-03-16T19:29:31.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f4fa57fa640 1 -- 172.21.15.172:0/3150799701 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4f94030b40 con 0x7f4f8c073620 2024-03-16T19:29:31.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f19967fc640 1 -- 172.21.15.172:0/3979271778 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f199006b4b0 msgr2=0x7f199006d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f19967fc640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f199006b4b0 0x7f199006d970 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f19a0036000 tx=0x7f19a0008030 comp rx=0 tx=0).stop 2024-03-16T19:29:31.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f19967fc640 1 -- 172.21.15.172:0/3979271778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a40db210 msgr2=0x7f19a4050720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f19967fc640 1 --2- 172.21.15.172:0/3979271778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19a40db210 0x7f19a4050720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f199800c530 tx=0x7f199800c9f0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f19b0b1e640 1 -- 172.21.15.172:0/3979271778 reap_dead start 2024-03-16T19:29:31.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f19967fc640 1 -- 172.21.15.172:0/3979271778 shutdown_connections 2024-03-16T19:29:31.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f19967fc640 1 -- 172.21.15.172:0/3979271778 >> 172.21.15.172:0/3979271778 conn(0x7f19a404d4b0 msgr2=0x7f19a40cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7fdacd7fa640 1 -- 172.21.15.172:0/1609314534 shutdown_connections 2024-03-16T19:29:31.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7fdacd7fa640 1 -- 172.21.15.172:0/1609314534 wait complete. 2024-03-16T19:29:31.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe3440c9ee0 con 0x7fe330073590 2024-03-16T19:29:31.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4f8c073620 msgr2=0x7f4f8c075a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7fae527fc640 1 -- 172.21.15.172:0/4005144185 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fae44002d30 con 0x7fae400736a0 2024-03-16T19:29:31.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f4faea88640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4f8c073620 0x7f4f8c075a40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f19967fc640 1 -- 172.21.15.172:0/3979271778 shutdown_connections 2024-03-16T19:29:31.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8c06b480 msgr2=0x7f4f8c06d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f4faea88640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8c06b480 0x7f4f8c06d940 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c006fd0 tx=0x7f4f9c009060 comp rx=0 tx=0).stop 2024-03-16T19:29:31.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f19967fc640 1 -- 172.21.15.172:0/3979271778 wait complete. 2024-03-16T19:29:31.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa8156dc0 msgr2=0x7f4fa8140040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f4faea88640 1 --2- 172.21.15.172:0/3150799701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fa8156dc0 0x7f4fa8140040 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0007d40 tx=0x7f4fa000e4b0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.883+0000 7f4fad025640 1 -- 172.21.15.172:0/3150799701 reap_dead start 2024-03-16T19:29:31.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 shutdown_connections 2024-03-16T19:29:31.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 >> 172.21.15.172:0/3150799701 conn(0x7f4fa80cc030 msgr2=0x7f4fa8072a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7fe32b7fe640 1 -- 172.21.15.172:0/2593560127 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe3440c9ee0 con 0x7fe330073590 2024-03-16T19:29:31.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 shutdown_connections 2024-03-16T19:29:31.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7f4faea88640 1 -- 172.21.15.172:0/3150799701 wait complete. 2024-03-16T19:29:31.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fae400736a0 msgr2=0x7fae40075ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7fae63a6b640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fae400736a0 0x7fae40075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fe330073590 msgr2=0x7fe3300759b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fae4006b480 msgr2=0x7fae4006d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7fae63a6b640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fae4006b480 0x7fae4006d940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fae5c162210 tx=0x7fae4c008040 comp rx=0 tx=0).stop 2024-03-16T19:29:31.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae5c072ba0 msgr2=0x7fae5c1f82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fae63a6b640 1 --2- 172.21.15.172:0/4005144185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae5c072ba0 0x7fae5c1f82f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fae5800c440 tx=0x7fae58001e70 comp rx=0 tx=0).stop 2024-03-16T19:29:31.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fae62008640 1 -- 172.21.15.172:0/4005144185 reap_dead start 2024-03-16T19:29:31.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a70002d30 con 0x7f3a6c073700 2024-03-16T19:29:31.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 shutdown_connections 2024-03-16T19:29:31.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 >> 172.21.15.172:0/4005144185 conn(0x7fae5c067040 msgr2=0x7fae5c0c7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7f3a7effd640 1 -- 172.21.15.172:0/3817666976 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3a70002d30 con 0x7f3a6c073700 2024-03-16T19:29:31.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 shutdown_connections 2024-03-16T19:29:31.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fe349623640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fe330073590 0x7fe3300759b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fae63a6b640 1 -- 172.21.15.172:0/4005144185 wait complete. 2024-03-16T19:29:31.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe33006b480 msgr2=0x7fe33006d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fe349623640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe33006b480 0x7fe33006d940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe33c007ee0 tx=0x7fe33c0073a0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.884+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e74002d30 con 0x7f2e800736a0 2024-03-16T19:29:31.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344154570 msgr2=0x7fe3441f50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fe349623640 1 --2- 172.21.15.172:0/2593560127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe344154570 0x7fe3441f50e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe33400e840 tx=0x7fe33400ed00 comp rx=0 tx=0).stop 2024-03-16T19:29:31.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7fe343fff640 1 -- 172.21.15.172:0/2593560127 reap_dead start 2024-03-16T19:29:31.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f2e7bfff640 1 -- 172.21.15.172:0/2990209872 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2e74002d30 con 0x7f2e800736a0 2024-03-16T19:29:31.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.885+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3a6c073700 msgr2=0x7f3a6c075b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a90455640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3a6c073700 0x7f3a6c075b20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a6c06b480 msgr2=0x7f3a6c06d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a90455640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a6c06b480 0x7f3a6c06d940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3a80000b60 tx=0x7f3a8000b040 comp rx=0 tx=0).stop 2024-03-16T19:29:31.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a88154c70 msgr2=0x7f3a88148700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a90455640 1 --2- 172.21.15.172:0/3817666976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a88154c70 0x7f3a88148700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400c400 tx=0x7f3a8400c8c0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a8e9f2640 1 -- 172.21.15.172:0/3817666976 reap_dead start 2024-03-16T19:29:31.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 shutdown_connections 2024-03-16T19:29:31.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 >> 172.21.15.172:0/2593560127 conn(0x7fe3440ca0c0 msgr2=0x7fe344157270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 shutdown_connections 2024-03-16T19:29:31.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 >> 172.21.15.172:0/3817666976 conn(0x7f3a88067040 msgr2=0x7f3a88150a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 shutdown_connections 2024-03-16T19:29:31.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7fe349623640 1 -- 172.21.15.172:0/2593560127 wait complete. 2024-03-16T19:29:31.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 shutdown_connections 2024-03-16T19:29:31.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.886+0000 7f3a90455640 1 -- 172.21.15.172:0/3817666976 wait complete. 2024-03-16T19:29:31.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.887+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f2e800736a0 msgr2=0x7f2e80075ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.887+0000 7f2e99df4640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f2e800736a0 0x7f2e80075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.887+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2e8006b480 msgr2=0x7f2e8006d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.887+0000 7f2e99df4640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2e8006b480 0x7f2e8006d940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e84000ff0 tx=0x7f2e840079b0 comp rx=0 tx=0).stop 2024-03-16T19:29:31.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.887+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 msgr2=0x7f2e941457c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:31.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.887+0000 7f2e99df4640 1 --2- 172.21.15.172:0/2990209872 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e940c39d0 0x7f2e941457c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c00cbf0 tx=0x7f2e8c00a060 comp rx=0 tx=0).stop 2024-03-16T19:29:31.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.887+0000 7f2e93fff640 1 -- 172.21.15.172:0/2990209872 reap_dead start 2024-03-16T19:29:31.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.887+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 shutdown_connections 2024-03-16T19:29:31.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.887+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 >> 172.21.15.172:0/2990209872 conn(0x7f2e940cc4d0 msgr2=0x7f2e940725f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:31.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.888+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 shutdown_connections 2024-03-16T19:29:31.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:31.892+0000 7f2e99df4640 1 -- 172.21.15.172:0/2990209872 wait complete. 2024-03-16T19:29:31.931 INFO:teuthology.orchestra.run.smithi172.stdout:55834574855 2024-03-16T19:29:31.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-16T19:29:31.932 INFO:teuthology.orchestra.run.smithi172.stdout:55834574855 2024-03-16T19:29:31.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-16T19:29:31.933 INFO:teuthology.orchestra.run.smithi172.stdout:55834574854 2024-03-16T19:29:31.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-16T19:29:31.934 INFO:teuthology.orchestra.run.smithi172.stdout:55834574854 2024-03-16T19:29:31.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-16T19:29:31.935 INFO:teuthology.orchestra.run.smithi172.stdout:55834574853 2024-03-16T19:29:31.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-16T19:29:31.936 INFO:teuthology.orchestra.run.smithi172.stdout:55834574855 2024-03-16T19:29:31.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-16T19:29:31.937 INFO:teuthology.orchestra.run.smithi172.stdout:55834574854 2024-03-16T19:29:31.938 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-16T19:29:31.938 INFO:teuthology.orchestra.run.smithi172.stdout:55834574856 2024-03-16T19:29:31.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-16T19:29:32.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.159+0000 7f74e6e61640 1 Processor -- start 2024-03-16T19:29:32.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.159+0000 7f74e6e61640 1 -- start start 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e6e61640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e00b0ce0 0x7f74e0152960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e6e61640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f74e015bad0 0x7f74e00c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e6e61640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e00c3640 con 0x7f74e0152580 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e6e61640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e015bf40 con 0x7f74e015bad0 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e6e61640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00c4010 con 0x7f74e00b0ce0 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e565e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e00b0ce0 0x7f74e0152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e5e5f640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f74e0152580 0x7f74e006fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56174/0 (socket says 172.21.15.172:56174) 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e5e5f640 1 -- 172.21.15.172:0/704472192 learned_addr learned my addr 172.21.15.172:0/704472192 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e6660640 1 --2- 172.21.15.172:0/704472192 >> v2:172.21.15.174:3300/0 conn(0x7f74e015bad0 0x7f74e00c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.160+0000 7f74e4e5d640 1 -- 172.21.15.172:0/704472192 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236833756 0 0) 0x7f74e00c3640 con 0x7f74e0152580 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.161+0000 7f74e4e5d640 1 -- 172.21.15.172:0/704472192 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74cc003520 con 0x7f74e0152580 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.161+0000 7f74e565e640 1 -- 172.21.15.172:0/704472192 >> v2:172.21.15.174:3300/0 conn(0x7f74e015bad0 msgr2=0x7f74e00c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.161+0000 7f74e565e640 1 --2- 172.21.15.172:0/704472192 >> v2:172.21.15.174:3300/0 conn(0x7f74e015bad0 0x7f74e00c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.161+0000 7f74e565e640 1 -- 172.21.15.172:0/704472192 >> v1:172.21.15.172:6789/0 conn(0x7f74e0152580 legacy=0x7f74e006fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.161+0000 7f74e565e640 1 -- 172.21.15.172:0/704472192 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00c3310 con 0x7f74e00b0ce0 2024-03-16T19:29:32.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.162+0000 7f74e565e640 1 --2- 172.21.15.172:0/704472192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e00b0ce0 0x7f74e0152960 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74d00097a0 tx=0x7f74d002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaf990e6214d57a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.163+0000 7f74e4e5d640 1 -- 172.21.15.172:0/704472192 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0031830 con 0x7f74e00b0ce0 2024-03-16T19:29:32.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.163+0000 7f74e4e5d640 1 -- 172.21.15.172:0/704472192 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d0031990 con 0x7f74e00b0ce0 2024-03-16T19:29:32.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.163+0000 7f74e4e5d640 1 -- 172.21.15.172:0/704472192 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0031d60 con 0x7f74e00b0ce0 2024-03-16T19:29:32.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.163+0000 7f0dc236c640 1 Processor -- start 2024-03-16T19:29:32.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.163+0000 7f0dc236c640 1 -- start start 2024-03-16T19:29:32.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f9990dd4640 1 Processor -- start 2024-03-16T19:29:32.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f74e6e61640 1 -- 172.21.15.172:0/704472192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e00b0ce0 msgr2=0x7f74e0152960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f74e6e61640 1 --2- 172.21.15.172:0/704472192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e00b0ce0 0x7f74e0152960 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74d00097a0 tx=0x7f74d002b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f74e6e61640 1 -- 172.21.15.172:0/704472192 shutdown_connections 2024-03-16T19:29:32.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f74e6e61640 1 --2- 172.21.15.172:0/704472192 >> v2:172.21.15.174:3300/0 conn(0x7f74e015bad0 0x7f74e00c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f74e6e61640 1 --2- 172.21.15.172:0/704472192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e00b0ce0 0x7f74e0152960 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f74e6e61640 1 -- 172.21.15.172:0/704472192 >> 172.21.15.172:0/704472192 conn(0x7f74e0066c20 msgr2=0x7f74e0067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f0dc236c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db40d92e0 0x7f0db404b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f0dc236c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0db4008180 0x7f0db4046a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f0dc236c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db40062f0 con 0x7f0db40459c0 2024-03-16T19:29:32.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f0dc236c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db4007ca0 con 0x7f0db4008180 2024-03-16T19:29:32.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f0dc236c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40e2e90 con 0x7f0db40d92e0 2024-03-16T19:29:32.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f74e6e61640 1 -- 172.21.15.172:0/704472192 shutdown_connections 2024-03-16T19:29:32.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f74e6e61640 1 -- 172.21.15.172:0/704472192 wait complete. 2024-03-16T19:29:32.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.164+0000 7f9990dd4640 1 -- start start 2024-03-16T19:29:32.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f29811d9640 1 Processor -- start 2024-03-16T19:29:32.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f29811d9640 1 -- start start 2024-03-16T19:29:32.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f0dc0909640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0db4008180 0x7f0db4046a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f0dc0909640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0db4008180 0x7f0db4046a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42612/0 (socket says 172.21.15.172:42612) 2024-03-16T19:29:32.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f0dc0909640 1 -- 172.21.15.172:0/628130655 learned_addr learned my addr 172.21.15.172:0/628130655 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f0dbbfff640 1 --2- 172.21.15.172:0/628130655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db40d92e0 0x7f0db404b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f9990dd4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99840db200 0x7f9984006aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f9990dd4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f99840dc6f0 0x7f99840dcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f1000a2f640 1 Processor -- start 2024-03-16T19:29:32.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f9990dd4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99840080d0 con 0x7f99840d51f0 2024-03-16T19:29:32.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f9990dd4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9984047c40 con 0x7f99840dc6f0 2024-03-16T19:29:32.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f998a59c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f99840d51f0 0x7f998404b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56184/0 (socket says 172.21.15.172:56184) 2024-03-16T19:29:32.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f9990dd4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9984006fe0 con 0x7f99840db200 2024-03-16T19:29:32.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f998a59c640 1 -- 172.21.15.172:0/1946698590 learned_addr learned my addr 172.21.15.172:0/1946698590 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:32.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f0dbbfff640 1 -- 172.21.15.172:0/628130655 >> v2:172.21.15.174:3300/0 conn(0x7f0db4008180 msgr2=0x7f0db4046a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f0dbbfff640 1 --2- 172.21.15.172:0/628130655 >> v2:172.21.15.174:3300/0 conn(0x7f0db4008180 0x7f0db4046a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f0dbbfff640 1 -- 172.21.15.172:0/628130655 >> v1:172.21.15.174:6789/0 conn(0x7f0db40459c0 legacy=0x7f0db4046360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f0dbbfff640 1 -- 172.21.15.172:0/628130655 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db40e3150 con 0x7f0db40d92e0 2024-03-16T19:29:32.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f0dc0909640 1 --2- 172.21.15.172:0/628130655 >> v2:172.21.15.174:3300/0 conn(0x7f0db4008180 0x7f0db4046a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:32.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f29811d9640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f29740d6bd0 0x7f2974036fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92793330 0 0) 0x7f99840080d0 con 0x7f99840d51f0 2024-03-16T19:29:32.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9974003440 con 0x7f99840d51f0 2024-03-16T19:29:32.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f29811d9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2974006a90 0x7f29740343e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f297bfff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f29740d6bd0 0x7f2974036fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f297bfff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f29740d6bd0 0x7f2974036fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46292/0 (socket says 172.21.15.172:46292) 2024-03-16T19:29:32.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f4fa67d0640 1 Processor -- start 2024-03-16T19:29:32.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.165+0000 7f1000a2f640 1 -- start start 2024-03-16T19:29:32.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f1000a2f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0725e0 0x7f0ffc071950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f1000a2f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ffc0c6a90 con 0x7f0ffc0c5d70 2024-03-16T19:29:32.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f1000a2f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ffc15c520 con 0x7f0ffc071090 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f1000a2f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc162e50 con 0x7f0ffc0725e0 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f0ffb7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0ffc0c5d70 0x7f0ffc0c6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56186/0 (socket says 172.21.15.172:56186) 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f0ffb7fe640 1 -- 172.21.15.172:0/905748321 learned_addr learned my addr 172.21.15.172:0/905748321 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4041963179 0 0) 0x7f9974003440 con 0x7f99840d51f0 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99840db200 msgr2=0x7f9984006aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f998959a640 1 --2- 172.21.15.172:0/1946698590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99840db200 0x7f9984006aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 >> v2:172.21.15.174:3300/0 conn(0x7f99840dc6f0 msgr2=0x7f99840dcb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f998959a640 1 --2- 172.21.15.172:0/1946698590 >> v2:172.21.15.174:3300/0 conn(0x7f99840dc6f0 0x7f99840dcb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99840dd060 con 0x7f99840d51f0 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f997c001670 con 0x7f99840d51f0 2024-03-16T19:29:32.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 >> v1:172.21.15.172:6789/0 conn(0x7f99840d51f0 legacy=0x7f998404b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f998959a640 1 --2- 172.21.15.172:0/1946698590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9974003ae0 0x7f9974003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f0dbbfff640 1 --2- 172.21.15.172:0/628130655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db40d92e0 0x7f0db404b3a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c884f4657f81f403 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f6a9651e640 1 Processor -- start 2024-03-16T19:29:32.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f6a9651e640 1 -- start start 2024-03-16T19:29:32.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f0ffaffd640 1 --2- 172.21.15.172:0/905748321 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0725e0 0x7f0ffc071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f998959a640 1 --2- 172.21.15.172:0/1946698590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974008970 0x7f997400ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f998959a640 1 --2- 172.21.15.172:0/1946698590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997400b5b0 0x7f997400da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9984047c40 con 0x7f9974003ae0 2024-03-16T19:29:32.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9984006fe0 con 0x7f997400b5b0 2024-03-16T19:29:32.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997400e3a0 con 0x7f9974008970 2024-03-16T19:29:32.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f998a59c640 1 --2- 172.21.15.172:0/1946698590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9974003ae0 0x7f9974003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f0dbaffd640 1 -- 172.21.15.172:0/628130655 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db002e070 con 0x7f0db40d92e0 2024-03-16T19:29:32.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f0dbaffd640 1 -- 172.21.15.172:0/628130655 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db00315a0 con 0x7f0db40d92e0 2024-03-16T19:29:32.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f4fa67d0640 1 -- start start 2024-03-16T19:29:32.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f74e6e61640 1 Processor -- start 2024-03-16T19:29:32.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.166+0000 7f74e6e61640 1 -- start start 2024-03-16T19:29:32.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f74e6e61640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74e00b0ce0 0x7f74e01ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f297bfff640 1 -- 172.21.15.172:0/4038550328 learned_addr learned my addr 172.21.15.172:0/4038550328 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f29811d9640 1 -- 172.21.15.172:0/4038550328 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29740d6fb0 con 0x7f2974006e70 2024-03-16T19:29:32.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f29811d9640 1 -- 172.21.15.172:0/4038550328 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29740dd980 con 0x7f29740d6bd0 2024-03-16T19:29:32.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f29811d9640 1 -- 172.21.15.172:0/4038550328 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29740dcb30 con 0x7f2974006a90 2024-03-16T19:29:32.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.167+0000 7f4fa67d0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4f98007d60 0x7f4f9804bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e5e5f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74e00b0ce0 0x7f74e01ef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e6e61640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e0152580 0x7f74e0158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e6e61640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e015bad0 0x7f74e01ec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e6e61640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e014db60 con 0x7f74e00b0ce0 2024-03-16T19:29:32.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e6e61640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e01efe30 con 0x7f74e015bad0 2024-03-16T19:29:32.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e6e61640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e01ebb10 con 0x7f74e0152580 2024-03-16T19:29:32.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e565e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e0152580 0x7f74e0158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f297bfff640 1 -- 172.21.15.172:0/4038550328 >> v2:172.21.15.174:3300/0 conn(0x7f2974006a90 msgr2=0x7f29740343e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:29:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f297bfff640 1 --2- 172.21.15.172:0/4038550328 >> v2:172.21.15.174:3300/0 conn(0x7f2974006a90 0x7f29740343e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0dbaffd640 1 -- 172.21.15.172:0/628130655 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db002e070 con 0x7f0db40d92e0 2024-03-16T19:29:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f297bfff640 1 -- 172.21.15.172:0/4038550328 >> v1:172.21.15.174:6789/0 conn(0x7f2974006e70 legacy=0x7f2974008220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e6660640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e015bad0 0x7f74e01ec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e6660640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e015bad0 0x7f74e01ec2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42634/0 (socket says 172.21.15.172:42634) 2024-03-16T19:29:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f74e6660640 1 -- 172.21.15.172:0/2367365928 learned_addr learned my addr 172.21.15.172:0/2367365928 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0dc236c640 1 -- 172.21.15.172:0/628130655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db40d92e0 msgr2=0x7f0db404b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0dc236c640 1 --2- 172.21.15.172:0/628130655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db40d92e0 0x7f0db404b3a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b840 comp rx=0 tx=0).stop 2024-03-16T19:29:32.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f297bfff640 1 -- 172.21.15.172:0/4038550328 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29740dcd90 con 0x7f29740d6bd0 2024-03-16T19:29:32.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f4f9ffff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f4f980dd6b0 0x7f4f98047be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:53332/0 (socket says 172.21.15.172:53332) 2024-03-16T19:29:32.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f4f9f7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4f980462f0 0x7f4f980470a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56198/0 (socket says 172.21.15.172:56198) 2024-03-16T19:29:32.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f6a9651e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6a901549d0 0x7f6a9015e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0dc236c640 1 -- 172.21.15.172:0/628130655 shutdown_connections 2024-03-16T19:29:32.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0dc236c640 1 --2- 172.21.15.172:0/628130655 >> v2:172.21.15.174:3300/0 conn(0x7f0db4008180 0x7f0db4046a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0dc236c640 1 --2- 172.21.15.172:0/628130655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db40d92e0 0x7f0db404b3a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0dc236c640 1 -- 172.21.15.172:0/628130655 >> 172.21.15.172:0/628130655 conn(0x7f0db404d4a0 msgr2=0x7f0db404d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f6a9651e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd5f0 0x7f6a9015be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f998a59c640 1 -- 172.21.15.172:0/1946698590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974008970 msgr2=0x7f997400ae00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:29:32.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f998a59c640 1 --2- 172.21.15.172:0/1946698590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974008970 0x7f997400ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f998a59c640 1 -- 172.21.15.172:0/1946698590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997400b5b0 msgr2=0x7f997400da40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:29:32.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f998a59c640 1 --2- 172.21.15.172:0/1946698590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997400b5b0 0x7f997400da40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f998a59c640 1 -- 172.21.15.172:0/1946698590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9974003440 con 0x7f9974003ae0 2024-03-16T19:29:32.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f4f9ffff640 1 -- 172.21.15.172:0/2514849349 learned_addr learned my addr 172.21.15.172:0/2514849349 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:32.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f4fa4d6d640 1 --2- 172.21.15.172:0/2514849349 >> v2:172.21.15.174:3300/0 conn(0x7f4f98007d60 0x7f4f9804bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0ffaffd640 1 -- 172.21.15.172:0/905748321 >> v1:172.21.15.174:6789/0 conn(0x7f0ffc071090 legacy=0x7f0ffc15ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f6a9651e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900cb910 con 0x7f6a900cd1d0 2024-03-16T19:29:32.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f6a9651e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a900cda80 con 0x7f6a901549d0 2024-03-16T19:29:32.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f6a9651e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a900caa90 con 0x7f6a900cd5f0 2024-03-16T19:29:32.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f6a8ffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6a901549d0 0x7f6a9015e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f4fa67d0640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f980078d0 con 0x7f4f980462f0 2024-03-16T19:29:32.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2514849349 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f98006fe0 con 0x7f4f980dd6b0 2024-03-16T19:29:32.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2514849349 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f980dda90 con 0x7f4f98007d60 2024-03-16T19:29:32.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f0dc236c640 1 -- 172.21.15.172:0/628130655 shutdown_connections 2024-03-16T19:29:32.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f297bfff640 1 --2- 172.21.15.172:0/4038550328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29740d6bd0 0x7f2974036fa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f297c075730 tx=0x7f297c0a7ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68167ecc9964ffa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74e6660640 1 -- 172.21.15.172:0/2367365928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e0152580 msgr2=0x7f74e0158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74e6660640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e0152580 0x7f74e0158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74e6660640 1 -- 172.21.15.172:0/2367365928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74e00b0ce0 msgr2=0x7f74e01ef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74e6660640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74e00b0ce0 0x7f74e01ef8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74e6660640 1 -- 172.21.15.172:0/2367365928 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d0009400 con 0x7f74e015bad0 2024-03-16T19:29:32.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74e565e640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e0152580 0x7f74e0158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:32.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f0dc236c640 1 -- 172.21.15.172:0/628130655 wait complete. 2024-03-16T19:29:32.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f6a8ffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6a901549d0 0x7f6a9015e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46320/0 (socket says 172.21.15.172:46320) 2024-03-16T19:29:32.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f6a8ffff640 1 -- 172.21.15.172:0/871240573 learned_addr learned my addr 172.21.15.172:0/871240573 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f6a94abb640 1 --2- 172.21.15.172:0/871240573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd5f0 0x7f6a9015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f6a8ffff640 1 -- 172.21.15.172:0/871240573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd5f0 msgr2=0x7f6a9015be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f998a59c640 1 --2- 172.21.15.172:0/1946698590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9974003ae0 0x7f9974003ec0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f997c004b70 tx=0x7f99740036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8237b837a761f7a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0ffaffd640 1 -- 172.21.15.172:0/905748321 >> v1:172.21.15.172:6789/0 conn(0x7f0ffc0c5d70 legacy=0x7f0ffc0c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.168+0000 7f0ffaffd640 1 -- 172.21.15.172:0/905748321 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc1621d0 con 0x7f0ffc0725e0 2024-03-16T19:29:32.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f6a8ffff640 1 --2- 172.21.15.172:0/871240573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd5f0 0x7f6a9015be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f6a8ffff640 1 -- 172.21.15.172:0/871240573 >> v1:172.21.15.174:6789/0 conn(0x7f6a900cd1d0 legacy=0x7f6a900cffc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T19:29:32.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f0ffaffd640 1 --2- 172.21.15.172:0/905748321 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0725e0 0x7f0ffc071950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00097d0 tx=0x7f0ff002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8d7c2c5570c2862 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f6a8ffff640 1 -- 172.21.15.172:0/871240573 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a900cad00 con 0x7f6a901549d0 2024-03-16T19:29:32.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c032040 con 0x7f9974003ae0 2024-03-16T19:29:32.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f297affd640 1 -- 172.21.15.172:0/4038550328 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c0a9930 con 0x7f29740d6bd0 2024-03-16T19:29:32.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f0ffa7fc640 1 -- 172.21.15.172:0/905748321 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff002f050 con 0x7f0ffc0725e0 2024-03-16T19:29:32.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f0ffa7fc640 1 -- 172.21.15.172:0/905748321 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff0031700 con 0x7f0ffc0725e0 2024-03-16T19:29:32.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f0ffa7fc640 1 -- 172.21.15.172:0/905748321 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0031ad0 con 0x7f0ffc0725e0 2024-03-16T19:29:32.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f4fa4d6d640 1 -- 172.21.15.172:0/2514849349 >> v1:172.21.15.174:6789/0 conn(0x7f4f980dd6b0 legacy=0x7f4f98047be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74e6660640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e015bad0 0x7f74e01ec2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f74d8007e60 tx=0x7f74d800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74d67fc640 1 -- 172.21.15.172:0/2367365928 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8014420 con 0x7f74e015bad0 2024-03-16T19:29:32.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74d67fc640 1 -- 172.21.15.172:0/2367365928 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d80145a0 con 0x7f74e015bad0 2024-03-16T19:29:32.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74e6e61640 1 -- 172.21.15.172:0/2367365928 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e00c72b0 con 0x7f74e015bad0 2024-03-16T19:29:32.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f74e6e61640 1 -- 172.21.15.172:0/2367365928 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e00c5b20 con 0x7f74e015bad0 2024-03-16T19:29:32.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f74d67fc640 1 -- 172.21.15.172:0/2367365928 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8014950 con 0x7f74e015bad0 2024-03-16T19:29:32.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f4fa4d6d640 1 -- 172.21.15.172:0/2514849349 >> v1:172.21.15.172:6789/0 conn(0x7f4f980462f0 legacy=0x7f4f980470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.169+0000 7f4fa4d6d640 1 -- 172.21.15.172:0/2514849349 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f980e3ea0 con 0x7f4f98007d60 2024-03-16T19:29:32.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f4fa4d6d640 1 --2- 172.21.15.172:0/2514849349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f98007d60 0x7f4f9804bc20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4fa002d4f0 tx=0x7f4fa00a7b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=623f2adf39342e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c005ad0 con 0x7f9974003ae0 2024-03-16T19:29:32.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f998959a640 1 -- 172.21.15.172:0/1946698590 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c005e60 con 0x7f9974003ae0 2024-03-16T19:29:32.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f4f9effd640 1 -- 172.21.15.172:0/2514849349 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa00a99d0 con 0x7f4f98007d60 2024-03-16T19:29:32.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f4f9effd640 1 -- 172.21.15.172:0/2514849349 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa00a9b30 con 0x7f4f98007d60 2024-03-16T19:29:32.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f4f9effd640 1 -- 172.21.15.172:0/2514849349 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa00a8430 con 0x7f4f98007d60 2024-03-16T19:29:32.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f0dc236c640 1 Processor -- start 2024-03-16T19:29:32.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f297affd640 1 -- 172.21.15.172:0/4038550328 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297c0a9ad0 con 0x7f29740d6bd0 2024-03-16T19:29:32.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.170+0000 7f297affd640 1 -- 172.21.15.172:0/4038550328 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c0b3480 con 0x7f29740d6bd0 2024-03-16T19:29:32.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f29811d9640 1 -- 172.21.15.172:0/4038550328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29740d6bd0 msgr2=0x7f2974036fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f29811d9640 1 --2- 172.21.15.172:0/4038550328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29740d6bd0 0x7f2974036fa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f297c075730 tx=0x7f297c0a7ab0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f9990dd4640 1 -- 172.21.15.172:0/1946698590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9974003ae0 msgr2=0x7f9974003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f9990dd4640 1 --2- 172.21.15.172:0/1946698590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9974003ae0 0x7f9974003ec0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f997c004b70 tx=0x7f99740036f0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f998ad9d640 1 -- 172.21.15.172:0/1946698590 reap_dead start 2024-03-16T19:29:32.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f74e6e61640 1 -- 172.21.15.172:0/2367365928 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c8005290 con 0x7f74e015bad0 2024-03-16T19:29:32.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f29811d9640 1 -- 172.21.15.172:0/4038550328 shutdown_connections 2024-03-16T19:29:32.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f29811d9640 1 --2- 172.21.15.172:0/4038550328 >> v2:172.21.15.174:3300/0 conn(0x7f2974006a90 0x7f29740343e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f29811d9640 1 --2- 172.21.15.172:0/4038550328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29740d6bd0 0x7f2974036fa0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f29811d9640 1 -- 172.21.15.172:0/4038550328 >> 172.21.15.172:0/4038550328 conn(0x7f297404dd30 msgr2=0x7f297404e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f37f102c640 1 Processor -- start 2024-03-16T19:29:32.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f29811d9640 1 -- 172.21.15.172:0/4038550328 shutdown_connections 2024-03-16T19:29:32.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f9990dd4640 1 -- 172.21.15.172:0/1946698590 shutdown_connections 2024-03-16T19:29:32.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f9990dd4640 1 -- 172.21.15.172:0/1946698590 >> 172.21.15.172:0/1946698590 conn(0x7f998404d4a0 msgr2=0x7f998404d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f37f102c640 1 -- start start 2024-03-16T19:29:32.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f6a8ffff640 1 --2- 172.21.15.172:0/871240573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 0x7f6a9015e7d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009870 tx=0x7f6a8002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc878b807bac504 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f1000a2f640 1 -- 172.21.15.172:0/905748321 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0725e0 msgr2=0x7f0ffc071950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f1000a2f640 1 --2- 172.21.15.172:0/905748321 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0725e0 0x7f0ffc071950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00097d0 tx=0x7f0ff002b7a0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f1000a2f640 1 -- 172.21.15.172:0/905748321 shutdown_connections 2024-03-16T19:29:32.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f1000a2f640 1 --2- 172.21.15.172:0/905748321 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0725e0 0x7f0ffc071950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f1000a2f640 1 -- 172.21.15.172:0/905748321 >> 172.21.15.172:0/905748321 conn(0x7f0ffc06aed0 msgr2=0x7f0ffc06b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2514849349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f98007d60 msgr2=0x7f4f9804bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f4fa67d0640 1 --2- 172.21.15.172:0/2514849349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f98007d60 0x7f4f9804bc20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4fa002d4f0 tx=0x7f4fa00a7b50 comp rx=0 tx=0).stop 2024-03-16T19:29:32.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f0dc236c640 1 -- start start 2024-03-16T19:29:32.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2514849349 shutdown_connections 2024-03-16T19:29:32.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f4fa67d0640 1 --2- 172.21.15.172:0/2514849349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f98007d60 0x7f4f9804bc20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2514849349 >> 172.21.15.172:0/2514849349 conn(0x7f4f9804dd20 msgr2=0x7f4f9804e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f1000a2f640 1 -- 172.21.15.172:0/905748321 shutdown_connections 2024-03-16T19:29:32.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f9990dd4640 1 -- 172.21.15.172:0/1946698590 shutdown_connections 2024-03-16T19:29:32.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.171+0000 7f29811d9640 1 -- 172.21.15.172:0/4038550328 wait complete. 2024-03-16T19:29:32.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f1000a2f640 1 -- 172.21.15.172:0/905748321 wait complete. 2024-03-16T19:29:32.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f6a8effd640 1 -- 172.21.15.172:0/871240573 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80030840 con 0x7f6a901549d0 2024-03-16T19:29:32.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f9990dd4640 1 -- 172.21.15.172:0/1946698590 wait complete. 2024-03-16T19:29:32.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f0dc236c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db4008180 0x7f0db40487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f6a8effd640 1 -- 172.21.15.172:0/871240573 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a800309a0 con 0x7f6a901549d0 2024-03-16T19:29:32.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f0dc236c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db40d92e0 0x7f0db4179a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f6a8effd640 1 -- 172.21.15.172:0/871240573 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80030d30 con 0x7f6a901549d0 2024-03-16T19:29:32.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f37f102c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f37e40dd3b0 0x7f37e4046450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f0dc236c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db40cf0e0 0x7f0db404a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f0dc236c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40df600 con 0x7f0db40d92e0 2024-03-16T19:29:32.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f0dc236c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40da890 con 0x7f0db40cf0e0 2024-03-16T19:29:32.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f0dc236c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40d9ff0 con 0x7f0db4008180 2024-03-16T19:29:32.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f37f102c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e404ba50 0x7f37e404be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f0dbb7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db40d92e0 0x7f0db4179a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbb7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db40d92e0 0x7f0db4179a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46322/0 (socket says 172.21.15.172:46322) 2024-03-16T19:29:32.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbb7fe640 1 -- 172.21.15.172:0/3045604960 learned_addr learned my addr 172.21.15.172:0/3045604960 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.172+0000 7f37f102c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e4045440 con 0x7f37e40e3080 2024-03-16T19:29:32.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37f102c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e4045fc0 con 0x7f37e40dd3b0 2024-03-16T19:29:32.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37f102c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40e3c20 con 0x7f37e404ba50 2024-03-16T19:29:32.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbbfff640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db4008180 0x7f0db40487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37eb59e640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f37e40e3080 0x7f37e40e3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56214/0 (socket says 172.21.15.172:56214) 2024-03-16T19:29:32.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37eb59e640 1 -- 172.21.15.172:0/3033711502 learned_addr learned my addr 172.21.15.172:0/3033711502 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:32.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37ead9d640 1 --2- 172.21.15.172:0/3033711502 >> v2:172.21.15.174:3300/0 conn(0x7f37e40dd3b0 0x7f37e4046450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37ea59c640 1 --2- 172.21.15.172:0/3033711502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e404ba50 0x7f37e404be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37ead9d640 1 -- 172.21.15.172:0/3033711502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e404ba50 msgr2=0x7f37e404be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37ead9d640 1 --2- 172.21.15.172:0/3033711502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e404ba50 0x7f37e404be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbb7fe640 1 -- 172.21.15.172:0/3045604960 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db4008180 msgr2=0x7f0db40487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f6a9651e640 1 -- 172.21.15.172:0/871240573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 msgr2=0x7f6a9015e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbb7fe640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db4008180 0x7f0db40487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbb7fe640 1 -- 172.21.15.172:0/3045604960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db40cf0e0 msgr2=0x7f0db404a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbb7fe640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db40cf0e0 0x7f0db404a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f6a9651e640 1 --2- 172.21.15.172:0/871240573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 0x7f6a9015e7d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009870 tx=0x7f6a8002b980 comp rx=0 tx=0).stop 2024-03-16T19:29:32.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbb7fe640 1 -- 172.21.15.172:0/3045604960 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc073d10 con 0x7f0db40d92e0 2024-03-16T19:29:32.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbbfff640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db4008180 0x7f0db40487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:32.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37e9d9b640 1 -- 172.21.15.172:0/3033711502 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115825699 0 0) 0x7f37e4045440 con 0x7f37e40e3080 2024-03-16T19:29:32.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37ead9d640 1 -- 172.21.15.172:0/3033711502 >> v1:172.21.15.172:6789/0 conn(0x7f37e40e3080 legacy=0x7f37e40e3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f0dbb7fe640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db40d92e0 0x7f0db4179a40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c410 tx=0x7f0da800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f29811d9640 1 Processor -- start 2024-03-16T19:29:32.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f37ead9d640 1 -- 172.21.15.172:0/3033711502 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40dd790 con 0x7f37e40dd3b0 2024-03-16T19:29:32.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f6a9651e640 1 -- 172.21.15.172:0/871240573 shutdown_connections 2024-03-16T19:29:32.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f6a9651e640 1 --2- 172.21.15.172:0/871240573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd5f0 0x7f6a9015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f6a9651e640 1 --2- 172.21.15.172:0/871240573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 0x7f6a9015e7d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f1000a2f640 1 Processor -- start 2024-03-16T19:29:32.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f6a9651e640 1 -- 172.21.15.172:0/871240573 >> 172.21.15.172:0/871240573 conn(0x7f6a9006d640 msgr2=0x7f6a9006da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f0db8ff9640 1 -- 172.21.15.172:0/3045604960 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8003610 con 0x7f0db40d92e0 2024-03-16T19:29:32.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.173+0000 7f74d67fc640 1 -- 172.21.15.172:0/2367365928 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f74d8014ab0 con 0x7f74e015bad0 2024-03-16T19:29:32.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f0db8ff9640 1 -- 172.21.15.172:0/3045604960 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da8003770 con 0x7f0db40d92e0 2024-03-16T19:29:32.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f0db8ff9640 1 -- 172.21.15.172:0/3045604960 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8003b40 con 0x7f0db40d92e0 2024-03-16T19:29:32.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db0009400 con 0x7f0db40d92e0 2024-03-16T19:29:32.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f29811d9640 1 -- start start 2024-03-16T19:29:32.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2514849349 shutdown_connections 2024-03-16T19:29:32.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2514849349 wait complete. 2024-03-16T19:29:32.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f1000a2f640 1 -- start start 2024-03-16T19:29:32.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f74d67fc640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f74c4073cc0 0x7f74c4076180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f74d67fc640 1 -- 172.21.15.172:0/2367365928 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74d808a4c0 con 0x7f74e015bad0 2024-03-16T19:29:32.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f74e5e5f640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f74c4073cc0 0x7f74c4076180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db40da5b0 con 0x7f0db40d92e0 2024-03-16T19:29:32.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f6a9651e640 1 -- 172.21.15.172:0/871240573 shutdown_connections 2024-03-16T19:29:32.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f9990dd4640 1 Processor -- start 2024-03-16T19:29:32.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f9990dd4640 1 -- start start 2024-03-16T19:29:32.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f6a9651e640 1 -- 172.21.15.172:0/871240573 wait complete. 2024-03-16T19:29:32.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.174+0000 7f37ea59c640 1 --2- 172.21.15.172:0/3033711502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e404ba50 0x7f37e404be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:32.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f29811d9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2974006a90 0x7f29740df9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f74e5e5f640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f74c4073cc0 0x7f74c4076180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0011e0 tx=0x7f74dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f1000a2f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ffc071090 0x7f0ffc1ea130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f297bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2974006a90 0x7f29740df9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f297bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2974006a90 0x7f29740df9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46332/0 (socket says 172.21.15.172:46332) 2024-03-16T19:29:32.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f297bfff640 1 -- 172.21.15.172:0/1550554613 learned_addr learned my addr 172.21.15.172:0/1550554613 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f29811d9640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2974006e70 0x7f297404ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f1000a2f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ffc0725e0 0x7f0ffc1eae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f0ffb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ffc071090 0x7f0ffc1ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f0ffb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ffc071090 0x7f0ffc1ea130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42670/0 (socket says 172.21.15.172:42670) 2024-03-16T19:29:32.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f29811d9640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2974050500 0x7f29740e0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.175+0000 7f297b7fe640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2974006e70 0x7f297404ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f37ead9d640 1 --2- 172.21.15.172:0/3033711502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40dd3b0 0x7f37e4046450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0097a0 tx=0x7f37dc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ffd484f5b3a0522 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db40057d0 con 0x7f0db40d92e0 2024-03-16T19:29:32.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f29811d9640 1 -- 172.21.15.172:0/1550554613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29740dee20 con 0x7f2974006a90 2024-03-16T19:29:32.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f29811d9640 1 -- 172.21.15.172:0/1550554613 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29740dfef0 con 0x7f2974050500 2024-03-16T19:29:32.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f29811d9640 1 -- 172.21.15.172:0/1550554613 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29740cf6d0 con 0x7f2974006e70 2024-03-16T19:29:32.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f29809d8640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2974050500 0x7f29740e0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f0db8ff9640 1 -- 172.21.15.172:0/3045604960 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0da8003ca0 con 0x7f0db40d92e0 2024-03-16T19:29:32.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f297b7fe640 1 -- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2974050500 msgr2=0x7f29740e0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f0ffaffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ffc0725e0 0x7f0ffc1eae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f297b7fe640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2974050500 0x7f29740e0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f297b7fe640 1 -- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2974006a90 msgr2=0x7f29740df9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f297b7fe640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2974006a90 0x7f29740df9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f297b7fe640 1 -- 172.21.15.172:0/1550554613 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c073d70 con 0x7f2974006e70 2024-03-16T19:29:32.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f297bfff640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2974006a90 0x7f29740df9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:32.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f0db8ff9640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0da006b3e0 0x7f0da006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f1000a2f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0c5d70 0x7f0ffc1eb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f1000a2f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc1ea8c0 con 0x7f0ffc0725e0 2024-03-16T19:29:32.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f0ffb7fe640 1 -- 172.21.15.172:0/774730840 learned_addr learned my addr 172.21.15.172:0/774730840 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f9990dd4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974003ae0 0x7f99840d4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.176+0000 7f0db8ff9640 1 -- 172.21.15.172:0/3045604960 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0da80891f0 con 0x7f0db40d92e0 2024-03-16T19:29:32.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc1e9200 con 0x7f0ffc071090 2024-03-16T19:29:32.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f9990dd4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99840cca30 0x7f99840cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc0cac60 con 0x7f0ffc0c5d70 2024-03-16T19:29:32.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f4fa67d0640 1 Processor -- start 2024-03-16T19:29:32.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f998a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974003ae0 0x7f99840d4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f998a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974003ae0 0x7f99840d4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59946/0 (socket says 172.21.15.172:59946) 2024-03-16T19:29:32.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f998a59c640 1 -- 172.21.15.172:0/100396854 learned_addr learned my addr 172.21.15.172:0/100396854 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f9990dd4640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99840cfa50 0x7f99840cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9984171780 con 0x7f99840cca30 2024-03-16T19:29:32.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99840da200 con 0x7f99840cfa50 2024-03-16T19:29:32.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f0dbbfff640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0da006b3e0 0x7f0da006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99840368b0 con 0x7f9974003ae0 2024-03-16T19:29:32.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f297b7fe640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2974006e70 0x7f297404ffc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2970009fd0 tx=0x7f297000b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f998ad9d640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99840cfa50 0x7f99840cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f9989d9b640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99840cca30 0x7f99840cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f6a9651e640 1 Processor -- start 2024-03-16T19:29:32.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f2978ff9640 1 -- 172.21.15.172:0/1550554613 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970011830 con 0x7f2974006e70 2024-03-16T19:29:32.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f4fa67d0640 1 -- start start 2024-03-16T19:29:32.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f2978ff9640 1 -- 172.21.15.172:0/1550554613 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2970011990 con 0x7f2974006e70 2024-03-16T19:29:32.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f2978ff9640 1 -- 172.21.15.172:0/1550554613 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970011d20 con 0x7f2974006e70 2024-03-16T19:29:32.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f0ffbfff640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0c5d70 0x7f0ffc1eb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f6a9651e640 1 -- start start 2024-03-16T19:29:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f0dbbfff640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0da006b3e0 0x7f0da006d8a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0db0036000 tx=0x7f0db0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f998a59c640 1 -- 172.21.15.172:0/100396854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99840cfa50 msgr2=0x7f99840cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f998a59c640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99840cfa50 0x7f99840cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f998a59c640 1 -- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99840cca30 msgr2=0x7f99840cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f998a59c640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99840cca30 0x7f99840cbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.177+0000 7f998a59c640 1 -- 172.21.15.172:0/100396854 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9984174dc0 con 0x7f9974003ae0 2024-03-16T19:29:32.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f29811d9640 1 -- 172.21.15.172:0/1550554613 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2974050990 con 0x7f2974006e70 2024-03-16T19:29:32.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f29811d9640 1 -- 172.21.15.172:0/1550554613 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29740cf110 con 0x7f2974006e70 2024-03-16T19:29:32.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f0ffb7fe640 1 -- 172.21.15.172:0/774730840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0c5d70 msgr2=0x7f0ffc1eb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f0ffb7fe640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0c5d70 0x7f0ffc1eb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f0ffb7fe640 1 -- 172.21.15.172:0/774730840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ffc0725e0 msgr2=0x7f0ffc1eae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f0ffb7fe640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ffc0725e0 0x7f0ffc1eae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f0ffb7fe640 1 -- 172.21.15.172:0/774730840 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff0009400 con 0x7f0ffc071090 2024-03-16T19:29:32.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f998a59c640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974003ae0 0x7f99840d4140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f997c005080 tx=0x7f997c004980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f0ffb7fe640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ffc071090 0x7f0ffc1ea130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400cbc0 tx=0x7f0ff400a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f37e9d9b640 1 -- 172.21.15.172:0/3033711502 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc031640 con 0x7f37e40dd3b0 2024-03-16T19:29:32.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f37e9d9b640 1 -- 172.21.15.172:0/3033711502 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc0317a0 con 0x7f37e40dd3b0 2024-03-16T19:29:32.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f37e9d9b640 1 -- 172.21.15.172:0/3033711502 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc031640 con 0x7f37e40dd3b0 2024-03-16T19:29:32.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f4fa67d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4f98007d60 0x7f4f9817d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f4fa67d0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f980462f0 0x7f4f98036890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f4fa67d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f980dd6b0 0x7f4f9804a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f4f9f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f980462f0 0x7f4f98036890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f4f9f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f980462f0 0x7f4f98036890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46370/0 (socket says 172.21.15.172:46370) 2024-03-16T19:29:32.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f4f9f7fe640 1 -- 172.21.15.172:0/2255526655 learned_addr learned my addr 172.21.15.172:0/2255526655 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f0fe3fff640 1 -- 172.21.15.172:0/774730840 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff400aca0 con 0x7f0ffc071090 2024-03-16T19:29:32.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f74d67fc640 1 -- 172.21.15.172:0/2367365928 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74d8055100 con 0x7f74e015bad0 2024-03-16T19:29:32.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f980e4c10 con 0x7f4f980462f0 2024-03-16T19:29:32.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f98049f20 con 0x7f4f980dd6b0 2024-03-16T19:29:32.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.178+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9804db30 con 0x7f4f98007d60 2024-03-16T19:29:32.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f4fa4d6d640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f980dd6b0 0x7f4f9804a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f0fe3fff640 1 -- 172.21.15.172:0/774730840 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff400ae00 con 0x7f0ffc071090 2024-03-16T19:29:32.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f4f9ffff640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4f98007d60 0x7f4f9817d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ffc0caf40 con 0x7f0ffc071090 2024-03-16T19:29:32.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ffc0cb500 con 0x7f0ffc071090 2024-03-16T19:29:32.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f4f9f7fe640 1 -- 172.21.15.172:0/2255526655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4f98007d60 msgr2=0x7f4f9817d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f4f9f7fe640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4f98007d60 0x7f4f9817d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f4f9f7fe640 1 -- 172.21.15.172:0/2255526655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f980dd6b0 msgr2=0x7f4f9804a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f4f9f7fe640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f980dd6b0 0x7f4f9804a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f4f9f7fe640 1 -- 172.21.15.172:0/2255526655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa0073d10 con 0x7f4f980462f0 2024-03-16T19:29:32.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f6a9651e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd1d0 0x7f6a90159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f37f102c640 1 -- 172.21.15.172:0/3033711502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40dd3b0 msgr2=0x7f37e4046450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f37f102c640 1 --2- 172.21.15.172:0/3033711502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40dd3b0 0x7f37e4046450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0097a0 tx=0x7f37dc02b8f0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f37f102c640 1 -- 172.21.15.172:0/3033711502 shutdown_connections 2024-03-16T19:29:32.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f37f102c640 1 --2- 172.21.15.172:0/3033711502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e404ba50 0x7f37e404be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f37f102c640 1 --2- 172.21.15.172:0/3033711502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40dd3b0 0x7f37e4046450 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f37f102c640 1 -- 172.21.15.172:0/3033711502 >> 172.21.15.172:0/3033711502 conn(0x7f37e4050170 msgr2=0x7f37e4050580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f4f9f7fe640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f980462f0 0x7f4f98036890 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00c3f0 tx=0x7f4f8c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f4fa4d6d640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f980dd6b0 0x7f4f9804a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:32.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f29667fc640 1 -- 172.21.15.172:0/1550554613 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29740057e0 con 0x7f2974006e70 2024-03-16T19:29:32.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f6a9651e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a900cd5f0 0x7f6a900c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f6a9651e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 0x7f6a900cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f6a9651e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a901586f0 con 0x7f6a901549d0 2024-03-16T19:29:32.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f6a9651e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a901f0670 con 0x7f6a900cd5f0 2024-03-16T19:29:32.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f6a9651e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a901f0950 con 0x7f6a900cd1d0 2024-03-16T19:29:32.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f0fe3fff640 1 -- 172.21.15.172:0/774730840 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4012600 con 0x7f0ffc071090 2024-03-16T19:29:32.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f997affd640 1 -- 172.21.15.172:0/100396854 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c008030 con 0x7f9974003ae0 2024-03-16T19:29:32.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.179+0000 7f997affd640 1 -- 172.21.15.172:0/100396854 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c008190 con 0x7f9974003ae0 2024-03-16T19:29:32.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f997affd640 1 -- 172.21.15.172:0/100396854 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c008540 con 0x7f9974003ae0 2024-03-16T19:29:32.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9984170de0 con 0x7f9974003ae0 2024-03-16T19:29:32.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99841713a0 con 0x7f9974003ae0 2024-03-16T19:29:32.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f6a8ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd1d0 0x7f6a90159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f37f102c640 1 -- 172.21.15.172:0/3033711502 shutdown_connections 2024-03-16T19:29:32.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f37f102c640 1 -- 172.21.15.172:0/3033711502 wait complete. 2024-03-16T19:29:32.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f6a8ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd1d0 0x7f6a90159710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59972/0 (socket says 172.21.15.172:59972) 2024-03-16T19:29:32.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f6a8ffff640 1 -- 172.21.15.172:0/2395399998 learned_addr learned my addr 172.21.15.172:0/2395399998 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f2978ff9640 1 -- 172.21.15.172:0/1550554613 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f297001a400 con 0x7f2974006e70 2024-03-16T19:29:32.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.180+0000 7f6a94abb640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 0x7f6a900cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fdc005290 con 0x7f0ffc071090 2024-03-16T19:29:32.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f6a8f7fe640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a900cd5f0 0x7f6a900c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f2978ff9640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f296006b3e0 0x7f296006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f2978ff9640 1 -- 172.21.15.172:0/1550554613 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2970089d50 con 0x7f2974006e70 2024-03-16T19:29:32.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f297bfff640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f296006b3e0 0x7f296006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f6a94abb640 1 -- 172.21.15.172:0/2395399998 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd1d0 msgr2=0x7f6a90159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f6a94abb640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd1d0 0x7f6a90159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f6a94abb640 1 -- 172.21.15.172:0/2395399998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a900cd5f0 msgr2=0x7f6a900c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f6a94abb640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a900cd5f0 0x7f6a900c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.181+0000 7f6a94abb640 1 -- 172.21.15.172:0/2395399998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a800094d0 con 0x7f6a901549d0 2024-03-16T19:29:32.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.182+0000 7f297bfff640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f296006b3e0 0x7f296006d8a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f297c02d4f0 tx=0x7f297c0ad000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.182+0000 7f0fe3fff640 1 -- 172.21.15.172:0/774730840 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0ff4007070 con 0x7f0ffc071090 2024-03-16T19:29:32.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.182+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99840057d0 con 0x7f9974003ae0 2024-03-16T19:29:32.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.182+0000 7f6a94abb640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 0x7f6a900cf3a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800efc0 tx=0x7f6a8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.182+0000 7f0fe3fff640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0fe806b400 0x7f0fe806d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.182+0000 7f0fe3fff640 1 -- 172.21.15.172:0/774730840 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ff4089b00 con 0x7f0ffc071090 2024-03-16T19:29:32.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.182+0000 7f0ffaffd640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0fe806b400 0x7f0fe806d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.182+0000 7f37f102c640 1 Processor -- start 2024-03-16T19:29:32.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.183+0000 7f37f102c640 1 -- start start 2024-03-16T19:29:32.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.183+0000 7f0ffaffd640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0fe806b400 0x7f0fe806d8c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0035000 tx=0x7f0ff0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.183+0000 7f997affd640 1 -- 172.21.15.172:0/100396854 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f997c04b450 con 0x7f9974003ae0 2024-03-16T19:29:32.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.183+0000 7f997affd640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f996c06b4b0 0x7f996c06d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.183+0000 7f997affd640 1 -- 172.21.15.172:0/100396854 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f997c0abbe0 con 0x7f9974003ae0 2024-03-16T19:29:32.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.183+0000 7f37f102c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e404ba50 0x7f37e40dadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.183+0000 7f37f102c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e40dd3b0 0x7f37e4049c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.183+0000 7f37f102c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40e3080 0x7f37e40df820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e404ba50 0x7f37e40dadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e404ba50 0x7f37e40dadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46388/0 (socket says 172.21.15.172:46388) 2024-03-16T19:29:32.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37ead9d640 1 -- 172.21.15.172:0/2062718185 learned_addr learned my addr 172.21.15.172:0/2062718185 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40df390 con 0x7f37e404ba50 2024-03-16T19:29:32.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4178750 con 0x7f37e40e3080 2024-03-16T19:29:32.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40dea60 con 0x7f37e40dd3b0 2024-03-16T19:29:32.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f9989d9b640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f996c06b4b0 0x7f996c06d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37eb59e640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40e3080 0x7f37e40df820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37ea59c640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e40dd3b0 0x7f37e4049c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37eb59e640 1 -- 172.21.15.172:0/2062718185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e40dd3b0 msgr2=0x7f37e4049c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f4f9cff9640 1 -- 172.21.15.172:0/2255526655 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c0074e0 con 0x7f4f980462f0 2024-03-16T19:29:32.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37eb59e640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e40dd3b0 0x7f37e4049c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37eb59e640 1 -- 172.21.15.172:0/2062718185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e404ba50 msgr2=0x7f37e40dadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37eb59e640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e404ba50 0x7f37e40dadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9817d900 con 0x7f4f980462f0 2024-03-16T19:29:32.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f98048cd0 con 0x7f4f980462f0 2024-03-16T19:29:32.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f4f9cff9640 1 -- 172.21.15.172:0/2255526655 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c007640 con 0x7f4f980462f0 2024-03-16T19:29:32.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f4f9cff9640 1 -- 172.21.15.172:0/2255526655 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c0049e0 con 0x7f4f980462f0 2024-03-16T19:29:32.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37eb59e640 1 -- 172.21.15.172:0/2062718185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc009400 con 0x7f37e40e3080 2024-03-16T19:29:32.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f6a8cff9640 1 -- 172.21.15.172:0/2395399998 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a88014480 con 0x7f6a901549d0 2024-03-16T19:29:32.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f37eb59e640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40e3080 0x7f37e40df820 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f37e000c450 tx=0x7f37e000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f6a8cff9640 1 -- 172.21.15.172:0/2395399998 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8800f040 con 0x7f6a901549d0 2024-03-16T19:29:32.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.184+0000 7f6a8cff9640 1 -- 172.21.15.172:0/2395399998 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a88014480 con 0x7f6a901549d0 2024-03-16T19:29:32.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.185+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a901f0c30 con 0x7f6a901549d0 2024-03-16T19:29:32.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.185+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a90150b40 con 0x7f6a901549d0 2024-03-16T19:29:32.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.185+0000 7f0db8ff9640 1 -- 172.21.15.172:0/3045604960 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0da8053dd0 con 0x7f0db40d92e0 2024-03-16T19:29:32.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.185+0000 7f9989d9b640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f996c06b4b0 0x7f996c06d970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f99840d8dd0 tx=0x7f9980009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.186+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f80005290 con 0x7f4f980462f0 2024-03-16T19:29:32.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.187+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a90158860 con 0x7f6a901549d0 2024-03-16T19:29:32.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.187+0000 7f2978ff9640 1 -- 172.21.15.172:0/1550554613 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2970054940 con 0x7f2974006e70 2024-03-16T19:29:32.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.187+0000 7f6a8cff9640 1 -- 172.21.15.172:0/2395399998 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6a880145e0 con 0x7f6a901549d0 2024-03-16T19:29:32.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.187+0000 7f0fe3fff640 1 -- 172.21.15.172:0/774730840 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ff408be60 con 0x7f0ffc071090 2024-03-16T19:29:32.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.187+0000 7f37d37fe640 1 -- 172.21.15.172:0/2062718185 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e00074e0 con 0x7f37e40e3080 2024-03-16T19:29:32.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.187+0000 7f37d37fe640 1 -- 172.21.15.172:0/2062718185 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e0007660 con 0x7f37e40e3080 2024-03-16T19:29:32.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.187+0000 7f37d37fe640 1 -- 172.21.15.172:0/2062718185 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e0004a20 con 0x7f37e40e3080 2024-03-16T19:29:32.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.187+0000 7f6a8cff9640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a7c06b470 0x7f6a7c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.187+0000 7f6a8cff9640 1 -- 172.21.15.172:0/2395399998 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a8808a610 con 0x7f6a901549d0 2024-03-16T19:29:32.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.188+0000 7f6a8ffff640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a7c06b470 0x7f6a7c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.188+0000 7f4f9cff9640 1 -- 172.21.15.172:0/2255526655 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4f8c004b40 con 0x7f4f980462f0 2024-03-16T19:29:32.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.188+0000 7f4f9cff9640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8406b3b0 0x7f4f8406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.188+0000 7f4f9cff9640 1 -- 172.21.15.172:0/2255526655 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f8c089d50 con 0x7f4f980462f0 2024-03-16T19:29:32.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.188+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e40ded40 con 0x7f37e40e3080 2024-03-16T19:29:32.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.188+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e4178a10 con 0x7f37e40e3080 2024-03-16T19:29:32.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.189+0000 7f4f9ffff640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8406b3b0 0x7f4f8406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.189+0000 7f4f9ffff640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8406b3b0 0x7f4f8406d870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4f94005880 tx=0x7f4f94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.188+0000 7f6a8ffff640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a7c06b470 0x7f6a7c06d930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6a8002b9c0 tx=0x7f6a80008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.190+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e40df500 con 0x7f37e40e3080 2024-03-16T19:29:32.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.191+0000 7f37d37fe640 1 -- 172.21.15.172:0/2062718185 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f37e0004b80 con 0x7f37e40e3080 2024-03-16T19:29:32.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.191+0000 7f37d37fe640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d806b400 0x7f37d806d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.191+0000 7f37d37fe640 1 -- 172.21.15.172:0/2062718185 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37e0089730 con 0x7f37e40e3080 2024-03-16T19:29:32.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.192+0000 7f37ead9d640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d806b400 0x7f37d806d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.192+0000 7f37ead9d640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d806b400 0x7f37d806d8c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0097a0 tx=0x7f37dc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.192+0000 7f997affd640 1 -- 172.21.15.172:0/100396854 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f997c076960 con 0x7f9974003ae0 2024-03-16T19:29:32.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.194+0000 7f4f9cff9640 1 -- 172.21.15.172:0/2255526655 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f8c054910 con 0x7f4f980462f0 2024-03-16T19:29:32.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.197+0000 7f37d37fe640 1 -- 172.21.15.172:0/2062718185 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37e00545b0 con 0x7f37e40e3080 2024-03-16T19:29:32.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.204+0000 7f6a8cff9640 1 -- 172.21.15.172:0/2395399998 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a88055210 con 0x7f6a901549d0 2024-03-16T19:29:32.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.568+0000 7f74e6e61640 1 -- 172.21.15.172:0/2367365928 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f74c8005540 con 0x7f74e015bad0 2024-03-16T19:29:32.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.569+0000 7f74d67fc640 1 -- 172.21.15.172:0/2367365928 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f74d80587c0 con 0x7f74e015bad0 2024-03-16T19:29:32.550 INFO:teuthology.orchestra.run.smithi172.stdout:55834574854 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.572+0000 7f74c3fff640 1 -- 172.21.15.172:0/2367365928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f74c4073cc0 msgr2=0x7f74c4076180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.573+0000 7f74c3fff640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f74c4073cc0 0x7f74c4076180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0011e0 tx=0x7f74dc009040 comp rx=0 tx=0).stop 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.573+0000 7f74c3fff640 1 -- 172.21.15.172:0/2367365928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e015bad0 msgr2=0x7f74e01ec2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.573+0000 7f74c3fff640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e015bad0 0x7f74e01ec2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f74d8007e60 tx=0x7f74d800e4b0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.573+0000 7f74c3fff640 1 -- 172.21.15.172:0/2367365928 shutdown_connections 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.573+0000 7f74c3fff640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e015bad0 0x7f74e01ec2d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.573+0000 7f74c3fff640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f74c4073cc0 0x7f74c4076180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.573+0000 7f74c3fff640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e0152580 0x7f74e0158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.573+0000 7f74c3fff640 1 --2- 172.21.15.172:0/2367365928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f74e00b0ce0 0x7f74e01ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.573+0000 7f74c3fff640 1 -- 172.21.15.172:0/2367365928 >> 172.21.15.172:0/2367365928 conn(0x7f74e0066c20 msgr2=0x7f74e0151dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.574+0000 7f74c3fff640 1 -- 172.21.15.172:0/2367365928 shutdown_connections 2024-03-16T19:29:32.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.574+0000 7f74c3fff640 1 -- 172.21.15.172:0/2367365928 wait complete. 2024-03-16T19:29:32.571 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-03-16T19:29:32.571 DEBUG:teuthology.parallel:result is None 2024-03-16T19:29:32.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.595+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f0db40077d0 con 0x7f0db40d92e0 2024-03-16T19:29:32.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.595+0000 7f0db8ff9640 1 -- 172.21.15.172:0/3045604960 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0da8017090 con 0x7f0db40d92e0 2024-03-16T19:29:32.576 INFO:teuthology.orchestra.run.smithi172.stdout:55834574855 2024-03-16T19:29:32.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.597+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f99840368b0 con 0x7f9974003ae0 2024-03-16T19:29:32.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.598+0000 7f997affd640 1 -- 172.21.15.172:0/100396854 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f997c07a020 con 0x7f9974003ae0 2024-03-16T19:29:32.579 INFO:teuthology.orchestra.run.smithi172.stdout:55834574853 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0da006b3e0 msgr2=0x7f0da006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0da006b3e0 0x7f0da006d8a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0db0036000 tx=0x7f0db0008030 comp rx=0 tx=0).stop 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db40d92e0 msgr2=0x7f0db4179a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db40d92e0 0x7f0db4179a40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c410 tx=0x7f0da800c8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 shutdown_connections 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db40cf0e0 0x7f0db404a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0db40d92e0 0x7f0db4179a40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0da006b3e0 0x7f0da006d8a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 --2- 172.21.15.172:0/3045604960 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db4008180 0x7f0db40487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 >> 172.21.15.172:0/3045604960 conn(0x7f0db404d4a0 msgr2=0x7f0db40d6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.581 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 shutdown_connections 2024-03-16T19:29:32.581 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.599+0000 7f0dc236c640 1 -- 172.21.15.172:0/3045604960 wait complete. 2024-03-16T19:29:32.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.600+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f996c06b4b0 msgr2=0x7f996c06d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.600+0000 7f9990dd4640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f996c06b4b0 0x7f996c06d970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f99840d8dd0 tx=0x7f9980009060 comp rx=0 tx=0).stop 2024-03-16T19:29:32.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.600+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974003ae0 msgr2=0x7f99840d4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.600+0000 7f9990dd4640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974003ae0 0x7f99840d4140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f997c005080 tx=0x7f997c004980 comp rx=0 tx=0).stop 2024-03-16T19:29:32.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.601+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 shutdown_connections 2024-03-16T19:29:32.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.601+0000 7f9990dd4640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99840cfa50 0x7f99840cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.601+0000 7f9990dd4640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99840cca30 0x7f99840cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.601+0000 7f9990dd4640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f996c06b4b0 0x7f996c06d970 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.601+0000 7f9990dd4640 1 --2- 172.21.15.172:0/100396854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9974003ae0 0x7f99840d4140 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.601+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 >> 172.21.15.172:0/100396854 conn(0x7f998404d4a0 msgr2=0x7f99840d9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.601+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 shutdown_connections 2024-03-16T19:29:32.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.601+0000 7f9990dd4640 1 -- 172.21.15.172:0/100396854 wait complete. 2024-03-16T19:29:32.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.602+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f4f80005810 con 0x7f4f980462f0 2024-03-16T19:29:32.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.602+0000 7f4f9cff9640 1 -- 172.21.15.172:0/2255526655 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4f8c057fd0 con 0x7f4f980462f0 2024-03-16T19:29:32.585 INFO:teuthology.orchestra.run.smithi172.stdout:55834574855 2024-03-16T19:29:32.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.603+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0fdc005540 con 0x7f0ffc071090 2024-03-16T19:29:32.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.604+0000 7f0fe3fff640 1 -- 172.21.15.172:0/774730840 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0ff4017020 con 0x7f0ffc071090 2024-03-16T19:29:32.587 INFO:teuthology.orchestra.run.smithi172.stdout:55834574854 2024-03-16T19:29:32.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.604+0000 7f29667fc640 1 -- 172.21.15.172:0/1550554613 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f29740074f0 con 0x7f2974006e70 2024-03-16T19:29:32.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.605+0000 7f2978ff9640 1 -- 172.21.15.172:0/1550554613 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f29700091f0 con 0x7f2974006e70 2024-03-16T19:29:32.588 INFO:teuthology.orchestra.run.smithi172.stdout:55834574855 2024-03-16T19:29:32.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.605+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8406b3b0 msgr2=0x7f4f8406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.605+0000 7f4fa67d0640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8406b3b0 0x7f4f8406d870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4f94005880 tx=0x7f4f94009040 comp rx=0 tx=0).stop 2024-03-16T19:29:32.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.605+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f980462f0 msgr2=0x7f4f98036890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.605+0000 7f4fa67d0640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f980462f0 0x7f4f98036890 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00c3f0 tx=0x7f4f8c00c8b0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.605+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 shutdown_connections 2024-03-16T19:29:32.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.605+0000 7f4fa67d0640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f980dd6b0 0x7f4f9804a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.605+0000 7f4fa67d0640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f980462f0 0x7f4f98036890 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.605+0000 7f4fa67d0640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4f8406b3b0 0x7f4f8406d870 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.606+0000 7f4fa67d0640 1 --2- 172.21.15.172:0/2255526655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4f98007d60 0x7f4f9817d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.606+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 >> 172.21.15.172:0/2255526655 conn(0x7f4f9804dd20 msgr2=0x7f4f980db080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.606+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 shutdown_connections 2024-03-16T19:29:32.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.606+0000 7f4fa67d0640 1 -- 172.21.15.172:0/2255526655 wait complete. 2024-03-16T19:29:32.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0fe806b400 msgr2=0x7f0fe806d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0fe806b400 0x7f0fe806d8c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0035000 tx=0x7f0ff0008030 comp rx=0 tx=0).stop 2024-03-16T19:29:32.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ffc071090 msgr2=0x7f0ffc1ea130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ffc071090 0x7f0ffc1ea130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400cbc0 tx=0x7f0ff400a060 comp rx=0 tx=0).stop 2024-03-16T19:29:32.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 shutdown_connections 2024-03-16T19:29:32.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0fe806b400 0x7f0fe806d8c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ffc0c5d70 0x7f0ffc1eb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ffc0725e0 0x7f0ffc1eae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 --2- 172.21.15.172:0/774730840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ffc071090 0x7f0ffc1ea130 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 >> 172.21.15.172:0/774730840 conn(0x7f0ffc06aed0 msgr2=0x7f0ffc0c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 shutdown_connections 2024-03-16T19:29:32.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.607+0000 7f1000a2f640 1 -- 172.21.15.172:0/774730840 wait complete. 2024-03-16T19:29:32.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 -- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f296006b3e0 msgr2=0x7f296006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f296006b3e0 0x7f296006d8a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f297c02d4f0 tx=0x7f297c0ad000 comp rx=0 tx=0).stop 2024-03-16T19:29:32.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 -- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2974006e70 msgr2=0x7f297404ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2974006e70 0x7f297404ffc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2970009fd0 tx=0x7f297000b870 comp rx=0 tx=0).stop 2024-03-16T19:29:32.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 -- 172.21.15.172:0/1550554613 shutdown_connections 2024-03-16T19:29:32.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2974050500 0x7f29740e0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f296006b3e0 0x7f296006d8a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2974006e70 0x7f297404ffc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 --2- 172.21.15.172:0/1550554613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2974006a90 0x7f29740df9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.608+0000 7f29667fc640 1 -- 172.21.15.172:0/1550554613 >> 172.21.15.172:0/1550554613 conn(0x7f297404dd30 msgr2=0x7f2974008c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.609+0000 7f29667fc640 1 -- 172.21.15.172:0/1550554613 shutdown_connections 2024-03-16T19:29:32.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.609+0000 7f29667fc640 1 -- 172.21.15.172:0/1550554613 wait complete. 2024-03-16T19:29:32.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.610+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f37e4005380 con 0x7f37e40e3080 2024-03-16T19:29:32.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.611+0000 7f37d37fe640 1 -- 172.21.15.172:0/2062718185 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f37e0057c70 con 0x7f37e40e3080 2024-03-16T19:29:32.599 INFO:teuthology.orchestra.run.smithi172.stdout:55834574856 2024-03-16T19:29:32.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.614+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6a900758f0 con 0x7f6a901549d0 2024-03-16T19:29:32.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d806b400 msgr2=0x7f37d806d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d806b400 0x7f37d806d8c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0097a0 tx=0x7f37dc03d040 comp rx=0 tx=0).stop 2024-03-16T19:29:32.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40e3080 msgr2=0x7f37e40df820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40e3080 0x7f37e40df820 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f37e000c450 tx=0x7f37e000c910 comp rx=0 tx=0).stop 2024-03-16T19:29:32.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 shutdown_connections 2024-03-16T19:29:32.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e40e3080 0x7f37e40df820 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d806b400 0x7f37d806d8c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37e40dd3b0 0x7f37e4049c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 --2- 172.21.15.172:0/2062718185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e404ba50 0x7f37e40dadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 >> 172.21.15.172:0/2062718185 conn(0x7f37e4050170 msgr2=0x7f37e40e2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f6a8cff9640 1 -- 172.21.15.172:0/2395399998 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a88055210 con 0x7f6a901549d0 2024-03-16T19:29:32.604 INFO:teuthology.orchestra.run.smithi172.stdout:55834574854 2024-03-16T19:29:32.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 shutdown_connections 2024-03-16T19:29:32.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.615+0000 7f37f102c640 1 -- 172.21.15.172:0/2062718185 wait complete. 2024-03-16T19:29:32.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a7c06b470 msgr2=0x7f6a7c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a7c06b470 0x7f6a7c06d930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6a8002b9c0 tx=0x7f6a80008030 comp rx=0 tx=0).stop 2024-03-16T19:29:32.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 msgr2=0x7f6a900cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 0x7f6a900cf3a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800efc0 tx=0x7f6a8800c490 comp rx=0 tx=0).stop 2024-03-16T19:29:32.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 shutdown_connections 2024-03-16T19:29:32.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a901549d0 0x7f6a900cf3a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a900cd5f0 0x7f6a900c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a7c06b470 0x7f6a7c06d930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 --2- 172.21.15.172:0/2395399998 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a900cd1d0 0x7f6a90159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.619+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 >> 172.21.15.172:0/2395399998 conn(0x7f6a9006d640 msgr2=0x7f6a900b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.620+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 shutdown_connections 2024-03-16T19:29:32.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.620+0000 7f6a9651e640 1 -- 172.21.15.172:0/2395399998 wait complete. 2024-03-16T19:29:32.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-03-16T19:29:32.609 DEBUG:teuthology.parallel:result is None 2024-03-16T19:29:32.609 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2024-03-16T19:29:32.610 DEBUG:teuthology.parallel:result is None 2024-03-16T19:29:32.610 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-03-16T19:29:32.611 DEBUG:teuthology.parallel:result is None 2024-03-16T19:29:32.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-03-16T19:29:32.612 DEBUG:teuthology.parallel:result is None 2024-03-16T19:29:32.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-03-16T19:29:32.612 DEBUG:teuthology.parallel:result is None 2024-03-16T19:29:32.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-03-16T19:29:32.615 DEBUG:teuthology.parallel:result is None 2024-03-16T19:29:32.616 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-03-16T19:29:32.616 DEBUG:teuthology.parallel:result is None 2024-03-16T19:29:32.616 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-16T19:29:32.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:29:32.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.775+0000 7f194f82a640 1 Processor -- start 2024-03-16T19:29:32.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.775+0000 7f194f82a640 1 -- start start 2024-03-16T19:29:32.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f194f82a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f19480cb2e0 0x7f194806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f194f82a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19480cbba0 con 0x7f1948154cb0 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f194f82a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19480c91f0 con 0x7f194806fe10 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f194f82a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194806c180 con 0x7f19480cb2e0 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f194d5c6640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1948154cb0 0x7f194815de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56228/0 (socket says 172.21.15.172:56228) 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f194d5c6640 1 -- 172.21.15.172:0/2933152211 learned_addr learned my addr 172.21.15.172:0/2933152211 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041735695 0 0) 0x7f19480cbba0 con 0x7f1948154cb0 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1934003520 con 0x7f1948154cb0 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f194ddc7640 1 --2- 172.21.15.172:0/2933152211 >> v2:172.21.15.174:3300/0 conn(0x7f19480cb2e0 0x7f194806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2549253210 0 0) 0x7f1934003520 con 0x7f1948154cb0 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 >> v2:172.21.15.174:3300/0 conn(0x7f19480cb2e0 msgr2=0x7f194806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f193ffff640 1 --2- 172.21.15.172:0/2933152211 >> v2:172.21.15.174:3300/0 conn(0x7f19480cb2e0 0x7f194806dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.776+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 >> v1:172.21.15.174:6789/0 conn(0x7f194806fe10 legacy=0x7f19480cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194806c460 con 0x7f1948154cb0 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f19440019f0 con 0x7f1948154cb0 2024-03-16T19:29:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 >> v1:172.21.15.172:6789/0 conn(0x7f1948154cb0 legacy=0x7f194815de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f193ffff640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1934003970 0x7f1934003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f193ffff640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19340047c0 0x7f1934004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f193ffff640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1934005380 0x7f193400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194806c180 con 0x7f19340047c0 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193400c210 con 0x7f1934003970 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193400c3d0 con 0x7f1934005380 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f194cdc5640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19340047c0 0x7f1934004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f194d5c6640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1934003970 0x7f1934003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f194ddc7640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1934005380 0x7f193400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f194cdc5640 1 -- 172.21.15.172:0/2933152211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1934005380 msgr2=0x7f193400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f194cdc5640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1934005380 0x7f193400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f194cdc5640 1 -- 172.21.15.172:0/2933152211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1934003970 msgr2=0x7f1934003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f194cdc5640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1934003970 0x7f1934003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f194ddc7640 1 -- 172.21.15.172:0/2933152211 reap_dead start 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.777+0000 7f194cdc5640 1 -- 172.21.15.172:0/2933152211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1934003520 con 0x7f19340047c0 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f194cdc5640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19340047c0 0x7f1934004c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1938001750 tx=0x7f19340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e038902fc3ba5e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380052d0 con 0x7f19340047c0 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193802e050 con 0x7f19340047c0 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f193ffff640 1 -- 172.21.15.172:0/2933152211 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938003830 con 0x7f19340047c0 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f194f82a640 1 -- 172.21.15.172:0/2933152211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19340047c0 msgr2=0x7f1934004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f194f82a640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19340047c0 0x7f1934004c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1938001750 tx=0x7f19340057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f194f82a640 1 -- 172.21.15.172:0/2933152211 shutdown_connections 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f194f82a640 1 --2- 172.21.15.172:0/2933152211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19340047c0 0x7f1934004c40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f194f82a640 1 -- 172.21.15.172:0/2933152211 >> 172.21.15.172:0/2933152211 conn(0x7f1948070860 msgr2=0x7f1948070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.778+0000 7f194f82a640 1 -- 172.21.15.172:0/2933152211 shutdown_connections 2024-03-16T19:29:32.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.779+0000 7f194f82a640 1 -- 172.21.15.172:0/2933152211 wait complete. 2024-03-16T19:29:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194f82a640 1 Processor -- start 2024-03-16T19:29:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194f82a640 1 -- start start 2024-03-16T19:29:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194f82a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19340047c0 0x7f19481f02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194f82a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194806f090 0x7f19480c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194f82a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19480b1bd0 0x7f194814fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194f82a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19480b15f0 con 0x7f194806f090 2024-03-16T19:29:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194f82a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19480c7c20 con 0x7f19480b1bd0 2024-03-16T19:29:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194f82a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1948150110 con 0x7f19340047c0 2024-03-16T19:29:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194cdc5640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194806f090 0x7f19480c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194cdc5640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194806f090 0x7f19480c76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46400/0 (socket says 172.21.15.172:46400) 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194cdc5640 1 -- 172.21.15.172:0/1993978178 learned_addr learned my addr 172.21.15.172:0/1993978178 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.780+0000 7f194d5c6640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19340047c0 0x7f19481f02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f194ddc7640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19480b1bd0 0x7f194814fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f194cdc5640 1 -- 172.21.15.172:0/1993978178 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19340047c0 msgr2=0x7f19481f02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f194cdc5640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19340047c0 0x7f19481f02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f194cdc5640 1 -- 172.21.15.172:0/1993978178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19480b1bd0 msgr2=0x7f194814fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f194cdc5640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19480b1bd0 0x7f194814fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f194cdc5640 1 -- 172.21.15.172:0/1993978178 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19480c6360 con 0x7f194806f090 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f194cdc5640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194806f090 0x7f19480c76e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1938005040 tx=0x7f1938001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f193dffb640 1 -- 172.21.15.172:0/1993978178 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938007a30 con 0x7f194806f090 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f193dffb640 1 -- 172.21.15.172:0/1993978178 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1938007640 con 0x7f194806f090 2024-03-16T19:29:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f193dffb640 1 -- 172.21.15.172:0/1993978178 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938006c00 con 0x7f194806f090 2024-03-16T19:29:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19480c6640 con 0x7f194806f090 2024-03-16T19:29:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.781+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19480c6c00 con 0x7f194806f090 2024-03-16T19:29:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.782+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19480681c0 con 0x7f194806f090 2024-03-16T19:29:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.782+0000 7f193dffb640 1 -- 172.21.15.172:0/1993978178 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f19380035f0 con 0x7f194806f090 2024-03-16T19:29:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.783+0000 7f193dffb640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f192c06b3b0 0x7f192c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.783+0000 7f194d5c6640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f192c06b3b0 0x7f192c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.783+0000 7f193dffb640 1 -- 172.21.15.172:0/1993978178 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19380ab610 con 0x7f194806f090 2024-03-16T19:29:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.783+0000 7f194d5c6640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f192c06b3b0 0x7f192c06d870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1944004d00 tx=0x7f1944006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:32.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.787+0000 7f193dffb640 1 -- 172.21.15.172:0/1993978178 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1938076270 con 0x7f194806f090 2024-03-16T19:29:32.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.955+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19480c9790 con 0x7f192c06b3b0 2024-03-16T19:29:32.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.960+0000 7f193dffb640 1 -- 172.21.15.172:0/1993978178 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251947 (secure 0 0 0) 0x7f1944004d30 con 0x7f192c06b3b0 2024-03-16T19:29:32.944 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:32.945 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:29:32.946 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f192c06b3b0 msgr2=0x7f192c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f192c06b3b0 0x7f192c06d870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1944004d00 tx=0x7f1944006040 comp rx=0 tx=0).stop 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194806f090 msgr2=0x7f19480c76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194806f090 0x7f19480c76e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1938005040 tx=0x7f1938001120 comp rx=0 tx=0).stop 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 shutdown_connections 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19480b1bd0 0x7f194814fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194806f090 0x7f19480c76e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f192c06b3b0 0x7f192c06d870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 --2- 172.21.15.172:0/1993978178 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19340047c0 0x7f19481f02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.963+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 >> 172.21.15.172:0/1993978178 conn(0x7f1948070860 msgr2=0x7f19480cbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.964+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 shutdown_connections 2024-03-16T19:29:32.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:32.964+0000 7f194f82a640 1 -- 172.21.15.172:0/1993978178 wait complete. 2024-03-16T19:29:32.955 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-16T19:29:32.200894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219040,"kb_used_data":2896,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755680,"statfs":{"total":773094113280,"available":772869816320,"internally_reserved":0,"allocated":2965504,"data_stored":1788166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.876361"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344501+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:29:27.344501+0000","last_peered":"2024-03-16T19:29:27.344501+0000","last_clean":"2024-03-16T19:29:27.344501+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:29:27.344501+0000","last_undegraded":"2024-03-16T19:29:27.344501+0000","last_fullsized":"2024-03-16T19:29:27.344501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339631+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:29:27.339631+0000","last_peered":"2024-03-16T19:29:27.339631+0000","last_clean":"2024-03-16T19:29:27.339631+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:29:27.339631+0000","last_undegraded":"2024-03-16T19:29:27.339631+0000","last_fullsized":"2024-03-16T19:29:27.339631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062822+0000","last_change":"2024-03-16T19:29:26.347979+0000","last_active":"2024-03-16T19:29:28.062822+0000","last_peered":"2024-03-16T19:29:28.062822+0000","last_clean":"2024-03-16T19:29:28.062822+0000","last_became_active":"2024-03-16T19:29:26.347711+0000","last_became_peered":"2024-03-16T19:29:26.347711+0000","last_unstale":"2024-03-16T19:29:28.062822+0000","last_undegraded":"2024-03-16T19:29:28.062822+0000","last_fullsized":"2024-03-16T19:29:28.062822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:08.017182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343558+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:29:27.343558+0000","last_peered":"2024-03-16T19:29:27.343558+0000","last_clean":"2024-03-16T19:29:27.343558+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:29:27.343558+0000","last_undegraded":"2024-03-16T19:29:27.343558+0000","last_fullsized":"2024-03-16T19:29:27.343558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344270+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:29:27.344270+0000","last_peered":"2024-03-16T19:29:27.344270+0000","last_clean":"2024-03-16T19:29:27.344270+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:29:27.344270+0000","last_undegraded":"2024-03-16T19:29:27.344270+0000","last_fullsized":"2024-03-16T19:29:27.344270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343104+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:29:27.343104+0000","last_peered":"2024-03-16T19:29:27.343104+0000","last_clean":"2024-03-16T19:29:27.343104+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:29:27.343104+0000","last_undegraded":"2024-03-16T19:29:27.343104+0000","last_fullsized":"2024-03-16T19:29:27.343104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344269+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:29:27.344269+0000","last_peered":"2024-03-16T19:29:27.344269+0000","last_clean":"2024-03-16T19:29:27.344269+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:29:27.344269+0000","last_undegraded":"2024-03-16T19:29:27.344269+0000","last_fullsized":"2024-03-16T19:29:27.344269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561737+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:29:27.561737+0000","last_peered":"2024-03-16T19:29:27.561737+0000","last_clean":"2024-03-16T19:29:27.561737+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:29:27.561737+0000","last_undegraded":"2024-03-16T19:29:27.561737+0000","last_fullsized":"2024-03-16T19:29:27.561737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344497+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:29:27.344497+0000","last_peered":"2024-03-16T19:29:27.344497+0000","last_clean":"2024-03-16T19:29:27.344497+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:29:27.344497+0000","last_undegraded":"2024-03-16T19:29:27.344497+0000","last_fullsized":"2024-03-16T19:29:27.344497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063286+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:29:28.063286+0000","last_peered":"2024-03-16T19:29:28.063286+0000","last_clean":"2024-03-16T19:29:28.063286+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:29:28.063286+0000","last_undegraded":"2024-03-16T19:29:28.063286+0000","last_fullsized":"2024-03-16T19:29:28.063286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062996+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:29:28.062996+0000","last_peered":"2024-03-16T19:29:28.062996+0000","last_clean":"2024-03-16T19:29:28.062996+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:29:28.062996+0000","last_undegraded":"2024-03-16T19:29:28.062996+0000","last_fullsized":"2024-03-16T19:29:28.062996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063177+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:29:28.063177+0000","last_peered":"2024-03-16T19:29:28.063177+0000","last_clean":"2024-03-16T19:29:28.063177+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:29:28.063177+0000","last_undegraded":"2024-03-16T19:29:28.063177+0000","last_fullsized":"2024-03-16T19:29:28.063177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561930+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:29:27.561930+0000","last_peered":"2024-03-16T19:29:27.561930+0000","last_clean":"2024-03-16T19:29:27.561930+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:29:27.561930+0000","last_undegraded":"2024-03-16T19:29:27.561930+0000","last_fullsized":"2024-03-16T19:29:27.561930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062462+0000","last_change":"2024-03-16T19:29:26.349644+0000","last_active":"2024-03-16T19:29:28.062462+0000","last_peered":"2024-03-16T19:29:28.062462+0000","last_clean":"2024-03-16T19:29:28.062462+0000","last_became_active":"2024-03-16T19:29:26.349397+0000","last_became_peered":"2024-03-16T19:29:26.349397+0000","last_unstale":"2024-03-16T19:29:28.062462+0000","last_undegraded":"2024-03-16T19:29:28.062462+0000","last_fullsized":"2024-03-16T19:29:28.062462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:47:20.566393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339397+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:29:27.339397+0000","last_peered":"2024-03-16T19:29:27.339397+0000","last_clean":"2024-03-16T19:29:27.339397+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:29:27.339397+0000","last_undegraded":"2024-03-16T19:29:27.339397+0000","last_fullsized":"2024-03-16T19:29:27.339397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344311+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:29:27.344311+0000","last_peered":"2024-03-16T19:29:27.344311+0000","last_clean":"2024-03-16T19:29:27.344311+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:29:27.344311+0000","last_undegraded":"2024-03-16T19:29:27.344311+0000","last_fullsized":"2024-03-16T19:29:27.344311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062569+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:29:28.062569+0000","last_peered":"2024-03-16T19:29:28.062569+0000","last_clean":"2024-03-16T19:29:28.062569+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:29:28.062569+0000","last_undegraded":"2024-03-16T19:29:28.062569+0000","last_fullsized":"2024-03-16T19:29:28.062569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344648+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:29:27.344648+0000","last_peered":"2024-03-16T19:29:27.344648+0000","last_clean":"2024-03-16T19:29:27.344648+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:29:27.344648+0000","last_undegraded":"2024-03-16T19:29:27.344648+0000","last_fullsized":"2024-03-16T19:29:27.344648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063112+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:29:28.063112+0000","last_peered":"2024-03-16T19:29:28.063112+0000","last_clean":"2024-03-16T19:29:28.063112+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:29:28.063112+0000","last_undegraded":"2024-03-16T19:29:28.063112+0000","last_fullsized":"2024-03-16T19:29:28.063112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062498+0000","last_change":"2024-03-16T19:29:26.357191+0000","last_active":"2024-03-16T19:29:28.062498+0000","last_peered":"2024-03-16T19:29:28.062498+0000","last_clean":"2024-03-16T19:29:28.062498+0000","last_became_active":"2024-03-16T19:29:26.356947+0000","last_became_peered":"2024-03-16T19:29:26.356947+0000","last_unstale":"2024-03-16T19:29:28.062498+0000","last_undegraded":"2024-03-16T19:29:28.062498+0000","last_fullsized":"2024-03-16T19:29:28.062498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:40.884164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343265+0000","last_change":"2024-03-16T19:29:26.358307+0000","last_active":"2024-03-16T19:29:27.343265+0000","last_peered":"2024-03-16T19:29:27.343265+0000","last_clean":"2024-03-16T19:29:27.343265+0000","last_became_active":"2024-03-16T19:29:26.357970+0000","last_became_peered":"2024-03-16T19:29:26.357970+0000","last_unstale":"2024-03-16T19:29:27.343265+0000","last_undegraded":"2024-03-16T19:29:27.343265+0000","last_fullsized":"2024-03-16T19:29:27.343265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:33.976868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561749+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:29:27.561749+0000","last_peered":"2024-03-16T19:29:27.561749+0000","last_clean":"2024-03-16T19:29:27.561749+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:29:27.561749+0000","last_undegraded":"2024-03-16T19:29:27.561749+0000","last_fullsized":"2024-03-16T19:29:27.561749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343286+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:29:27.343286+0000","last_peered":"2024-03-16T19:29:27.343286+0000","last_clean":"2024-03-16T19:29:27.343286+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:29:27.343286+0000","last_undegraded":"2024-03-16T19:29:27.343286+0000","last_fullsized":"2024-03-16T19:29:27.343286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562192+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:29:27.562192+0000","last_peered":"2024-03-16T19:29:27.562192+0000","last_clean":"2024-03-16T19:29:27.562192+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:29:27.562192+0000","last_undegraded":"2024-03-16T19:29:27.562192+0000","last_fullsized":"2024-03-16T19:29:27.562192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063443+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:29:28.063443+0000","last_peered":"2024-03-16T19:29:28.063443+0000","last_clean":"2024-03-16T19:29:28.063443+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:29:28.063443+0000","last_undegraded":"2024-03-16T19:29:28.063443+0000","last_fullsized":"2024-03-16T19:29:28.063443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339564+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:29:27.339564+0000","last_peered":"2024-03-16T19:29:27.339564+0000","last_clean":"2024-03-16T19:29:27.339564+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:29:27.339564+0000","last_undegraded":"2024-03-16T19:29:27.339564+0000","last_fullsized":"2024-03-16T19:29:27.339564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562214+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:29:27.562214+0000","last_peered":"2024-03-16T19:29:27.562214+0000","last_clean":"2024-03-16T19:29:27.562214+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:29:27.562214+0000","last_undegraded":"2024-03-16T19:29:27.562214+0000","last_fullsized":"2024-03-16T19:29:27.562214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062637+0000","last_change":"2024-03-16T19:29:26.349443+0000","last_active":"2024-03-16T19:29:28.062637+0000","last_peered":"2024-03-16T19:29:28.062637+0000","last_clean":"2024-03-16T19:29:28.062637+0000","last_became_active":"2024-03-16T19:29:26.349181+0000","last_became_peered":"2024-03-16T19:29:26.349181+0000","last_unstale":"2024-03-16T19:29:28.062637+0000","last_undegraded":"2024-03-16T19:29:28.062637+0000","last_fullsized":"2024-03-16T19:29:28.062637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:16.483346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343433+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:29:27.343433+0000","last_peered":"2024-03-16T19:29:27.343433+0000","last_clean":"2024-03-16T19:29:27.343433+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:29:27.343433+0000","last_undegraded":"2024-03-16T19:29:27.343433+0000","last_fullsized":"2024-03-16T19:29:27.343433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062168+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:29:28.062168+0000","last_peered":"2024-03-16T19:29:28.062168+0000","last_clean":"2024-03-16T19:29:28.062168+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:29:28.062168+0000","last_undegraded":"2024-03-16T19:29:28.062168+0000","last_fullsized":"2024-03-16T19:29:28.062168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062728+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:29:28.062728+0000","last_peered":"2024-03-16T19:29:28.062728+0000","last_clean":"2024-03-16T19:29:28.062728+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:29:28.062728+0000","last_undegraded":"2024-03-16T19:29:28.062728+0000","last_fullsized":"2024-03-16T19:29:28.062728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063484+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:29:28.063484+0000","last_peered":"2024-03-16T19:29:28.063484+0000","last_clean":"2024-03-16T19:29:28.063484+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:29:28.063484+0000","last_undegraded":"2024-03-16T19:29:28.063484+0000","last_fullsized":"2024-03-16T19:29:28.063484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.215645+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:29:28.215645+0000","last_peered":"2024-03-16T19:29:28.215645+0000","last_clean":"2024-03-16T19:29:28.215645+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:29:28.215645+0000","last_undegraded":"2024-03-16T19:29:28.215645+0000","last_fullsized":"2024-03-16T19:29:28.215645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062765+0000","last_change":"2024-03-16T19:29:26.353288+0000","last_active":"2024-03-16T19:29:28.062765+0000","last_peered":"2024-03-16T19:29:28.062765+0000","last_clean":"2024-03-16T19:29:28.062765+0000","last_became_active":"2024-03-16T19:29:26.352973+0000","last_became_peered":"2024-03-16T19:29:26.352973+0000","last_unstale":"2024-03-16T19:29:28.062765+0000","last_undegraded":"2024-03-16T19:29:28.062765+0000","last_fullsized":"2024-03-16T19:29:28.062765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:22:41.505003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.210595+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:29:28.210595+0000","last_peered":"2024-03-16T19:29:28.210595+0000","last_clean":"2024-03-16T19:29:28.210595+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:29:28.210595+0000","last_undegraded":"2024-03-16T19:29:28.210595+0000","last_fullsized":"2024-03-16T19:29:28.210595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561856+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:29:27.561856+0000","last_peered":"2024-03-16T19:29:27.561856+0000","last_clean":"2024-03-16T19:29:27.561856+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:29:27.561856+0000","last_undegraded":"2024-03-16T19:29:27.561856+0000","last_fullsized":"2024-03-16T19:29:27.561856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561778+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:29:27.561778+0000","last_peered":"2024-03-16T19:29:27.561778+0000","last_clean":"2024-03-16T19:29:27.561778+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:29:27.561778+0000","last_undegraded":"2024-03-16T19:29:27.561778+0000","last_fullsized":"2024-03-16T19:29:27.561778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343316+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:29:27.343316+0000","last_peered":"2024-03-16T19:29:27.343316+0000","last_clean":"2024-03-16T19:29:27.343316+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:29:27.343316+0000","last_undegraded":"2024-03-16T19:29:27.343316+0000","last_fullsized":"2024-03-16T19:29:27.343316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339401+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:29:27.339401+0000","last_peered":"2024-03-16T19:29:27.339401+0000","last_clean":"2024-03-16T19:29:27.339401+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:29:27.339401+0000","last_undegraded":"2024-03-16T19:29:27.339401+0000","last_fullsized":"2024-03-16T19:29:27.339401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063312+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:29:28.063312+0000","last_peered":"2024-03-16T19:29:28.063312+0000","last_clean":"2024-03-16T19:29:28.063312+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:29:28.063312+0000","last_undegraded":"2024-03-16T19:29:28.063312+0000","last_fullsized":"2024-03-16T19:29:28.063312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344324+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:29:27.344324+0000","last_peered":"2024-03-16T19:29:27.344324+0000","last_clean":"2024-03-16T19:29:27.344324+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:29:27.344324+0000","last_undegraded":"2024-03-16T19:29:27.344324+0000","last_fullsized":"2024-03-16T19:29:27.344324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343744+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:29:27.343744+0000","last_peered":"2024-03-16T19:29:27.343744+0000","last_clean":"2024-03-16T19:29:27.343744+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:29:27.343744+0000","last_undegraded":"2024-03-16T19:29:27.343744+0000","last_fullsized":"2024-03-16T19:29:27.343744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062922+0000","last_change":"2024-03-16T19:29:26.353918+0000","last_active":"2024-03-16T19:29:28.062922+0000","last_peered":"2024-03-16T19:29:28.062922+0000","last_clean":"2024-03-16T19:29:28.062922+0000","last_became_active":"2024-03-16T19:29:26.353463+0000","last_became_peered":"2024-03-16T19:29:26.353463+0000","last_unstale":"2024-03-16T19:29:28.062922+0000","last_undegraded":"2024-03-16T19:29:28.062922+0000","last_fullsized":"2024-03-16T19:29:28.062922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:36.035692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.362815+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:29:31.362815+0000","last_peered":"2024-03-16T19:29:31.362815+0000","last_clean":"2024-03-16T19:29:31.362815+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:29:31.362815+0000","last_undegraded":"2024-03-16T19:29:31.362815+0000","last_fullsized":"2024-03-16T19:29:31.362815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063360+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:29:28.063360+0000","last_peered":"2024-03-16T19:29:28.063360+0000","last_clean":"2024-03-16T19:29:28.063360+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:29:28.063360+0000","last_undegraded":"2024-03-16T19:29:28.063360+0000","last_fullsized":"2024-03-16T19:29:28.063360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365125+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:29:31.365125+0000","last_peered":"2024-03-16T19:29:31.365125+0000","last_clean":"2024-03-16T19:29:31.365125+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:29:31.365125+0000","last_undegraded":"2024-03-16T19:29:31.365125+0000","last_fullsized":"2024-03-16T19:29:31.365125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344465+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:29:27.344465+0000","last_peered":"2024-03-16T19:29:27.344465+0000","last_clean":"2024-03-16T19:29:27.344465+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:29:27.344465+0000","last_undegraded":"2024-03-16T19:29:27.344465+0000","last_fullsized":"2024-03-16T19:29:27.344465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365001+0000","last_change":"2024-03-16T19:29:25.336296+0000","last_active":"2024-03-16T19:29:31.365001+0000","last_peered":"2024-03-16T19:29:31.365001+0000","last_clean":"2024-03-16T19:29:31.365001+0000","last_became_active":"2024-03-16T19:29:25.333046+0000","last_became_peered":"2024-03-16T19:29:25.333046+0000","last_unstale":"2024-03-16T19:29:31.365001+0000","last_undegraded":"2024-03-16T19:29:31.365001+0000","last_fullsized":"2024-03-16T19:29:31.365001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:33.538342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062507+0000","last_change":"2024-03-16T19:29:26.354709+0000","last_active":"2024-03-16T19:29:28.062507+0000","last_peered":"2024-03-16T19:29:28.062507+0000","last_clean":"2024-03-16T19:29:28.062507+0000","last_became_active":"2024-03-16T19:29:26.354377+0000","last_became_peered":"2024-03-16T19:29:26.354377+0000","last_unstale":"2024-03-16T19:29:28.062507+0000","last_undegraded":"2024-03-16T19:29:28.062507+0000","last_fullsized":"2024-03-16T19:29:28.062507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:41:56.277270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343553+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:29:27.343553+0000","last_peered":"2024-03-16T19:29:27.343553+0000","last_clean":"2024-03-16T19:29:27.343553+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:29:27.343553+0000","last_undegraded":"2024-03-16T19:29:27.343553+0000","last_fullsized":"2024-03-16T19:29:27.343553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344364+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:29:27.344364+0000","last_peered":"2024-03-16T19:29:27.344364+0000","last_clean":"2024-03-16T19:29:27.344364+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:29:27.344364+0000","last_undegraded":"2024-03-16T19:29:27.344364+0000","last_fullsized":"2024-03-16T19:29:27.344364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062737+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:29:28.062737+0000","last_peered":"2024-03-16T19:29:28.062737+0000","last_clean":"2024-03-16T19:29:28.062737+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:29:28.062737+0000","last_undegraded":"2024-03-16T19:29:28.062737+0000","last_fullsized":"2024-03-16T19:29:28.062737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062954+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:29:28.062954+0000","last_peered":"2024-03-16T19:29:28.062954+0000","last_clean":"2024-03-16T19:29:28.062954+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:29:28.062954+0000","last_undegraded":"2024-03-16T19:29:28.062954+0000","last_fullsized":"2024-03-16T19:29:28.062954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365607+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:29:31.365607+0000","last_peered":"2024-03-16T19:29:31.365607+0000","last_clean":"2024-03-16T19:29:31.365607+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:29:31.365607+0000","last_undegraded":"2024-03-16T19:29:31.365607+0000","last_fullsized":"2024-03-16T19:29:31.365607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344168+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:29:27.344168+0000","last_peered":"2024-03-16T19:29:27.344168+0000","last_clean":"2024-03-16T19:29:27.344168+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:29:27.344168+0000","last_undegraded":"2024-03-16T19:29:27.344168+0000","last_fullsized":"2024-03-16T19:29:27.344168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.211252+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:29:28.211252+0000","last_peered":"2024-03-16T19:29:28.211252+0000","last_clean":"2024-03-16T19:29:28.211252+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:29:28.211252+0000","last_undegraded":"2024-03-16T19:29:28.211252+0000","last_fullsized":"2024-03-16T19:29:28.211252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344198+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:29:27.344198+0000","last_peered":"2024-03-16T19:29:27.344198+0000","last_clean":"2024-03-16T19:29:27.344198+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:29:27.344198+0000","last_undegraded":"2024-03-16T19:29:27.344198+0000","last_fullsized":"2024-03-16T19:29:27.344198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.362435+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:29:31.362435+0000","last_peered":"2024-03-16T19:29:31.362435+0000","last_clean":"2024-03-16T19:29:31.362435+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:29:31.362435+0000","last_undegraded":"2024-03-16T19:29:31.362435+0000","last_fullsized":"2024-03-16T19:29:31.362435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344664+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:29:27.344664+0000","last_peered":"2024-03-16T19:29:27.344664+0000","last_clean":"2024-03-16T19:29:27.344664+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:29:27.344664+0000","last_undegraded":"2024-03-16T19:29:27.344664+0000","last_fullsized":"2024-03-16T19:29:27.344664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.219180+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:29:28.219180+0000","last_peered":"2024-03-16T19:29:28.219180+0000","last_clean":"2024-03-16T19:29:28.219180+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:29:28.219180+0000","last_undegraded":"2024-03-16T19:29:28.219180+0000","last_fullsized":"2024-03-16T19:29:28.219180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344128+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:29:27.344128+0000","last_peered":"2024-03-16T19:29:27.344128+0000","last_clean":"2024-03-16T19:29:27.344128+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:29:27.344128+0000","last_undegraded":"2024-03-16T19:29:27.344128+0000","last_fullsized":"2024-03-16T19:29:27.344128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062516+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:29:28.062516+0000","last_peered":"2024-03-16T19:29:28.062516+0000","last_clean":"2024-03-16T19:29:28.062516+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:29:28.062516+0000","last_undegraded":"2024-03-16T19:29:28.062516+0000","last_fullsized":"2024-03-16T19:29:28.062516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344351+0000","last_change":"2024-03-16T19:29:26.364742+0000","last_active":"2024-03-16T19:29:27.344351+0000","last_peered":"2024-03-16T19:29:27.344351+0000","last_clean":"2024-03-16T19:29:27.344351+0000","last_became_active":"2024-03-16T19:29:26.364328+0000","last_became_peered":"2024-03-16T19:29:26.364328+0000","last_unstale":"2024-03-16T19:29:27.344351+0000","last_undegraded":"2024-03-16T19:29:27.344351+0000","last_fullsized":"2024-03-16T19:29:27.344351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:18.877774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343226+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:29:27.343226+0000","last_peered":"2024-03-16T19:29:27.343226+0000","last_clean":"2024-03-16T19:29:27.343226+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:29:27.343226+0000","last_undegraded":"2024-03-16T19:29:27.343226+0000","last_fullsized":"2024-03-16T19:29:27.343226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561997+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:29:27.561997+0000","last_peered":"2024-03-16T19:29:27.561997+0000","last_clean":"2024-03-16T19:29:27.561997+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:29:27.561997+0000","last_undegraded":"2024-03-16T19:29:27.561997+0000","last_fullsized":"2024-03-16T19:29:27.561997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343035+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:29:27.343035+0000","last_peered":"2024-03-16T19:29:27.343035+0000","last_clean":"2024-03-16T19:29:27.343035+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:29:27.343035+0000","last_undegraded":"2024-03-16T19:29:27.343035+0000","last_fullsized":"2024-03-16T19:29:27.343035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062305+0000","last_change":"2024-03-16T19:29:26.352692+0000","last_active":"2024-03-16T19:29:28.062305+0000","last_peered":"2024-03-16T19:29:28.062305+0000","last_clean":"2024-03-16T19:29:28.062305+0000","last_became_active":"2024-03-16T19:29:26.352419+0000","last_became_peered":"2024-03-16T19:29:26.352419+0000","last_unstale":"2024-03-16T19:29:28.062305+0000","last_undegraded":"2024-03-16T19:29:28.062305+0000","last_fullsized":"2024-03-16T19:29:28.062305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:33.056538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062749+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:29:28.062749+0000","last_peered":"2024-03-16T19:29:28.062749+0000","last_clean":"2024-03-16T19:29:28.062749+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:29:28.062749+0000","last_undegraded":"2024-03-16T19:29:28.062749+0000","last_fullsized":"2024-03-16T19:29:28.062749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339393+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:29:27.339393+0000","last_peered":"2024-03-16T19:29:27.339393+0000","last_clean":"2024-03-16T19:29:27.339393+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:29:27.339393+0000","last_undegraded":"2024-03-16T19:29:27.339393+0000","last_fullsized":"2024-03-16T19:29:27.339393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344073+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:29:27.344073+0000","last_peered":"2024-03-16T19:29:27.344073+0000","last_clean":"2024-03-16T19:29:27.344073+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:29:27.344073+0000","last_undegraded":"2024-03-16T19:29:27.344073+0000","last_fullsized":"2024-03-16T19:29:27.344073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063637+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:29:28.063637+0000","last_peered":"2024-03-16T19:29:28.063637+0000","last_clean":"2024-03-16T19:29:28.063637+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:29:28.063637+0000","last_undegraded":"2024-03-16T19:29:28.063637+0000","last_fullsized":"2024-03-16T19:29:28.063637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339499+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:29:27.339499+0000","last_peered":"2024-03-16T19:29:27.339499+0000","last_clean":"2024-03-16T19:29:27.339499+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:29:27.339499+0000","last_undegraded":"2024-03-16T19:29:27.339499+0000","last_fullsized":"2024-03-16T19:29:27.339499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343975+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:29:27.343975+0000","last_peered":"2024-03-16T19:29:27.343975+0000","last_clean":"2024-03-16T19:29:27.343975+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:29:27.343975+0000","last_undegraded":"2024-03-16T19:29:27.343975+0000","last_fullsized":"2024-03-16T19:29:27.343975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344573+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:29:27.344573+0000","last_peered":"2024-03-16T19:29:27.344573+0000","last_clean":"2024-03-16T19:29:27.344573+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:29:27.344573+0000","last_undegraded":"2024-03-16T19:29:27.344573+0000","last_fullsized":"2024-03-16T19:29:27.344573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339540+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:29:27.339540+0000","last_peered":"2024-03-16T19:29:27.339540+0000","last_clean":"2024-03-16T19:29:27.339540+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:29:27.339540+0000","last_undegraded":"2024-03-16T19:29:27.339540+0000","last_fullsized":"2024-03-16T19:29:27.339540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339712+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:29:27.339712+0000","last_peered":"2024-03-16T19:29:27.339712+0000","last_clean":"2024-03-16T19:29:27.339712+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:29:27.339712+0000","last_undegraded":"2024-03-16T19:29:27.339712+0000","last_fullsized":"2024-03-16T19:29:27.339712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562225+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:29:27.562225+0000","last_peered":"2024-03-16T19:29:27.562225+0000","last_clean":"2024-03-16T19:29:27.562225+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:29:27.562225+0000","last_undegraded":"2024-03-16T19:29:27.562225+0000","last_fullsized":"2024-03-16T19:29:27.562225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.360976+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:29:31.360976+0000","last_peered":"2024-03-16T19:29:31.360976+0000","last_clean":"2024-03-16T19:29:31.360976+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:29:31.360976+0000","last_undegraded":"2024-03-16T19:29:31.360976+0000","last_fullsized":"2024-03-16T19:29:31.360976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062725+0000","last_change":"2024-03-16T19:29:26.350351+0000","last_active":"2024-03-16T19:29:28.062725+0000","last_peered":"2024-03-16T19:29:28.062725+0000","last_clean":"2024-03-16T19:29:28.062725+0000","last_became_active":"2024-03-16T19:29:26.350074+0000","last_became_peered":"2024-03-16T19:29:26.350074+0000","last_unstale":"2024-03-16T19:29:28.062725+0000","last_undegraded":"2024-03-16T19:29:28.062725+0000","last_fullsized":"2024-03-16T19:29:28.062725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:49.416395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.342931+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:29:27.342931+0000","last_peered":"2024-03-16T19:29:27.342931+0000","last_clean":"2024-03-16T19:29:27.342931+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:29:27.342931+0000","last_undegraded":"2024-03-16T19:29:27.342931+0000","last_fullsized":"2024-03-16T19:29:27.342931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063379+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:29:28.063379+0000","last_peered":"2024-03-16T19:29:28.063379+0000","last_clean":"2024-03-16T19:29:28.063379+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:29:28.063379+0000","last_undegraded":"2024-03-16T19:29:28.063379+0000","last_fullsized":"2024-03-16T19:29:28.063379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339690+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:29:27.339690+0000","last_peered":"2024-03-16T19:29:27.339690+0000","last_clean":"2024-03-16T19:29:27.339690+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:29:27.339690+0000","last_undegraded":"2024-03-16T19:29:27.339690+0000","last_fullsized":"2024-03-16T19:29:27.339690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561913+0000","last_change":"2024-03-16T19:29:26.347388+0000","last_active":"2024-03-16T19:29:27.561913+0000","last_peered":"2024-03-16T19:29:27.561913+0000","last_clean":"2024-03-16T19:29:27.561913+0000","last_became_active":"2024-03-16T19:29:26.346582+0000","last_became_peered":"2024-03-16T19:29:26.346582+0000","last_unstale":"2024-03-16T19:29:27.561913+0000","last_undegraded":"2024-03-16T19:29:27.561913+0000","last_fullsized":"2024-03-16T19:29:27.561913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:05.294895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339497+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:29:27.339497+0000","last_peered":"2024-03-16T19:29:27.339497+0000","last_clean":"2024-03-16T19:29:27.339497+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:29:27.339497+0000","last_undegraded":"2024-03-16T19:29:27.339497+0000","last_fullsized":"2024-03-16T19:29:27.339497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561968+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:29:27.561968+0000","last_peered":"2024-03-16T19:29:27.561968+0000","last_clean":"2024-03-16T19:29:27.561968+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:29:27.561968+0000","last_undegraded":"2024-03-16T19:29:27.561968+0000","last_fullsized":"2024-03-16T19:29:27.561968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365383+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:29:31.365383+0000","last_peered":"2024-03-16T19:29:31.365383+0000","last_clean":"2024-03-16T19:29:31.365383+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:29:31.365383+0000","last_undegraded":"2024-03-16T19:29:31.365383+0000","last_fullsized":"2024-03-16T19:29:31.365383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062837+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:29:28.062837+0000","last_peered":"2024-03-16T19:29:28.062837+0000","last_clean":"2024-03-16T19:29:28.062837+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:29:28.062837+0000","last_undegraded":"2024-03-16T19:29:28.062837+0000","last_fullsized":"2024-03-16T19:29:28.062837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343809+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:29:27.343809+0000","last_peered":"2024-03-16T19:29:27.343809+0000","last_clean":"2024-03-16T19:29:27.343809+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:29:27.343809+0000","last_undegraded":"2024-03-16T19:29:27.343809+0000","last_fullsized":"2024-03-16T19:29:27.343809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.360603+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:29:31.360603+0000","last_peered":"2024-03-16T19:29:31.360603+0000","last_clean":"2024-03-16T19:29:31.360603+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:29:31.360603+0000","last_undegraded":"2024-03-16T19:29:31.360603+0000","last_fullsized":"2024-03-16T19:29:31.360603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344208+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:29:27.344208+0000","last_peered":"2024-03-16T19:29:27.344208+0000","last_clean":"2024-03-16T19:29:27.344208+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:29:27.344208+0000","last_undegraded":"2024-03-16T19:29:27.344208+0000","last_fullsized":"2024-03-16T19:29:27.344208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365202+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:29:31.365202+0000","last_peered":"2024-03-16T19:29:31.365202+0000","last_clean":"2024-03-16T19:29:31.365202+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:29:31.365202+0000","last_undegraded":"2024-03-16T19:29:31.365202+0000","last_fullsized":"2024-03-16T19:29:31.365202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062891+0000","last_change":"2024-03-16T19:29:26.354303+0000","last_active":"2024-03-16T19:29:28.062891+0000","last_peered":"2024-03-16T19:29:28.062891+0000","last_clean":"2024-03-16T19:29:28.062891+0000","last_became_active":"2024-03-16T19:29:26.353804+0000","last_became_peered":"2024-03-16T19:29:26.353804+0000","last_unstale":"2024-03-16T19:29:28.062891+0000","last_undegraded":"2024-03-16T19:29:28.062891+0000","last_fullsized":"2024-03-16T19:29:28.062891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:10:45.368536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.215673+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:29:28.215673+0000","last_peered":"2024-03-16T19:29:28.215673+0000","last_clean":"2024-03-16T19:29:28.215673+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:29:28.215673+0000","last_undegraded":"2024-03-16T19:29:28.215673+0000","last_fullsized":"2024-03-16T19:29:28.215673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562176+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:29:27.562176+0000","last_peered":"2024-03-16T19:29:27.562176+0000","last_clean":"2024-03-16T19:29:27.562176+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:29:27.562176+0000","last_undegraded":"2024-03-16T19:29:27.562176+0000","last_fullsized":"2024-03-16T19:29:27.562176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365525+0000","last_change":"2024-03-16T19:29:25.342597+0000","last_active":"2024-03-16T19:29:31.365525+0000","last_peered":"2024-03-16T19:29:31.365525+0000","last_clean":"2024-03-16T19:29:31.365525+0000","last_became_active":"2024-03-16T19:29:25.342320+0000","last_became_peered":"2024-03-16T19:29:25.342320+0000","last_unstale":"2024-03-16T19:29:31.365525+0000","last_undegraded":"2024-03-16T19:29:31.365525+0000","last_fullsized":"2024-03-16T19:29:31.365525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:44:41.274396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":665973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:29:32.959 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:29:33.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.115+0000 7f41dfbd7640 1 Processor -- start 2024-03-16T19:29:33.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.116+0000 7f41dfbd7640 1 -- start start 2024-03-16T19:29:33.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.116+0000 7f41dfbd7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f41d8154c10 0x7f41d80c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.116+0000 7f41dfbd7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d8154ff0 con 0x7f41d806b2e0 2024-03-16T19:29:33.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.116+0000 7f41dfbd7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d806b980 con 0x7f41d806aed0 2024-03-16T19:29:33.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.116+0000 7f41dfbd7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d815e010 con 0x7f41d8154c10 2024-03-16T19:29:33.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.117+0000 7f41de174640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f41d806b2e0 0x7f41d815e430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56232/0 (socket says 172.21.15.172:56232) 2024-03-16T19:29:33.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.117+0000 7f41de174640 1 -- 172.21.15.172:0/1844339128 learned_addr learned my addr 172.21.15.172:0/1844339128 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:33.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.117+0000 7f41dd973640 1 --2- 172.21.15.172:0/1844339128 >> v2:172.21.15.174:3300/0 conn(0x7f41d8154c10 0x7f41d80c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.117+0000 7f41dc971640 1 -- 172.21.15.172:0/1844339128 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956751080 0 0) 0x7f41d0001490 con 0x7f41d806b2e0 2024-03-16T19:29:33.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.117+0000 7f41dc971640 1 -- 172.21.15.172:0/1844339128 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c4003520 con 0x7f41d806b2e0 2024-03-16T19:29:33.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.117+0000 7f41dd973640 1 -- 172.21.15.172:0/1844339128 >> v1:172.21.15.174:6789/0 conn(0x7f41d806aed0 legacy=0x7f41d8071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.117+0000 7f41dd973640 1 -- 172.21.15.172:0/1844339128 >> v1:172.21.15.172:6789/0 conn(0x7f41d806b2e0 legacy=0x7f41d815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.117+0000 7f41dd973640 1 -- 172.21.15.172:0/1844339128 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d806bc20 con 0x7f41d8154c10 2024-03-16T19:29:33.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dd973640 1 --2- 172.21.15.172:0/1844339128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8154c10 0x7f41d80c7a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f41c80097a0 tx=0x7f41c802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c71915636335346 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:33.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dc971640 1 -- 172.21.15.172:0/1844339128 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c802f810 con 0x7f41d8154c10 2024-03-16T19:29:33.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dc971640 1 -- 172.21.15.172:0/1844339128 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c802f970 con 0x7f41d8154c10 2024-03-16T19:29:33.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dc971640 1 -- 172.21.15.172:0/1844339128 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c802fce0 con 0x7f41d8154c10 2024-03-16T19:29:33.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dfbd7640 1 -- 172.21.15.172:0/1844339128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8154c10 msgr2=0x7f41d80c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dfbd7640 1 --2- 172.21.15.172:0/1844339128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8154c10 0x7f41d80c7a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f41c80097a0 tx=0x7f41c802b9f0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dfbd7640 1 -- 172.21.15.172:0/1844339128 shutdown_connections 2024-03-16T19:29:33.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dfbd7640 1 --2- 172.21.15.172:0/1844339128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8154c10 0x7f41d80c7a10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dfbd7640 1 -- 172.21.15.172:0/1844339128 >> 172.21.15.172:0/1844339128 conn(0x7f41d8072b40 msgr2=0x7f41d8072f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:33.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dfbd7640 1 -- 172.21.15.172:0/1844339128 shutdown_connections 2024-03-16T19:29:33.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.118+0000 7f41dfbd7640 1 -- 172.21.15.172:0/1844339128 wait complete. 2024-03-16T19:29:33.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dfbd7640 1 Processor -- start 2024-03-16T19:29:33.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dfbd7640 1 -- start start 2024-03-16T19:29:33.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dfbd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d806aed0 0x7f41d80cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dfbd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d806b2e0 0x7f41d80cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dfbd7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d8154c10 0x7f41d815d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dfbd7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d814c980 con 0x7f41d8154c10 2024-03-16T19:29:33.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dfbd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d814e270 con 0x7f41d806b2e0 2024-03-16T19:29:33.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dfbd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d81474f0 con 0x7f41d806aed0 2024-03-16T19:29:33.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41de174640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d8154c10 0x7f41d815d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41de174640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d8154c10 0x7f41d815d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46416/0 (socket says 172.21.15.172:46416) 2024-03-16T19:29:33.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41de174640 1 -- 172.21.15.172:0/3683507397 learned_addr learned my addr 172.21.15.172:0/3683507397 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:33.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dd973640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d806aed0 0x7f41d80cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.120+0000 7f41dd172640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d806b2e0 0x7f41d80cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41de174640 1 -- 172.21.15.172:0/3683507397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d806aed0 msgr2=0x7f41d80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41de174640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d806aed0 0x7f41d80cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41de174640 1 -- 172.21.15.172:0/3683507397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d806b2e0 msgr2=0x7f41d80cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41de174640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d806b2e0 0x7f41d80cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41de174640 1 -- 172.21.15.172:0/3683507397 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c8009400 con 0x7f41d8154c10 2024-03-16T19:29:33.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41de174640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d8154c10 0x7f41d815d060 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41d000cb40 tx=0x7f41d000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:33.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41ce7fc640 1 -- 172.21.15.172:0/3683507397 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d000aca0 con 0x7f41d8154c10 2024-03-16T19:29:33.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41ce7fc640 1 -- 172.21.15.172:0/3683507397 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d000ae00 con 0x7f41d8154c10 2024-03-16T19:29:33.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d814e000 con 0x7f41d8154c10 2024-03-16T19:29:33.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41ce7fc640 1 -- 172.21.15.172:0/3683507397 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0012620 con 0x7f41d8154c10 2024-03-16T19:29:33.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.121+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d81470d0 con 0x7f41d8154c10 2024-03-16T19:29:33.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.122+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d806ce30 con 0x7f41d8154c10 2024-03-16T19:29:33.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.122+0000 7f41ce7fc640 1 -- 172.21.15.172:0/3683507397 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f41d0007070 con 0x7f41d8154c10 2024-03-16T19:29:33.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.123+0000 7f41ce7fc640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.123+0000 7f41ce7fc640 1 -- 172.21.15.172:0/3683507397 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41d00899f0 con 0x7f41d8154c10 2024-03-16T19:29:33.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.123+0000 7f41dd973640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.123+0000 7f41dd973640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f41c8036000 tx=0x7f41c8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:33.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.127+0000 7f41ce7fc640 1 -- 172.21.15.172:0/3683507397 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41d0054700 con 0x7f41d8154c10 2024-03-16T19:29:33.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.295+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41d806c1e0 con 0x7f41bc06b3e0 2024-03-16T19:29:33.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.301+0000 7f41ce7fc640 1 -- 172.21.15.172:0/3683507397 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251947 (secure 0 0 0) 0x7f41c802fb60 con 0x7f41bc06b3e0 2024-03-16T19:29:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:33.288 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:29:33.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.303+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 msgr2=0x7f41bc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.303+0000 7f41dfbd7640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f41c8036000 tx=0x7f41c8008030 comp rx=0 tx=0).stop 2024-03-16T19:29:33.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.303+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d8154c10 msgr2=0x7f41d815d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.303+0000 7f41dfbd7640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d8154c10 0x7f41d815d060 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41d000cb40 tx=0x7f41d000a060 comp rx=0 tx=0).stop 2024-03-16T19:29:33.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.304+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 shutdown_connections 2024-03-16T19:29:33.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.304+0000 7f41dfbd7640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d8154c10 0x7f41d815d060 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.304+0000 7f41dfbd7640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d806b2e0 0x7f41d80cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.304+0000 7f41dfbd7640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.304+0000 7f41dfbd7640 1 --2- 172.21.15.172:0/3683507397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d806aed0 0x7f41d80cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.304+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 >> 172.21.15.172:0/3683507397 conn(0x7f41d8072b40 msgr2=0x7f41d8159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:33.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.304+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 shutdown_connections 2024-03-16T19:29:33.293 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.304+0000 7f41dfbd7640 1 -- 172.21.15.172:0/3683507397 wait complete. 2024-03-16T19:29:33.295 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-16T19:29:32.200894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219040,"kb_used_data":2896,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755680,"statfs":{"total":773094113280,"available":772869816320,"internally_reserved":0,"allocated":2965504,"data_stored":1788166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.876361"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344501+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:29:27.344501+0000","last_peered":"2024-03-16T19:29:27.344501+0000","last_clean":"2024-03-16T19:29:27.344501+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:29:27.344501+0000","last_undegraded":"2024-03-16T19:29:27.344501+0000","last_fullsized":"2024-03-16T19:29:27.344501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339631+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:29:27.339631+0000","last_peered":"2024-03-16T19:29:27.339631+0000","last_clean":"2024-03-16T19:29:27.339631+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:29:27.339631+0000","last_undegraded":"2024-03-16T19:29:27.339631+0000","last_fullsized":"2024-03-16T19:29:27.339631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062822+0000","last_change":"2024-03-16T19:29:26.347979+0000","last_active":"2024-03-16T19:29:28.062822+0000","last_peered":"2024-03-16T19:29:28.062822+0000","last_clean":"2024-03-16T19:29:28.062822+0000","last_became_active":"2024-03-16T19:29:26.347711+0000","last_became_peered":"2024-03-16T19:29:26.347711+0000","last_unstale":"2024-03-16T19:29:28.062822+0000","last_undegraded":"2024-03-16T19:29:28.062822+0000","last_fullsized":"2024-03-16T19:29:28.062822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:08.017182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343558+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:29:27.343558+0000","last_peered":"2024-03-16T19:29:27.343558+0000","last_clean":"2024-03-16T19:29:27.343558+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:29:27.343558+0000","last_undegraded":"2024-03-16T19:29:27.343558+0000","last_fullsized":"2024-03-16T19:29:27.343558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344270+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:29:27.344270+0000","last_peered":"2024-03-16T19:29:27.344270+0000","last_clean":"2024-03-16T19:29:27.344270+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:29:27.344270+0000","last_undegraded":"2024-03-16T19:29:27.344270+0000","last_fullsized":"2024-03-16T19:29:27.344270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343104+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:29:27.343104+0000","last_peered":"2024-03-16T19:29:27.343104+0000","last_clean":"2024-03-16T19:29:27.343104+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:29:27.343104+0000","last_undegraded":"2024-03-16T19:29:27.343104+0000","last_fullsized":"2024-03-16T19:29:27.343104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344269+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:29:27.344269+0000","last_peered":"2024-03-16T19:29:27.344269+0000","last_clean":"2024-03-16T19:29:27.344269+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:29:27.344269+0000","last_undegraded":"2024-03-16T19:29:27.344269+0000","last_fullsized":"2024-03-16T19:29:27.344269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561737+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:29:27.561737+0000","last_peered":"2024-03-16T19:29:27.561737+0000","last_clean":"2024-03-16T19:29:27.561737+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:29:27.561737+0000","last_undegraded":"2024-03-16T19:29:27.561737+0000","last_fullsized":"2024-03-16T19:29:27.561737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344497+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:29:27.344497+0000","last_peered":"2024-03-16T19:29:27.344497+0000","last_clean":"2024-03-16T19:29:27.344497+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:29:27.344497+0000","last_undegraded":"2024-03-16T19:29:27.344497+0000","last_fullsized":"2024-03-16T19:29:27.344497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063286+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:29:28.063286+0000","last_peered":"2024-03-16T19:29:28.063286+0000","last_clean":"2024-03-16T19:29:28.063286+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:29:28.063286+0000","last_undegraded":"2024-03-16T19:29:28.063286+0000","last_fullsized":"2024-03-16T19:29:28.063286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062996+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:29:28.062996+0000","last_peered":"2024-03-16T19:29:28.062996+0000","last_clean":"2024-03-16T19:29:28.062996+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:29:28.062996+0000","last_undegraded":"2024-03-16T19:29:28.062996+0000","last_fullsized":"2024-03-16T19:29:28.062996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063177+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:29:28.063177+0000","last_peered":"2024-03-16T19:29:28.063177+0000","last_clean":"2024-03-16T19:29:28.063177+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:29:28.063177+0000","last_undegraded":"2024-03-16T19:29:28.063177+0000","last_fullsized":"2024-03-16T19:29:28.063177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561930+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:29:27.561930+0000","last_peered":"2024-03-16T19:29:27.561930+0000","last_clean":"2024-03-16T19:29:27.561930+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:29:27.561930+0000","last_undegraded":"2024-03-16T19:29:27.561930+0000","last_fullsized":"2024-03-16T19:29:27.561930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062462+0000","last_change":"2024-03-16T19:29:26.349644+0000","last_active":"2024-03-16T19:29:28.062462+0000","last_peered":"2024-03-16T19:29:28.062462+0000","last_clean":"2024-03-16T19:29:28.062462+0000","last_became_active":"2024-03-16T19:29:26.349397+0000","last_became_peered":"2024-03-16T19:29:26.349397+0000","last_unstale":"2024-03-16T19:29:28.062462+0000","last_undegraded":"2024-03-16T19:29:28.062462+0000","last_fullsized":"2024-03-16T19:29:28.062462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:47:20.566393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339397+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:29:27.339397+0000","last_peered":"2024-03-16T19:29:27.339397+0000","last_clean":"2024-03-16T19:29:27.339397+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:29:27.339397+0000","last_undegraded":"2024-03-16T19:29:27.339397+0000","last_fullsized":"2024-03-16T19:29:27.339397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344311+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:29:27.344311+0000","last_peered":"2024-03-16T19:29:27.344311+0000","last_clean":"2024-03-16T19:29:27.344311+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:29:27.344311+0000","last_undegraded":"2024-03-16T19:29:27.344311+0000","last_fullsized":"2024-03-16T19:29:27.344311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062569+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:29:28.062569+0000","last_peered":"2024-03-16T19:29:28.062569+0000","last_clean":"2024-03-16T19:29:28.062569+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:29:28.062569+0000","last_undegraded":"2024-03-16T19:29:28.062569+0000","last_fullsized":"2024-03-16T19:29:28.062569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344648+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:29:27.344648+0000","last_peered":"2024-03-16T19:29:27.344648+0000","last_clean":"2024-03-16T19:29:27.344648+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:29:27.344648+0000","last_undegraded":"2024-03-16T19:29:27.344648+0000","last_fullsized":"2024-03-16T19:29:27.344648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063112+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:29:28.063112+0000","last_peered":"2024-03-16T19:29:28.063112+0000","last_clean":"2024-03-16T19:29:28.063112+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:29:28.063112+0000","last_undegraded":"2024-03-16T19:29:28.063112+0000","last_fullsized":"2024-03-16T19:29:28.063112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062498+0000","last_change":"2024-03-16T19:29:26.357191+0000","last_active":"2024-03-16T19:29:28.062498+0000","last_peered":"2024-03-16T19:29:28.062498+0000","last_clean":"2024-03-16T19:29:28.062498+0000","last_became_active":"2024-03-16T19:29:26.356947+0000","last_became_peered":"2024-03-16T19:29:26.356947+0000","last_unstale":"2024-03-16T19:29:28.062498+0000","last_undegraded":"2024-03-16T19:29:28.062498+0000","last_fullsized":"2024-03-16T19:29:28.062498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:40.884164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343265+0000","last_change":"2024-03-16T19:29:26.358307+0000","last_active":"2024-03-16T19:29:27.343265+0000","last_peered":"2024-03-16T19:29:27.343265+0000","last_clean":"2024-03-16T19:29:27.343265+0000","last_became_active":"2024-03-16T19:29:26.357970+0000","last_became_peered":"2024-03-16T19:29:26.357970+0000","last_unstale":"2024-03-16T19:29:27.343265+0000","last_undegraded":"2024-03-16T19:29:27.343265+0000","last_fullsized":"2024-03-16T19:29:27.343265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:33.976868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561749+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:29:27.561749+0000","last_peered":"2024-03-16T19:29:27.561749+0000","last_clean":"2024-03-16T19:29:27.561749+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:29:27.561749+0000","last_undegraded":"2024-03-16T19:29:27.561749+0000","last_fullsized":"2024-03-16T19:29:27.561749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343286+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:29:27.343286+0000","last_peered":"2024-03-16T19:29:27.343286+0000","last_clean":"2024-03-16T19:29:27.343286+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:29:27.343286+0000","last_undegraded":"2024-03-16T19:29:27.343286+0000","last_fullsized":"2024-03-16T19:29:27.343286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562192+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:29:27.562192+0000","last_peered":"2024-03-16T19:29:27.562192+0000","last_clean":"2024-03-16T19:29:27.562192+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:29:27.562192+0000","last_undegraded":"2024-03-16T19:29:27.562192+0000","last_fullsized":"2024-03-16T19:29:27.562192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063443+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:29:28.063443+0000","last_peered":"2024-03-16T19:29:28.063443+0000","last_clean":"2024-03-16T19:29:28.063443+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:29:28.063443+0000","last_undegraded":"2024-03-16T19:29:28.063443+0000","last_fullsized":"2024-03-16T19:29:28.063443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339564+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:29:27.339564+0000","last_peered":"2024-03-16T19:29:27.339564+0000","last_clean":"2024-03-16T19:29:27.339564+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:29:27.339564+0000","last_undegraded":"2024-03-16T19:29:27.339564+0000","last_fullsized":"2024-03-16T19:29:27.339564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562214+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:29:27.562214+0000","last_peered":"2024-03-16T19:29:27.562214+0000","last_clean":"2024-03-16T19:29:27.562214+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:29:27.562214+0000","last_undegraded":"2024-03-16T19:29:27.562214+0000","last_fullsized":"2024-03-16T19:29:27.562214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062637+0000","last_change":"2024-03-16T19:29:26.349443+0000","last_active":"2024-03-16T19:29:28.062637+0000","last_peered":"2024-03-16T19:29:28.062637+0000","last_clean":"2024-03-16T19:29:28.062637+0000","last_became_active":"2024-03-16T19:29:26.349181+0000","last_became_peered":"2024-03-16T19:29:26.349181+0000","last_unstale":"2024-03-16T19:29:28.062637+0000","last_undegraded":"2024-03-16T19:29:28.062637+0000","last_fullsized":"2024-03-16T19:29:28.062637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:16.483346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343433+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:29:27.343433+0000","last_peered":"2024-03-16T19:29:27.343433+0000","last_clean":"2024-03-16T19:29:27.343433+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:29:27.343433+0000","last_undegraded":"2024-03-16T19:29:27.343433+0000","last_fullsized":"2024-03-16T19:29:27.343433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062168+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:29:28.062168+0000","last_peered":"2024-03-16T19:29:28.062168+0000","last_clean":"2024-03-16T19:29:28.062168+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:29:28.062168+0000","last_undegraded":"2024-03-16T19:29:28.062168+0000","last_fullsized":"2024-03-16T19:29:28.062168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062728+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:29:28.062728+0000","last_peered":"2024-03-16T19:29:28.062728+0000","last_clean":"2024-03-16T19:29:28.062728+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:29:28.062728+0000","last_undegraded":"2024-03-16T19:29:28.062728+0000","last_fullsized":"2024-03-16T19:29:28.062728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063484+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:29:28.063484+0000","last_peered":"2024-03-16T19:29:28.063484+0000","last_clean":"2024-03-16T19:29:28.063484+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:29:28.063484+0000","last_undegraded":"2024-03-16T19:29:28.063484+0000","last_fullsized":"2024-03-16T19:29:28.063484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.215645+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:29:28.215645+0000","last_peered":"2024-03-16T19:29:28.215645+0000","last_clean":"2024-03-16T19:29:28.215645+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:29:28.215645+0000","last_undegraded":"2024-03-16T19:29:28.215645+0000","last_fullsized":"2024-03-16T19:29:28.215645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062765+0000","last_change":"2024-03-16T19:29:26.353288+0000","last_active":"2024-03-16T19:29:28.062765+0000","last_peered":"2024-03-16T19:29:28.062765+0000","last_clean":"2024-03-16T19:29:28.062765+0000","last_became_active":"2024-03-16T19:29:26.352973+0000","last_became_peered":"2024-03-16T19:29:26.352973+0000","last_unstale":"2024-03-16T19:29:28.062765+0000","last_undegraded":"2024-03-16T19:29:28.062765+0000","last_fullsized":"2024-03-16T19:29:28.062765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:22:41.505003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.210595+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:29:28.210595+0000","last_peered":"2024-03-16T19:29:28.210595+0000","last_clean":"2024-03-16T19:29:28.210595+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:29:28.210595+0000","last_undegraded":"2024-03-16T19:29:28.210595+0000","last_fullsized":"2024-03-16T19:29:28.210595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561856+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:29:27.561856+0000","last_peered":"2024-03-16T19:29:27.561856+0000","last_clean":"2024-03-16T19:29:27.561856+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:29:27.561856+0000","last_undegraded":"2024-03-16T19:29:27.561856+0000","last_fullsized":"2024-03-16T19:29:27.561856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561778+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:29:27.561778+0000","last_peered":"2024-03-16T19:29:27.561778+0000","last_clean":"2024-03-16T19:29:27.561778+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:29:27.561778+0000","last_undegraded":"2024-03-16T19:29:27.561778+0000","last_fullsized":"2024-03-16T19:29:27.561778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343316+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:29:27.343316+0000","last_peered":"2024-03-16T19:29:27.343316+0000","last_clean":"2024-03-16T19:29:27.343316+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:29:27.343316+0000","last_undegraded":"2024-03-16T19:29:27.343316+0000","last_fullsized":"2024-03-16T19:29:27.343316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339401+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:29:27.339401+0000","last_peered":"2024-03-16T19:29:27.339401+0000","last_clean":"2024-03-16T19:29:27.339401+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:29:27.339401+0000","last_undegraded":"2024-03-16T19:29:27.339401+0000","last_fullsized":"2024-03-16T19:29:27.339401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063312+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:29:28.063312+0000","last_peered":"2024-03-16T19:29:28.063312+0000","last_clean":"2024-03-16T19:29:28.063312+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:29:28.063312+0000","last_undegraded":"2024-03-16T19:29:28.063312+0000","last_fullsized":"2024-03-16T19:29:28.063312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344324+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:29:27.344324+0000","last_peered":"2024-03-16T19:29:27.344324+0000","last_clean":"2024-03-16T19:29:27.344324+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:29:27.344324+0000","last_undegraded":"2024-03-16T19:29:27.344324+0000","last_fullsized":"2024-03-16T19:29:27.344324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343744+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:29:27.343744+0000","last_peered":"2024-03-16T19:29:27.343744+0000","last_clean":"2024-03-16T19:29:27.343744+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:29:27.343744+0000","last_undegraded":"2024-03-16T19:29:27.343744+0000","last_fullsized":"2024-03-16T19:29:27.343744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062922+0000","last_change":"2024-03-16T19:29:26.353918+0000","last_active":"2024-03-16T19:29:28.062922+0000","last_peered":"2024-03-16T19:29:28.062922+0000","last_clean":"2024-03-16T19:29:28.062922+0000","last_became_active":"2024-03-16T19:29:26.353463+0000","last_became_peered":"2024-03-16T19:29:26.353463+0000","last_unstale":"2024-03-16T19:29:28.062922+0000","last_undegraded":"2024-03-16T19:29:28.062922+0000","last_fullsized":"2024-03-16T19:29:28.062922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:36.035692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.362815+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:29:31.362815+0000","last_peered":"2024-03-16T19:29:31.362815+0000","last_clean":"2024-03-16T19:29:31.362815+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:29:31.362815+0000","last_undegraded":"2024-03-16T19:29:31.362815+0000","last_fullsized":"2024-03-16T19:29:31.362815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063360+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:29:28.063360+0000","last_peered":"2024-03-16T19:29:28.063360+0000","last_clean":"2024-03-16T19:29:28.063360+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:29:28.063360+0000","last_undegraded":"2024-03-16T19:29:28.063360+0000","last_fullsized":"2024-03-16T19:29:28.063360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365125+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:29:31.365125+0000","last_peered":"2024-03-16T19:29:31.365125+0000","last_clean":"2024-03-16T19:29:31.365125+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:29:31.365125+0000","last_undegraded":"2024-03-16T19:29:31.365125+0000","last_fullsized":"2024-03-16T19:29:31.365125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344465+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:29:27.344465+0000","last_peered":"2024-03-16T19:29:27.344465+0000","last_clean":"2024-03-16T19:29:27.344465+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:29:27.344465+0000","last_undegraded":"2024-03-16T19:29:27.344465+0000","last_fullsized":"2024-03-16T19:29:27.344465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365001+0000","last_change":"2024-03-16T19:29:25.336296+0000","last_active":"2024-03-16T19:29:31.365001+0000","last_peered":"2024-03-16T19:29:31.365001+0000","last_clean":"2024-03-16T19:29:31.365001+0000","last_became_active":"2024-03-16T19:29:25.333046+0000","last_became_peered":"2024-03-16T19:29:25.333046+0000","last_unstale":"2024-03-16T19:29:31.365001+0000","last_undegraded":"2024-03-16T19:29:31.365001+0000","last_fullsized":"2024-03-16T19:29:31.365001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:33.538342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062507+0000","last_change":"2024-03-16T19:29:26.354709+0000","last_active":"2024-03-16T19:29:28.062507+0000","last_peered":"2024-03-16T19:29:28.062507+0000","last_clean":"2024-03-16T19:29:28.062507+0000","last_became_active":"2024-03-16T19:29:26.354377+0000","last_became_peered":"2024-03-16T19:29:26.354377+0000","last_unstale":"2024-03-16T19:29:28.062507+0000","last_undegraded":"2024-03-16T19:29:28.062507+0000","last_fullsized":"2024-03-16T19:29:28.062507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:41:56.277270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343553+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:29:27.343553+0000","last_peered":"2024-03-16T19:29:27.343553+0000","last_clean":"2024-03-16T19:29:27.343553+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:29:27.343553+0000","last_undegraded":"2024-03-16T19:29:27.343553+0000","last_fullsized":"2024-03-16T19:29:27.343553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344364+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:29:27.344364+0000","last_peered":"2024-03-16T19:29:27.344364+0000","last_clean":"2024-03-16T19:29:27.344364+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:29:27.344364+0000","last_undegraded":"2024-03-16T19:29:27.344364+0000","last_fullsized":"2024-03-16T19:29:27.344364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062737+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:29:28.062737+0000","last_peered":"2024-03-16T19:29:28.062737+0000","last_clean":"2024-03-16T19:29:28.062737+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:29:28.062737+0000","last_undegraded":"2024-03-16T19:29:28.062737+0000","last_fullsized":"2024-03-16T19:29:28.062737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062954+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:29:28.062954+0000","last_peered":"2024-03-16T19:29:28.062954+0000","last_clean":"2024-03-16T19:29:28.062954+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:29:28.062954+0000","last_undegraded":"2024-03-16T19:29:28.062954+0000","last_fullsized":"2024-03-16T19:29:28.062954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365607+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:29:31.365607+0000","last_peered":"2024-03-16T19:29:31.365607+0000","last_clean":"2024-03-16T19:29:31.365607+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:29:31.365607+0000","last_undegraded":"2024-03-16T19:29:31.365607+0000","last_fullsized":"2024-03-16T19:29:31.365607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344168+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:29:27.344168+0000","last_peered":"2024-03-16T19:29:27.344168+0000","last_clean":"2024-03-16T19:29:27.344168+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:29:27.344168+0000","last_undegraded":"2024-03-16T19:29:27.344168+0000","last_fullsized":"2024-03-16T19:29:27.344168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.211252+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:29:28.211252+0000","last_peered":"2024-03-16T19:29:28.211252+0000","last_clean":"2024-03-16T19:29:28.211252+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:29:28.211252+0000","last_undegraded":"2024-03-16T19:29:28.211252+0000","last_fullsized":"2024-03-16T19:29:28.211252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344198+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:29:27.344198+0000","last_peered":"2024-03-16T19:29:27.344198+0000","last_clean":"2024-03-16T19:29:27.344198+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:29:27.344198+0000","last_undegraded":"2024-03-16T19:29:27.344198+0000","last_fullsized":"2024-03-16T19:29:27.344198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.362435+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:29:31.362435+0000","last_peered":"2024-03-16T19:29:31.362435+0000","last_clean":"2024-03-16T19:29:31.362435+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:29:31.362435+0000","last_undegraded":"2024-03-16T19:29:31.362435+0000","last_fullsized":"2024-03-16T19:29:31.362435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344664+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:29:27.344664+0000","last_peered":"2024-03-16T19:29:27.344664+0000","last_clean":"2024-03-16T19:29:27.344664+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:29:27.344664+0000","last_undegraded":"2024-03-16T19:29:27.344664+0000","last_fullsized":"2024-03-16T19:29:27.344664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.219180+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:29:28.219180+0000","last_peered":"2024-03-16T19:29:28.219180+0000","last_clean":"2024-03-16T19:29:28.219180+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:29:28.219180+0000","last_undegraded":"2024-03-16T19:29:28.219180+0000","last_fullsized":"2024-03-16T19:29:28.219180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344128+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:29:27.344128+0000","last_peered":"2024-03-16T19:29:27.344128+0000","last_clean":"2024-03-16T19:29:27.344128+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:29:27.344128+0000","last_undegraded":"2024-03-16T19:29:27.344128+0000","last_fullsized":"2024-03-16T19:29:27.344128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062516+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:29:28.062516+0000","last_peered":"2024-03-16T19:29:28.062516+0000","last_clean":"2024-03-16T19:29:28.062516+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:29:28.062516+0000","last_undegraded":"2024-03-16T19:29:28.062516+0000","last_fullsized":"2024-03-16T19:29:28.062516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344351+0000","last_change":"2024-03-16T19:29:26.364742+0000","last_active":"2024-03-16T19:29:27.344351+0000","last_peered":"2024-03-16T19:29:27.344351+0000","last_clean":"2024-03-16T19:29:27.344351+0000","last_became_active":"2024-03-16T19:29:26.364328+0000","last_became_peered":"2024-03-16T19:29:26.364328+0000","last_unstale":"2024-03-16T19:29:27.344351+0000","last_undegraded":"2024-03-16T19:29:27.344351+0000","last_fullsized":"2024-03-16T19:29:27.344351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:18.877774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343226+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:29:27.343226+0000","last_peered":"2024-03-16T19:29:27.343226+0000","last_clean":"2024-03-16T19:29:27.343226+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:29:27.343226+0000","last_undegraded":"2024-03-16T19:29:27.343226+0000","last_fullsized":"2024-03-16T19:29:27.343226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561997+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:29:27.561997+0000","last_peered":"2024-03-16T19:29:27.561997+0000","last_clean":"2024-03-16T19:29:27.561997+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:29:27.561997+0000","last_undegraded":"2024-03-16T19:29:27.561997+0000","last_fullsized":"2024-03-16T19:29:27.561997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343035+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:29:27.343035+0000","last_peered":"2024-03-16T19:29:27.343035+0000","last_clean":"2024-03-16T19:29:27.343035+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:29:27.343035+0000","last_undegraded":"2024-03-16T19:29:27.343035+0000","last_fullsized":"2024-03-16T19:29:27.343035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062305+0000","last_change":"2024-03-16T19:29:26.352692+0000","last_active":"2024-03-16T19:29:28.062305+0000","last_peered":"2024-03-16T19:29:28.062305+0000","last_clean":"2024-03-16T19:29:28.062305+0000","last_became_active":"2024-03-16T19:29:26.352419+0000","last_became_peered":"2024-03-16T19:29:26.352419+0000","last_unstale":"2024-03-16T19:29:28.062305+0000","last_undegraded":"2024-03-16T19:29:28.062305+0000","last_fullsized":"2024-03-16T19:29:28.062305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:33.056538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062749+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:29:28.062749+0000","last_peered":"2024-03-16T19:29:28.062749+0000","last_clean":"2024-03-16T19:29:28.062749+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:29:28.062749+0000","last_undegraded":"2024-03-16T19:29:28.062749+0000","last_fullsized":"2024-03-16T19:29:28.062749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339393+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:29:27.339393+0000","last_peered":"2024-03-16T19:29:27.339393+0000","last_clean":"2024-03-16T19:29:27.339393+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:29:27.339393+0000","last_undegraded":"2024-03-16T19:29:27.339393+0000","last_fullsized":"2024-03-16T19:29:27.339393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344073+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:29:27.344073+0000","last_peered":"2024-03-16T19:29:27.344073+0000","last_clean":"2024-03-16T19:29:27.344073+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:29:27.344073+0000","last_undegraded":"2024-03-16T19:29:27.344073+0000","last_fullsized":"2024-03-16T19:29:27.344073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063637+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:29:28.063637+0000","last_peered":"2024-03-16T19:29:28.063637+0000","last_clean":"2024-03-16T19:29:28.063637+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:29:28.063637+0000","last_undegraded":"2024-03-16T19:29:28.063637+0000","last_fullsized":"2024-03-16T19:29:28.063637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339499+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:29:27.339499+0000","last_peered":"2024-03-16T19:29:27.339499+0000","last_clean":"2024-03-16T19:29:27.339499+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:29:27.339499+0000","last_undegraded":"2024-03-16T19:29:27.339499+0000","last_fullsized":"2024-03-16T19:29:27.339499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343975+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:29:27.343975+0000","last_peered":"2024-03-16T19:29:27.343975+0000","last_clean":"2024-03-16T19:29:27.343975+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:29:27.343975+0000","last_undegraded":"2024-03-16T19:29:27.343975+0000","last_fullsized":"2024-03-16T19:29:27.343975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344573+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:29:27.344573+0000","last_peered":"2024-03-16T19:29:27.344573+0000","last_clean":"2024-03-16T19:29:27.344573+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:29:27.344573+0000","last_undegraded":"2024-03-16T19:29:27.344573+0000","last_fullsized":"2024-03-16T19:29:27.344573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339540+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:29:27.339540+0000","last_peered":"2024-03-16T19:29:27.339540+0000","last_clean":"2024-03-16T19:29:27.339540+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:29:27.339540+0000","last_undegraded":"2024-03-16T19:29:27.339540+0000","last_fullsized":"2024-03-16T19:29:27.339540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339712+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:29:27.339712+0000","last_peered":"2024-03-16T19:29:27.339712+0000","last_clean":"2024-03-16T19:29:27.339712+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:29:27.339712+0000","last_undegraded":"2024-03-16T19:29:27.339712+0000","last_fullsized":"2024-03-16T19:29:27.339712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562225+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:29:27.562225+0000","last_peered":"2024-03-16T19:29:27.562225+0000","last_clean":"2024-03-16T19:29:27.562225+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:29:27.562225+0000","last_undegraded":"2024-03-16T19:29:27.562225+0000","last_fullsized":"2024-03-16T19:29:27.562225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.360976+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:29:31.360976+0000","last_peered":"2024-03-16T19:29:31.360976+0000","last_clean":"2024-03-16T19:29:31.360976+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:29:31.360976+0000","last_undegraded":"2024-03-16T19:29:31.360976+0000","last_fullsized":"2024-03-16T19:29:31.360976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062725+0000","last_change":"2024-03-16T19:29:26.350351+0000","last_active":"2024-03-16T19:29:28.062725+0000","last_peered":"2024-03-16T19:29:28.062725+0000","last_clean":"2024-03-16T19:29:28.062725+0000","last_became_active":"2024-03-16T19:29:26.350074+0000","last_became_peered":"2024-03-16T19:29:26.350074+0000","last_unstale":"2024-03-16T19:29:28.062725+0000","last_undegraded":"2024-03-16T19:29:28.062725+0000","last_fullsized":"2024-03-16T19:29:28.062725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:49.416395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.342931+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:29:27.342931+0000","last_peered":"2024-03-16T19:29:27.342931+0000","last_clean":"2024-03-16T19:29:27.342931+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:29:27.342931+0000","last_undegraded":"2024-03-16T19:29:27.342931+0000","last_fullsized":"2024-03-16T19:29:27.342931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063379+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:29:28.063379+0000","last_peered":"2024-03-16T19:29:28.063379+0000","last_clean":"2024-03-16T19:29:28.063379+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:29:28.063379+0000","last_undegraded":"2024-03-16T19:29:28.063379+0000","last_fullsized":"2024-03-16T19:29:28.063379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339690+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:29:27.339690+0000","last_peered":"2024-03-16T19:29:27.339690+0000","last_clean":"2024-03-16T19:29:27.339690+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:29:27.339690+0000","last_undegraded":"2024-03-16T19:29:27.339690+0000","last_fullsized":"2024-03-16T19:29:27.339690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561913+0000","last_change":"2024-03-16T19:29:26.347388+0000","last_active":"2024-03-16T19:29:27.561913+0000","last_peered":"2024-03-16T19:29:27.561913+0000","last_clean":"2024-03-16T19:29:27.561913+0000","last_became_active":"2024-03-16T19:29:26.346582+0000","last_became_peered":"2024-03-16T19:29:26.346582+0000","last_unstale":"2024-03-16T19:29:27.561913+0000","last_undegraded":"2024-03-16T19:29:27.561913+0000","last_fullsized":"2024-03-16T19:29:27.561913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:05.294895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339497+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:29:27.339497+0000","last_peered":"2024-03-16T19:29:27.339497+0000","last_clean":"2024-03-16T19:29:27.339497+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:29:27.339497+0000","last_undegraded":"2024-03-16T19:29:27.339497+0000","last_fullsized":"2024-03-16T19:29:27.339497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561968+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:29:27.561968+0000","last_peered":"2024-03-16T19:29:27.561968+0000","last_clean":"2024-03-16T19:29:27.561968+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:29:27.561968+0000","last_undegraded":"2024-03-16T19:29:27.561968+0000","last_fullsized":"2024-03-16T19:29:27.561968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365383+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:29:31.365383+0000","last_peered":"2024-03-16T19:29:31.365383+0000","last_clean":"2024-03-16T19:29:31.365383+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:29:31.365383+0000","last_undegraded":"2024-03-16T19:29:31.365383+0000","last_fullsized":"2024-03-16T19:29:31.365383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062837+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:29:28.062837+0000","last_peered":"2024-03-16T19:29:28.062837+0000","last_clean":"2024-03-16T19:29:28.062837+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:29:28.062837+0000","last_undegraded":"2024-03-16T19:29:28.062837+0000","last_fullsized":"2024-03-16T19:29:28.062837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343809+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:29:27.343809+0000","last_peered":"2024-03-16T19:29:27.343809+0000","last_clean":"2024-03-16T19:29:27.343809+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:29:27.343809+0000","last_undegraded":"2024-03-16T19:29:27.343809+0000","last_fullsized":"2024-03-16T19:29:27.343809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.360603+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:29:31.360603+0000","last_peered":"2024-03-16T19:29:31.360603+0000","last_clean":"2024-03-16T19:29:31.360603+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:29:31.360603+0000","last_undegraded":"2024-03-16T19:29:31.360603+0000","last_fullsized":"2024-03-16T19:29:31.360603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344208+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:29:27.344208+0000","last_peered":"2024-03-16T19:29:27.344208+0000","last_clean":"2024-03-16T19:29:27.344208+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:29:27.344208+0000","last_undegraded":"2024-03-16T19:29:27.344208+0000","last_fullsized":"2024-03-16T19:29:27.344208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365202+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:29:31.365202+0000","last_peered":"2024-03-16T19:29:31.365202+0000","last_clean":"2024-03-16T19:29:31.365202+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:29:31.365202+0000","last_undegraded":"2024-03-16T19:29:31.365202+0000","last_fullsized":"2024-03-16T19:29:31.365202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062891+0000","last_change":"2024-03-16T19:29:26.354303+0000","last_active":"2024-03-16T19:29:28.062891+0000","last_peered":"2024-03-16T19:29:28.062891+0000","last_clean":"2024-03-16T19:29:28.062891+0000","last_became_active":"2024-03-16T19:29:26.353804+0000","last_became_peered":"2024-03-16T19:29:26.353804+0000","last_unstale":"2024-03-16T19:29:28.062891+0000","last_undegraded":"2024-03-16T19:29:28.062891+0000","last_fullsized":"2024-03-16T19:29:28.062891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:10:45.368536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.215673+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:29:28.215673+0000","last_peered":"2024-03-16T19:29:28.215673+0000","last_clean":"2024-03-16T19:29:28.215673+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:29:28.215673+0000","last_undegraded":"2024-03-16T19:29:28.215673+0000","last_fullsized":"2024-03-16T19:29:28.215673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562176+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:29:27.562176+0000","last_peered":"2024-03-16T19:29:27.562176+0000","last_clean":"2024-03-16T19:29:27.562176+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:29:27.562176+0000","last_undegraded":"2024-03-16T19:29:27.562176+0000","last_fullsized":"2024-03-16T19:29:27.562176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365525+0000","last_change":"2024-03-16T19:29:25.342597+0000","last_active":"2024-03-16T19:29:31.365525+0000","last_peered":"2024-03-16T19:29:31.365525+0000","last_clean":"2024-03-16T19:29:31.365525+0000","last_became_active":"2024-03-16T19:29:25.342320+0000","last_became_peered":"2024-03-16T19:29:25.342320+0000","last_unstale":"2024-03-16T19:29:31.365525+0000","last_undegraded":"2024-03-16T19:29:31.365525+0000","last_fullsized":"2024-03-16T19:29:31.365525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:44:41.274396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":665973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:29:33.301 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-16T19:29:33.301 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-16T19:29:33.302 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-16T19:29:33.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-16T19:29:33.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.455+0000 7fa9e313e640 1 Processor -- start 2024-03-16T19:29:33.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.455+0000 7fa9e313e640 1 -- start start 2024-03-16T19:29:33.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9e313e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9dc06bb10 0x7fa9dc0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9e313e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa9dc06bef0 0x7fa9dc0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9e313e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc0cc720 con 0x7fa9dc0cc260 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9e313e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc0cc9c0 con 0x7fa9dc06bef0 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9e313e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc15de30 con 0x7fa9dc06bb10 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9e16db640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa9dc0cc260 0x7fa9dc154660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56242/0 (socket says 172.21.15.172:56242) 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9e16db640 1 -- 172.21.15.172:0/2324185677 learned_addr learned my addr 172.21.15.172:0/2324185677 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9dbfff640 1 --2- 172.21.15.172:0/2324185677 >> v2:172.21.15.174:3300/0 conn(0x7fa9dc06bef0 0x7fa9dc0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9e0eda640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9dc06bb10 0x7fa9dc0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375054127 0 0) 0x7fa9dc0cc720 con 0x7fa9dc0cc260 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.456+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc003520 con 0x7fa9dc0cc260 2024-03-16T19:29:33.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2779812020 0 0) 0x7fa9cc003520 con 0x7fa9dc0cc260 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9dc06bb10 msgr2=0x7fa9dc0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9dc06bb10 0x7fa9dc0cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 >> v2:172.21.15.174:3300/0 conn(0x7fa9dc06bef0 msgr2=0x7fa9dc0cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 --2- 172.21.15.172:0/2324185677 >> v2:172.21.15.174:3300/0 conn(0x7fa9dc06bef0 0x7fa9dc0cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc15e110 con 0x7fa9dc0cc260 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7fa9d40019f0 con 0x7fa9dc0cc260 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 >> v1:172.21.15.172:6789/0 conn(0x7fa9dc0cc260 legacy=0x7fa9dc154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9cc003970 0x7fa9cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9dc06bb10 0x7fa9cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9cc004d00 0x7fa9cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0cc9c0 con 0x7fa9cc003970 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc15de30 con 0x7fa9cc004d00 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc00b980 con 0x7fa9dc06bb10 2024-03-16T19:29:33.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9e0eda640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9cc003970 0x7fa9cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9dbfff640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9dc06bb10 0x7fa9cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9e16db640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9cc004d00 0x7fa9cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9e0eda640 1 -- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9dc06bb10 msgr2=0x7fa9cc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9e0eda640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9dc06bb10 0x7fa9cc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9e0eda640 1 -- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9cc004d00 msgr2=0x7fa9cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9e0eda640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9cc004d00 0x7fa9cc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9e0eda640 1 -- 172.21.15.172:0/2324185677 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc003520 con 0x7fa9cc003970 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.457+0000 7fa9dbfff640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9dc06bb10 0x7fa9cc0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.458+0000 7fa9e0eda640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9cc003970 0x7fa9cc003d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0009ac0 tx=0x7fa9cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48aa51c2293d479 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.458+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d002c6b0 con 0x7fa9cc003970 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.458+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d002c810 con 0x7fa9cc003970 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.458+0000 7fa9db7fe640 1 -- 172.21.15.172:0/2324185677 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d002cba0 con 0x7fa9cc003970 2024-03-16T19:29:33.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.458+0000 7fa9e313e640 1 -- 172.21.15.172:0/2324185677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9cc003970 msgr2=0x7fa9cc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.458+0000 7fa9e313e640 1 --2- 172.21.15.172:0/2324185677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9cc003970 0x7fa9cc003d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0009ac0 tx=0x7fa9cc0057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.458+0000 7fa9e16db640 1 -- 172.21.15.172:0/2324185677 reap_dead start 2024-03-16T19:29:33.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.458+0000 7fa9e313e640 1 -- 172.21.15.172:0/2324185677 shutdown_connections 2024-03-16T19:29:33.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.458+0000 7fa9e313e640 1 -- 172.21.15.172:0/2324185677 >> 172.21.15.172:0/2324185677 conn(0x7fa9dc067250 msgr2=0x7fa9dc0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:33.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.459+0000 7fa9e313e640 1 -- 172.21.15.172:0/2324185677 shutdown_connections 2024-03-16T19:29:33.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.459+0000 7fa9e313e640 1 -- 172.21.15.172:0/2324185677 wait complete. 2024-03-16T19:29:33.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.460+0000 7fa9e313e640 1 Processor -- start 2024-03-16T19:29:33.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.460+0000 7fa9e313e640 1 -- start start 2024-03-16T19:29:33.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.460+0000 7fa9e313e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9cc003970 0x7fa9dc0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.460+0000 7fa9e313e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9dc1f4160 0x7fa9dc0c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.460+0000 7fa9e313e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9dc0c8870 0x7fa9dc0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.460+0000 7fa9e313e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0b6300 con 0x7fa9dc1f4160 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.460+0000 7fa9e313e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0b5e70 con 0x7fa9dc0c8870 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.460+0000 7fa9e313e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0c8030 con 0x7fa9cc003970 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9dc1f4160 0x7fa9dc0c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9dc1f4160 0x7fa9dc0c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46440/0 (socket says 172.21.15.172:46440) 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9dbfff640 1 -- 172.21.15.172:0/2241651808 learned_addr learned my addr 172.21.15.172:0/2241651808 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9e0eda640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9cc003970 0x7fa9dc0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9e16db640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9dc0c8870 0x7fa9dc0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9dbfff640 1 -- 172.21.15.172:0/2241651808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9cc003970 msgr2=0x7fa9dc0b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9dbfff640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9cc003970 0x7fa9dc0b4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9dbfff640 1 -- 172.21.15.172:0/2241651808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9dc0c8870 msgr2=0x7fa9dc0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9dbfff640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9dc0c8870 0x7fa9dc0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9dbfff640 1 -- 172.21.15.172:0/2241651808 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc0b4f80 con 0x7fa9dc1f4160 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9e16db640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9dc0c8870 0x7fa9dc0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9dbfff640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9dc1f4160 0x7fa9dc0c8330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80116d0 tx=0x7fa9c8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9d97fa640 1 -- 172.21.15.172:0/2241651808 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c800f9c0 con 0x7fa9dc1f4160 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9d97fa640 1 -- 172.21.15.172:0/2241651808 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c800fb20 con 0x7fa9dc1f4160 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc0b5260 con 0x7fa9dc1f4160 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9d97fa640 1 -- 172.21.15.172:0/2241651808 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c800feb0 con 0x7fa9dc1f4160 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.461+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc0b5820 con 0x7fa9dc1f4160 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.462+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc0b6440 con 0x7fa9dc1f4160 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.462+0000 7fa9d97fa640 1 -- 172.21.15.172:0/2241651808 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa9c8015460 con 0x7fa9dc1f4160 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.463+0000 7fa9d97fa640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9c006b3b0 0x7fa9c006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.463+0000 7fa9d97fa640 1 -- 172.21.15.172:0/2241651808 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9c808a580 con 0x7fa9dc1f4160 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.463+0000 7fa9e0eda640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9c006b3b0 0x7fa9c006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.463+0000 7fa9e0eda640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9c006b3b0 0x7fa9c006d870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0036000 tx=0x7fa9d0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:33.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.467+0000 7fa9d97fa640 1 -- 172.21.15.172:0/2241651808 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9c8055150 con 0x7fa9dc1f4160 2024-03-16T19:29:33.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.675+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa9dc0681c0 con 0x7fa9dc1f4160 2024-03-16T19:29:33.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.676+0000 7fa9d97fa640 1 -- 172.21.15.172:0/2241651808 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa9c8058810 con 0x7fa9dc1f4160 2024-03-16T19:29:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:33.657 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-16T19:29:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9c006b3b0 msgr2=0x7fa9c006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9c006b3b0 0x7fa9c006d870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0036000 tx=0x7fa9d0030000 comp rx=0 tx=0).stop 2024-03-16T19:29:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9dc1f4160 msgr2=0x7fa9dc0c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9dc1f4160 0x7fa9dc0c8330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80116d0 tx=0x7fa9c8011b90 comp rx=0 tx=0).stop 2024-03-16T19:29:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 shutdown_connections 2024-03-16T19:29:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9dc0c8870 0x7fa9dc0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa9dc1f4160 0x7fa9dc0c8330 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa9c006b3b0 0x7fa9c006d870 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 --2- 172.21.15.172:0/2241651808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9cc003970 0x7fa9dc0b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 >> 172.21.15.172:0/2241651808 conn(0x7fa9dc067250 msgr2=0x7fa9dc152520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:33.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 shutdown_connections 2024-03-16T19:29:33.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.678+0000 7fa9e313e640 1 -- 172.21.15.172:0/2241651808 wait complete. 2024-03-16T19:29:33.670 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-16T19:29:33.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:33.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.790+0000 7f47a8d48640 1 Processor -- start 2024-03-16T19:29:33.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.790+0000 7f47a8d48640 1 -- start start 2024-03-16T19:29:33.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.791+0000 7f47a8d48640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406aed0 0x7f47a406dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.791+0000 7f47a8d48640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a406ba50 con 0x7f47a4154bd0 2024-03-16T19:29:33.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.791+0000 7f47a8d48640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a406bd30 con 0x7f47a406e2b0 2024-03-16T19:29:33.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.791+0000 7f47a8d48640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a415e0d0 con 0x7f47a406aed0 2024-03-16T19:29:33.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.791+0000 7f47a259c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f47a4154bd0 0x7f47a40c7b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56252/0 (socket says 172.21.15.172:56252) 2024-03-16T19:29:33.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.791+0000 7f47a259c640 1 -- 172.21.15.172:0/2618556988 learned_addr learned my addr 172.21.15.172:0/2618556988 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:33.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.791+0000 7f47a1d9b640 1 --2- 172.21.15.172:0/2618556988 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406aed0 0x7f47a406dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.791+0000 7f47a159a640 1 -- 172.21.15.172:0/2618556988 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222479404 0 0) 0x7f47a406ba50 con 0x7f47a4154bd0 2024-03-16T19:29:33.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.792+0000 7f47a159a640 1 -- 172.21.15.172:0/2618556988 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f478c003520 con 0x7f47a4154bd0 2024-03-16T19:29:33.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.792+0000 7f47a1d9b640 1 -- 172.21.15.172:0/2618556988 >> v1:172.21.15.174:6789/0 conn(0x7f47a406e2b0 legacy=0x7f47a415e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.792+0000 7f47a159a640 1 -- 172.21.15.172:0/2618556988 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3991702528 0 0) 0x7f478c003520 con 0x7f47a4154bd0 2024-03-16T19:29:33.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.792+0000 7f47a1d9b640 1 -- 172.21.15.172:0/2618556988 >> v1:172.21.15.172:6789/0 conn(0x7f47a4154bd0 legacy=0x7f47a40c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.792+0000 7f47a1d9b640 1 -- 172.21.15.172:0/2618556988 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a406c010 con 0x7f47a406aed0 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.792+0000 7f47a1d9b640 1 --2- 172.21.15.172:0/2618556988 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406aed0 0x7f47a406dd70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f47980097d0 tx=0x7f479802b6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63d7a6cc36b249a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a159a640 1 -- 172.21.15.172:0/2618556988 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47940019f0 con 0x7f47a406aed0 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a159a640 1 -- 172.21.15.172:0/2618556988 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479802e050 con 0x7f47a406aed0 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a159a640 1 -- 172.21.15.172:0/2618556988 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4798030710 con 0x7f47a406aed0 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a8d48640 1 -- 172.21.15.172:0/2618556988 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406aed0 msgr2=0x7f47a406dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a8d48640 1 --2- 172.21.15.172:0/2618556988 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406aed0 0x7f47a406dd70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f47980097d0 tx=0x7f479802b6f0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a8d48640 1 -- 172.21.15.172:0/2618556988 shutdown_connections 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a8d48640 1 --2- 172.21.15.172:0/2618556988 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406aed0 0x7f47a406dd70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a8d48640 1 -- 172.21.15.172:0/2618556988 >> 172.21.15.172:0/2618556988 conn(0x7f47a4071be0 msgr2=0x7f47a4071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a8d48640 1 -- 172.21.15.172:0/2618556988 shutdown_connections 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.793+0000 7f47a8d48640 1 -- 172.21.15.172:0/2618556988 wait complete. 2024-03-16T19:29:33.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a8d48640 1 Processor -- start 2024-03-16T19:29:33.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a8d48640 1 -- start start 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a8d48640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47a406aed0 0x7f47a41552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a8d48640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406e2b0 0x7f47a415bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a8d48640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a4154bd0 0x7f47a406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a8d48640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a4159010 con 0x7f47a4154bd0 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a8d48640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a4158800 con 0x7f47a406aed0 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a8d48640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a415b030 con 0x7f47a406e2b0 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a2d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a4154bd0 0x7f47a406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a2d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a4154bd0 0x7f47a406ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46456/0 (socket says 172.21.15.172:46456) 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.795+0000 7f47a2d9d640 1 -- 172.21.15.172:0/2995926318 learned_addr learned my addr 172.21.15.172:0/2995926318 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a1d9b640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406e2b0 0x7f47a415bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a259c640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47a406aed0 0x7f47a41552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a2d9d640 1 -- 172.21.15.172:0/2995926318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406e2b0 msgr2=0x7f47a415bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a2d9d640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406e2b0 0x7f47a415bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a2d9d640 1 -- 172.21.15.172:0/2995926318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47a406aed0 msgr2=0x7f47a41552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a2d9d640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47a406aed0 0x7f47a41552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a2d9d640 1 -- 172.21.15.172:0/2995926318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798009400 con 0x7f47a4154bd0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a2d9d640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a4154bd0 0x7f47a406ff80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f479c00c4e0 tx=0x7f479c00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f4792ffd640 1 -- 172.21.15.172:0/2995926318 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c005480 con 0x7f47a4154bd0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f4792ffd640 1 -- 172.21.15.172:0/2995926318 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479c0035f0 con 0x7f47a4154bd0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a4155130 con 0x7f47a4154bd0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f4792ffd640 1 -- 172.21.15.172:0/2995926318 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c0039a0 con 0x7f47a4154bd0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.796+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a4158d30 con 0x7f47a4154bd0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.797+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a415c0b0 con 0x7f47a4154bd0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.798+0000 7f4792ffd640 1 -- 172.21.15.172:0/2995926318 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f479c003b00 con 0x7f47a4154bd0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.798+0000 7f4792ffd640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f478406b3e0 0x7f478406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.798+0000 7f4792ffd640 1 -- 172.21.15.172:0/2995926318 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f479c089180 con 0x7f47a4154bd0 2024-03-16T19:29:33.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.798+0000 7f47a259c640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f478406b3e0 0x7f478406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:33.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.798+0000 7f47a259c640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f478406b3e0 0x7f478406d8a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4794002230 tx=0x7f4794001ce0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:33.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.802+0000 7f4792ffd640 1 -- 172.21.15.172:0/2995926318 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f479c053e90 con 0x7f47a4154bd0 2024-03-16T19:29:33.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.975+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47a415b1a0 con 0x7f47a4154bd0 2024-03-16T19:29:33.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.976+0000 7f4792ffd640 1 -- 172.21.15.172:0/2995926318 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f479c057550 con 0x7f47a4154bd0 2024-03-16T19:29:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:33.957 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":21,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:27.336190+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:33.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f478406b3e0 msgr2=0x7f478406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f478406b3e0 0x7f478406d8a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4794002230 tx=0x7f4794001ce0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a4154bd0 msgr2=0x7f47a406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:33.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a4154bd0 0x7f47a406ff80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f479c00c4e0 tx=0x7f479c00c9a0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 shutdown_connections 2024-03-16T19:29:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a4154bd0 0x7f47a406ff80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f478406b3e0 0x7f478406d8a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47a406e2b0 0x7f47a415bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 --2- 172.21.15.172:0/2995926318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47a406aed0 0x7f47a41552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 >> 172.21.15.172:0/2995926318 conn(0x7f47a4071be0 msgr2=0x7f47a40cc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 shutdown_connections 2024-03-16T19:29:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:33.978+0000 7f47a8d48640 1 -- 172.21.15.172:0/2995926318 wait complete. 2024-03-16T19:29:33.970 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:29:14.242742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:29:24.121132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:29:24.651099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T19:29:33.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:29:34.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.093+0000 7f463412c640 1 Processor -- start 2024-03-16T19:29:34.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.094+0000 7f463412c640 1 -- start start 2024-03-16T19:29:34.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.094+0000 7f463412c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c1583d0 0x7f462c0c9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.094+0000 7f463412c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f462c0b0df0 0x7f462c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.094+0000 7f463412c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462c06dc50 con 0x7f462c06fec0 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.094+0000 7f463412c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462c06df10 con 0x7f462c0b0df0 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.094+0000 7f463412c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462c06f240 con 0x7f462c1583d0 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.094+0000 7f46316c7640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f462c06fec0 0x7f462c0ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56254/0 (socket says 172.21.15.172:56254) 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.094+0000 7f46316c7640 1 -- 172.21.15.172:0/875357833 learned_addr learned my addr 172.21.15.172:0/875357833 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f46326c9640 1 --2- 172.21.15.172:0/875357833 >> v2:172.21.15.174:3300/0 conn(0x7f462c0b0df0 0x7f462c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745814709 0 0) 0x7f462c06dc50 con 0x7f462c06fec0 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4614003520 con 0x7f462c06fec0 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4631ec8640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c1583d0 0x7f462c0c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2630322225 0 0) 0x7f4614003520 con 0x7f462c06fec0 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c1583d0 msgr2=0x7f462c0c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c1583d0 0x7f462c0c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 >> v2:172.21.15.174:3300/0 conn(0x7f462c0b0df0 msgr2=0x7f462c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 --2- 172.21.15.172:0/875357833 >> v2:172.21.15.174:3300/0 conn(0x7f462c0b0df0 0x7f462c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462c06c180 con 0x7f462c06fec0 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f4620001670 con 0x7f462c06fec0 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 >> v1:172.21.15.172:6789/0 conn(0x7f462c06fec0 legacy=0x7f462c0ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f4614003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46140047c0 0x7f4614004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c1583d0 0x7f4614005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c06df10 con 0x7f4614003970 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c06f240 con 0x7f46140047c0 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461400b980 con 0x7f462c1583d0 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4631ec8640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f4614003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f46316c7640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46140047c0 0x7f4614004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f46326c9640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c1583d0 0x7f4614005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4631ec8640 1 -- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c1583d0 msgr2=0x7f4614005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4631ec8640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c1583d0 0x7f4614005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4631ec8640 1 -- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46140047c0 msgr2=0x7f4614004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4631ec8640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46140047c0 0x7f4614004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f4631ec8640 1 -- 172.21.15.172:0/875357833 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4614003520 con 0x7f4614003970 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.095+0000 7f46316c7640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46140047c0 0x7f4614004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.096+0000 7f4631ec8640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f4614003d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f462400ba30 tx=0x7f46140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b31c3adb3a67f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.096+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624032640 con 0x7f4614003970 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.096+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46240327a0 con 0x7f4614003970 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.096+0000 7f4630ec6640 1 -- 172.21.15.172:0/875357833 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624032b30 con 0x7f4614003970 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.097+0000 7f463412c640 1 -- 172.21.15.172:0/875357833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 msgr2=0x7f4614003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.097+0000 7f463412c640 1 --2- 172.21.15.172:0/875357833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f4614003d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f462400ba30 tx=0x7f46140057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.097+0000 7f46326c9640 1 -- 172.21.15.172:0/875357833 reap_dead start 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.097+0000 7f463412c640 1 -- 172.21.15.172:0/875357833 shutdown_connections 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.097+0000 7f463412c640 1 -- 172.21.15.172:0/875357833 >> 172.21.15.172:0/875357833 conn(0x7f462c066c20 msgr2=0x7f462c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.097+0000 7f463412c640 1 -- 172.21.15.172:0/875357833 shutdown_connections 2024-03-16T19:29:34.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.097+0000 7f463412c640 1 -- 172.21.15.172:0/875357833 wait complete. 2024-03-16T19:29:34.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.098+0000 7f463412c640 1 Processor -- start 2024-03-16T19:29:34.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.098+0000 7f463412c640 1 -- start start 2024-03-16T19:29:34.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f463412c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f462c0c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f463412c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c156c40 0x7f462c14b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f463412c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f462c0ae8a0 0x7f462c14b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f463412c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0ce780 con 0x7f4614003970 2024-03-16T19:29:34.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f463412c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0ce230 con 0x7f462c0ae8a0 2024-03-16T19:29:34.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f463412c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0ccca0 con 0x7f462c156c40 2024-03-16T19:29:34.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f4631ec8640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f462c0c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f4631ec8640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f462c0c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46484/0 (socket says 172.21.15.172:46484) 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f4631ec8640 1 -- 172.21.15.172:0/1789676539 learned_addr learned my addr 172.21.15.172:0/1789676539 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f4631ec8640 1 -- 172.21.15.172:0/1789676539 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c156c40 msgr2=0x7f462c14b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f46316c7640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c156c40 0x7f462c14b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f4631ec8640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c156c40 0x7f462c14b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f4631ec8640 1 -- 172.21.15.172:0/1789676539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f462c0ae8a0 msgr2=0x7f462c14b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f4631ec8640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f462c0ae8a0 0x7f462c14b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f4631ec8640 1 -- 172.21.15.172:0/1789676539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462c0ce4a0 con 0x7f4614003970 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f4631ec8640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f462c0c4440 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4624035040 tx=0x7f4624004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f462a7fc640 1 -- 172.21.15.172:0/1789676539 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624004220 con 0x7f4614003970 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f462a7fc640 1 -- 172.21.15.172:0/1789676539 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46240043a0 con 0x7f4614003970 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462c0cc370 con 0x7f4614003970 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f462a7fc640 1 -- 172.21.15.172:0/1789676539 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624003850 con 0x7f4614003970 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.099+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462c0cc930 con 0x7f4614003970 2024-03-16T19:29:34.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.100+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462c0688d0 con 0x7f4614003970 2024-03-16T19:29:34.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.101+0000 7f462a7fc640 1 -- 172.21.15.172:0/1789676539 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4624043070 con 0x7f4614003970 2024-03-16T19:29:34.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.101+0000 7f462a7fc640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f461806b3e0 0x7f461806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.101+0000 7f462a7fc640 1 -- 172.21.15.172:0/1789676539 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46240ac200 con 0x7f4614003970 2024-03-16T19:29:34.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.101+0000 7f46316c7640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f461806b3e0 0x7f461806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.102+0000 7f46316c7640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f461806b3e0 0x7f461806d8a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4620000fb0 tx=0x7f4620001de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.105+0000 7f462a7fc640 1 -- 172.21.15.172:0/1789676539 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4624076f80 con 0x7f4614003970 2024-03-16T19:29:34.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.274+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f462c06fca0 con 0x7f4614003970 2024-03-16T19:29:34.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.275+0000 7f462a7fc640 1 -- 172.21.15.172:0/1789676539 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f462407a640 con 0x7f4614003970 2024-03-16T19:29:34.256 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 1 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.276+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f461806b3e0 msgr2=0x7f461806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.276+0000 7f463412c640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f461806b3e0 0x7f461806d8a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4620000fb0 tx=0x7f4620001de0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.276+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 msgr2=0x7f462c0c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.276+0000 7f463412c640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f462c0c4440 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4624035040 tx=0x7f4624004730 comp rx=0 tx=0).stop 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.277+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 shutdown_connections 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.277+0000 7f463412c640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f462c0ae8a0 0x7f462c14b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.277+0000 7f463412c640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f461806b3e0 0x7f461806d8a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.277+0000 7f463412c640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f462c156c40 0x7f462c14b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.277+0000 7f463412c640 1 --2- 172.21.15.172:0/1789676539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4614003970 0x7f462c0c4440 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.277+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 >> 172.21.15.172:0/1789676539 conn(0x7f462c066c20 msgr2=0x7f462c154820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.277+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 shutdown_connections 2024-03-16T19:29:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.277+0000 7f463412c640 1 -- 172.21.15.172:0/1789676539 wait complete. 2024-03-16T19:29:34.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T19:29:34.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.393+0000 7ff49add7640 1 Processor -- start 2024-03-16T19:29:34.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.393+0000 7ff49add7640 1 -- start start 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.394+0000 7ff49add7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff4940cd290 0x7ff4940cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.394+0000 7ff49add7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49406b2b0 con 0x7ff49406cc50 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.394+0000 7ff49add7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49406b590 con 0x7ff494154c90 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.394+0000 7ff49add7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49415b470 con 0x7ff4940cd290 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.394+0000 7ff493fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff49406cc50 0x7ff4940c1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56260/0 (socket says 172.21.15.172:56260) 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.394+0000 7ff493fff640 1 -- 172.21.15.172:0/2473574516 learned_addr learned my addr 172.21.15.172:0/2473574516 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.394+0000 7ff499374640 1 --2- 172.21.15.172:0/2473574516 >> v2:172.21.15.174:3300/0 conn(0x7ff4940cd290 0x7ff4940cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.394+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120656339 0 0) 0x7ff49406b2b0 con 0x7ff49406cc50 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.394+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff47c003520 con 0x7ff49406cc50 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1325642921 0 0) 0x7ff47c003520 con 0x7ff49406cc50 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 >> v2:172.21.15.174:3300/0 conn(0x7ff4940cd290 msgr2=0x7ff4940cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 --2- 172.21.15.172:0/2473574516 >> v2:172.21.15.174:3300/0 conn(0x7ff4940cd290 0x7ff4940cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 >> v1:172.21.15.174:6789/0 conn(0x7ff494154c90 legacy=0x7ff4940c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49415adb0 con 0x7ff49406cc50 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7ff488001670 con 0x7ff49406cc50 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 >> v1:172.21.15.172:6789/0 conn(0x7ff49406cc50 legacy=0x7ff4940c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c003970 0x7ff47c003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff47c004370 0x7ff47c004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff47c005040 0x7ff47c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49415b470 con 0x7ff47c005040 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c00bfa0 con 0x7ff47c003970 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c00c160 con 0x7ff47c004370 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff499374640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff47c005040 0x7ff47c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff498b73640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c003970 0x7ff47c003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff493fff640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff47c004370 0x7ff47c004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff499374640 1 -- 172.21.15.172:0/2473574516 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff47c004370 msgr2=0x7ff47c004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff499374640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff47c004370 0x7ff47c004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff499374640 1 -- 172.21.15.172:0/2473574516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c003970 msgr2=0x7ff47c003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff499374640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c003970 0x7ff47c003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff499374640 1 -- 172.21.15.172:0/2473574516 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c003520 con 0x7ff47c005040 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.395+0000 7ff498b73640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c003970 0x7ff47c003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.396+0000 7ff499374640 1 -- 172.21.15.172:0/2473574516 reap_dead start 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.396+0000 7ff499374640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff47c005040 0x7ff47c00b980 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009fc0 tx=0x7ff47c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a06c3a8fc28253 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.396+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c02cc60 con 0x7ff47c005040 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.396+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48c00f040 con 0x7ff47c005040 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.396+0000 7ff4937fe640 1 -- 172.21.15.172:0/2473574516 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c003990 con 0x7ff47c005040 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.396+0000 7ff49add7640 1 -- 172.21.15.172:0/2473574516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff47c005040 msgr2=0x7ff47c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.396+0000 7ff49add7640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff47c005040 0x7ff47c00b980 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009fc0 tx=0x7ff47c0057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.396+0000 7ff49add7640 1 -- 172.21.15.172:0/2473574516 shutdown_connections 2024-03-16T19:29:34.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.397+0000 7ff49add7640 1 --2- 172.21.15.172:0/2473574516 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff47c005040 0x7ff47c00b980 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.397+0000 7ff49add7640 1 -- 172.21.15.172:0/2473574516 >> 172.21.15.172:0/2473574516 conn(0x7ff494066c20 msgr2=0x7ff494067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:34.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.397+0000 7ff49add7640 1 -- 172.21.15.172:0/2473574516 shutdown_connections 2024-03-16T19:29:34.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.397+0000 7ff49add7640 1 -- 172.21.15.172:0/2473574516 wait complete. 2024-03-16T19:29:34.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.398+0000 7ff49add7640 1 Processor -- start 2024-03-16T19:29:34.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.398+0000 7ff49add7640 1 -- start start 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.398+0000 7ff49add7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c005040 0x7ff494153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.398+0000 7ff49add7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff49414ccc0 0x7ff49414d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.398+0000 7ff49add7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4940c5820 0x7ff4940c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.398+0000 7ff49add7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4941521d0 con 0x7ff49414ccc0 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.398+0000 7ff49add7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff494153860 con 0x7ff47c005040 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.398+0000 7ff49add7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff494153060 con 0x7ff4940c5820 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff493fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff49414ccc0 0x7ff49414d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff493fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff49414ccc0 0x7ff49414d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46490/0 (socket says 172.21.15.172:46490) 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff493fff640 1 -- 172.21.15.172:0/1769408047 learned_addr learned my addr 172.21.15.172:0/1769408047 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff498b73640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c005040 0x7ff494153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff499374640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4940c5820 0x7ff4940c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff493fff640 1 -- 172.21.15.172:0/1769408047 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4940c5820 msgr2=0x7ff4940c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff493fff640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4940c5820 0x7ff4940c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff493fff640 1 -- 172.21.15.172:0/1769408047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c005040 msgr2=0x7ff494153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff493fff640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c005040 0x7ff494153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff493fff640 1 -- 172.21.15.172:0/1769408047 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4940c5390 con 0x7ff49414ccc0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff498b73640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c005040 0x7ff494153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff493fff640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff49414ccc0 0x7ff49414d140 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff48800e730 tx=0x7ff48800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff4917fa640 1 -- 172.21.15.172:0/1769408047 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48800ee40 con 0x7ff49414ccc0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff4917fa640 1 -- 172.21.15.172:0/1769408047 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48800c8d0 con 0x7ff49414ccc0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4940c61e0 con 0x7ff49414ccc0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff4917fa640 1 -- 172.21.15.172:0/1769408047 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48800cc80 con 0x7ff49414ccc0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.399+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4940b06f0 con 0x7ff49414ccc0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.400+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4940b1960 con 0x7ff49414ccc0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.401+0000 7ff4917fa640 1 -- 172.21.15.172:0/1769408047 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff48801d460 con 0x7ff49414ccc0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.401+0000 7ff4917fa640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff48006b3b0 0x7ff48006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.401+0000 7ff4917fa640 1 -- 172.21.15.172:0/1769408047 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4880899c0 con 0x7ff49414ccc0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.401+0000 7ff498b73640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff48006b3b0 0x7ff48006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.402+0000 7ff498b73640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff48006b3b0 0x7ff48006d870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff484001b50 tx=0x7ff484009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.405+0000 7ff4917fa640 1 -- 172.21.15.172:0/1769408047 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff488058290 con 0x7ff49414ccc0 2024-03-16T19:29:34.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.576+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4940688d0 con 0x7ff49414ccc0 2024-03-16T19:29:34.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.576+0000 7ff4917fa640 1 -- 172.21.15.172:0/1769408047 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff4880014f0 con 0x7ff49414ccc0 2024-03-16T19:29:34.557 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 32 2024-03-16T19:29:34.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff48006b3b0 msgr2=0x7ff48006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff48006b3b0 0x7ff48006d870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff484001b50 tx=0x7ff484009040 comp rx=0 tx=0).stop 2024-03-16T19:29:34.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff49414ccc0 msgr2=0x7ff49414d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff49414ccc0 0x7ff49414d140 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff48800e730 tx=0x7ff48800ebf0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 shutdown_connections 2024-03-16T19:29:34.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff48006b3b0 0x7ff48006d870 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4940c5820 0x7ff4940c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff49414ccc0 0x7ff49414d140 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 --2- 172.21.15.172:0/1769408047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff47c005040 0x7ff494153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 >> 172.21.15.172:0/1769408047 conn(0x7ff494066c20 msgr2=0x7ff4940b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:34.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 shutdown_connections 2024-03-16T19:29:34.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.578+0000 7ff49add7640 1 -- 172.21.15.172:0/1769408047 wait complete. 2024-03-16T19:29:34.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T19:29:34.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.692+0000 7f60277d7640 1 Processor -- start 2024-03-16T19:29:34.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.692+0000 7f60277d7640 1 -- start start 2024-03-16T19:29:34.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.693+0000 7f60277d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6020073310 0x7f602006ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.693+0000 7f60277d7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60201576b0 con 0x7f60200c79d0 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.693+0000 7f60277d7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6020071c10 con 0x7f6020151910 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.693+0000 7f60277d7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60200c9180 con 0x7f6020073310 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.693+0000 7f6025d74640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f60200c79d0 0x7f6020157b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56274/0 (socket says 172.21.15.172:56274) 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.693+0000 7f6025d74640 1 -- 172.21.15.172:0/240057113 learned_addr learned my addr 172.21.15.172:0/240057113 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6024d72640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6020073310 0x7f602006ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005139223 0 0) 0x7f60201576b0 con 0x7f60200c79d0 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c003520 con 0x7f60200c79d0 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 911556956 0 0) 0x7f600c003520 con 0x7f60200c79d0 2024-03-16T19:29:34.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6020073310 msgr2=0x7f602006ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6020073310 0x7f602006ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 >> v1:172.21.15.174:6789/0 conn(0x7f6020151910 legacy=0x7f60200cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60200c7e00 con 0x7f60200c79d0 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f6018001670 con 0x7f60200c79d0 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 >> v1:172.21.15.172:6789/0 conn(0x7f60200c79d0 legacy=0x7f6020157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f600c0039a0 0x7f600c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f600c0047c0 0x7f600c004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f600c005140 0x7f600c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60200c9180 con 0x7f600c0039a0 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c00bec0 con 0x7f600c005140 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c00c030 con 0x7f600c0047c0 2024-03-16T19:29:34.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6025573640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f600c0039a0 0x7f600c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6025d74640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f600c005140 0x7f600c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6024d72640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f600c0047c0 0x7f600c004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6025573640 1 -- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f600c0047c0 msgr2=0x7f600c004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6025573640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f600c0047c0 0x7f600c004c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6025573640 1 -- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f600c005140 msgr2=0x7f600c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6025573640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f600c005140 0x7f600c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6025573640 1 -- 172.21.15.172:0/240057113 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c003520 con 0x7f600c0039a0 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.694+0000 7f6024d72640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f600c0047c0 0x7f600c004c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.695+0000 7f6025573640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f600c0039a0 0x7f600c003dc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f601c001e50 tx=0x7f600c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a929573f0d9b2feb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.695+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c003b70 con 0x7f600c0039a0 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.695+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c003cd0 con 0x7f600c0039a0 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.695+0000 7f6017fff640 1 -- 172.21.15.172:0/240057113 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c005310 con 0x7f600c0039a0 2024-03-16T19:29:34.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.696+0000 7f60277d7640 1 -- 172.21.15.172:0/240057113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f600c0039a0 msgr2=0x7f600c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.696+0000 7f60277d7640 1 --2- 172.21.15.172:0/240057113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f600c0039a0 0x7f600c003dc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f601c001e50 tx=0x7f600c0057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.696+0000 7f6025d74640 1 -- 172.21.15.172:0/240057113 reap_dead start 2024-03-16T19:29:34.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.696+0000 7f60277d7640 1 -- 172.21.15.172:0/240057113 shutdown_connections 2024-03-16T19:29:34.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.696+0000 7f60277d7640 1 -- 172.21.15.172:0/240057113 >> 172.21.15.172:0/240057113 conn(0x7f6020066c20 msgr2=0x7f6020067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:34.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.696+0000 7f60277d7640 1 -- 172.21.15.172:0/240057113 shutdown_connections 2024-03-16T19:29:34.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.696+0000 7f60277d7640 1 -- 172.21.15.172:0/240057113 wait complete. 2024-03-16T19:29:34.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.697+0000 7f60277d7640 1 Processor -- start 2024-03-16T19:29:34.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.697+0000 7f60277d7640 1 -- start start 2024-03-16T19:29:34.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f60277d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f600c0039a0 0x7f602014bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f60277d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602006b9b0 0x7f602006bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f60277d7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60201466a0 0x7f6020146b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f60277d7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602006c330 con 0x7f60201466a0 2024-03-16T19:29:34.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f60277d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602006b520 con 0x7f600c0039a0 2024-03-16T19:29:34.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f60277d7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6020156370 con 0x7f602006b9b0 2024-03-16T19:29:34.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025d74640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60201466a0 0x7f6020146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025d74640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60201466a0 0x7f6020146b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46508/0 (socket says 172.21.15.172:46508) 2024-03-16T19:29:34.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025d74640 1 -- 172.21.15.172:0/887182619 learned_addr learned my addr 172.21.15.172:0/887182619 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025573640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f600c0039a0 0x7f602014bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6024d72640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602006b9b0 0x7f602006bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025d74640 1 -- 172.21.15.172:0/887182619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602006b9b0 msgr2=0x7f602006bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025d74640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602006b9b0 0x7f602006bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025d74640 1 -- 172.21.15.172:0/887182619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f600c0039a0 msgr2=0x7f602014bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025d74640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f600c0039a0 0x7f602014bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025d74640 1 -- 172.21.15.172:0/887182619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6020156630 con 0x7f60201466a0 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6025d74640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60201466a0 0x7f6020146b20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6020067740 tx=0x7f6018007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6015ffb640 1 -- 172.21.15.172:0/887182619 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6018001640 con 0x7f60201466a0 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6015ffb640 1 -- 172.21.15.172:0/887182619 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601800f890 con 0x7f60201466a0 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f6015ffb640 1 -- 172.21.15.172:0/887182619 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601800fc40 con 0x7f60201466a0 2024-03-16T19:29:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6020156910 con 0x7f60201466a0 2024-03-16T19:29:34.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.698+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602006b1b0 con 0x7f60201466a0 2024-03-16T19:29:34.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.699+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602006b340 con 0x7f60201466a0 2024-03-16T19:29:34.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.700+0000 7f6015ffb640 1 -- 172.21.15.172:0/887182619 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f601801d460 con 0x7f60201466a0 2024-03-16T19:29:34.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.700+0000 7f6015ffb640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f600406b3b0 0x7f600406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.700+0000 7f6025573640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f600406b3b0 0x7f600406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.700+0000 7f6015ffb640 1 -- 172.21.15.172:0/887182619 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6018089b20 con 0x7f60201466a0 2024-03-16T19:29:34.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.701+0000 7f6025573640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f600406b3b0 0x7f600406d870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f601c034000 tx=0x7f601c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.704+0000 7f6015ffb640 1 -- 172.21.15.172:0/887182619 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6018054810 con 0x7f60201466a0 2024-03-16T19:29:34.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.874+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6020156b00 con 0x7f60201466a0 2024-03-16T19:29:34.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.875+0000 7f6015ffb640 1 -- 172.21.15.172:0/887182619 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6018057ed0 con 0x7f60201466a0 2024-03-16T19:29:34.856 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 64 2024-03-16T19:29:34.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f600406b3b0 msgr2=0x7f600406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f600406b3b0 0x7f600406d870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f601c034000 tx=0x7f601c009030 comp rx=0 tx=0).stop 2024-03-16T19:29:34.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60201466a0 msgr2=0x7f6020146b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60201466a0 0x7f6020146b20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6020067740 tx=0x7f6018007720 comp rx=0 tx=0).stop 2024-03-16T19:29:34.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 shutdown_connections 2024-03-16T19:29:34.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60201466a0 0x7f6020146b20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f600406b3b0 0x7f600406d870 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602006b9b0 0x7f602006bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 --2- 172.21.15.172:0/887182619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f600c0039a0 0x7f602014bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 >> 172.21.15.172:0/887182619 conn(0x7f6020066c20 msgr2=0x7f60200cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:34.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 shutdown_connections 2024-03-16T19:29:34.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.877+0000 7f60277d7640 1 -- 172.21.15.172:0/887182619 wait complete. 2024-03-16T19:29:34.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:29:34.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.988+0000 7f28c273a640 1 Processor -- start 2024-03-16T19:29:34.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.988+0000 7f28c273a640 1 -- start start 2024-03-16T19:29:34.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.989+0000 7f28c273a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f28bc06bb10 0x7f28bc0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.989+0000 7f28c273a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc06bef0 0x7f28bc0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.989+0000 7f28c273a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc0cc720 con 0x7f28bc0cc260 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.989+0000 7f28c273a640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28bc0cc9c0 con 0x7f28bc06bb10 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.989+0000 7f28c273a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc15de30 con 0x7f28bc06bef0 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.989+0000 7f28bbfff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f28bc06bb10 0x7f28bc0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.989+0000 7f28c0cd7640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f28bc0cc260 0x7f28bc154660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56276/0 (socket says 172.21.15.172:56276) 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.989+0000 7f28c0cd7640 1 -- 172.21.15.172:0/3716031767 learned_addr learned my addr 172.21.15.172:0/3716031767 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28bbfff640 1 -- 172.21.15.172:0/3716031767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc06bef0 msgr2=0x7f28bc0cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28bb7fe640 1 --2- 172.21.15.172:0/3716031767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc06bef0 0x7f28bc0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28bbfff640 1 --2- 172.21.15.172:0/3716031767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc06bef0 0x7f28bc0cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28bbfff640 1 -- 172.21.15.172:0/3716031767 >> v1:172.21.15.172:6789/0 conn(0x7f28bc0cc260 legacy=0x7f28bc154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28baffd640 1 -- 172.21.15.172:0/3716031767 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606040911 0 0) 0x7f28bc0cc720 con 0x7f28bc0cc260 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28bbfff640 1 -- 172.21.15.172:0/3716031767 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc15e110 con 0x7f28bc06bb10 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28bbfff640 1 --2- 172.21.15.172:0/3716031767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 0x7f28bc0cae50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0097a0 tx=0x7f28ac02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b35dfea2f302ad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28baffd640 1 -- 172.21.15.172:0/3716031767 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac030610 con 0x7f28bc06bb10 2024-03-16T19:29:34.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28baffd640 1 -- 172.21.15.172:0/3716031767 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac030770 con 0x7f28bc06bb10 2024-03-16T19:29:34.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.990+0000 7f28baffd640 1 -- 172.21.15.172:0/3716031767 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac030b00 con 0x7f28bc06bb10 2024-03-16T19:29:34.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.991+0000 7f28c273a640 1 -- 172.21.15.172:0/3716031767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 msgr2=0x7f28bc0cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.991+0000 7f28c273a640 1 --2- 172.21.15.172:0/3716031767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 0x7f28bc0cae50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0097a0 tx=0x7f28ac02b960 comp rx=0 tx=0).stop 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.991+0000 7f28c273a640 1 -- 172.21.15.172:0/3716031767 shutdown_connections 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.991+0000 7f28c273a640 1 --2- 172.21.15.172:0/3716031767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc06bef0 0x7f28bc0cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.991+0000 7f28c273a640 1 --2- 172.21.15.172:0/3716031767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 0x7f28bc0cae50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.991+0000 7f28c273a640 1 -- 172.21.15.172:0/3716031767 >> 172.21.15.172:0/3716031767 conn(0x7f28bc067250 msgr2=0x7f28bc0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.991+0000 7f28c273a640 1 -- 172.21.15.172:0/3716031767 shutdown_connections 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.991+0000 7f28c273a640 1 -- 172.21.15.172:0/3716031767 wait complete. 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.992+0000 7f28c273a640 1 Processor -- start 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.992+0000 7f28c273a640 1 -- start start 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28c273a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 0x7f28bc157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28c273a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28bc06bef0 0x7f28bc0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28c273a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc0cc260 0x7f28bc1f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28c273a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1f4380 con 0x7f28bc06bb10 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28c273a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1581e0 con 0x7f28bc06bef0 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28c273a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1f1070 con 0x7f28bc0cc260 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bbfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 0x7f28bc157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bbfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 0x7f28bc157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46532/0 (socket says 172.21.15.172:46532) 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bbfff640 1 -- 172.21.15.172:0/1076594736 learned_addr learned my addr 172.21.15.172:0/1076594736 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28c0cd7640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc0cc260 0x7f28bc1f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bb7fe640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28bc06bef0 0x7f28bc0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bbfff640 1 -- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc0cc260 msgr2=0x7f28bc1f4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bbfff640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc0cc260 0x7f28bc1f4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bbfff640 1 -- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28bc06bef0 msgr2=0x7f28bc0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bbfff640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28bc06bef0 0x7f28bc0d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bbfff640 1 -- 172.21.15.172:0/1076594736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac009400 con 0x7f28bc06bb10 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28bb7fe640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28bc06bef0 0x7f28bc0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:34.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.993+0000 7f28c0cd7640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc0cc260 0x7f28bc1f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:29:34.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.994+0000 7f28bbfff640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 0x7f28bc157ca0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0098c0 tx=0x7f28ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.994+0000 7f28b8ff9640 1 -- 172.21.15.172:0/1076594736 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac03e040 con 0x7f28bc06bb10 2024-03-16T19:29:34.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.994+0000 7f28b8ff9640 1 -- 172.21.15.172:0/1076594736 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac039cb0 con 0x7f28bc06bb10 2024-03-16T19:29:34.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.994+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc1f1350 con 0x7f28bc06bb10 2024-03-16T19:29:34.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.994+0000 7f28b8ff9640 1 -- 172.21.15.172:0/1076594736 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac044630 con 0x7f28bc06bb10 2024-03-16T19:29:34.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.994+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc1f1910 con 0x7f28bc06bb10 2024-03-16T19:29:34.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.995+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc1f4550 con 0x7f28bc06bb10 2024-03-16T19:29:34.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.995+0000 7f28b8ff9640 1 -- 172.21.15.172:0/1076594736 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f28ac04d3b0 con 0x7f28bc06bb10 2024-03-16T19:29:34.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.996+0000 7f28b8ff9640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28a806b3b0 0x7f28a806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:34.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.996+0000 7f28b8ff9640 1 -- 172.21.15.172:0/1076594736 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28ac0ac130 con 0x7f28bc06bb10 2024-03-16T19:29:34.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.996+0000 7f28bb7fe640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28a806b3b0 0x7f28a806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:34.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:34.996+0000 7f28bb7fe640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28a806b3b0 0x7f28a806d870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28bc14e6f0 tx=0x7f28b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:34.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.000+0000 7f28b8ff9640 1 -- 172.21.15.172:0/1076594736 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28ac076d00 con 0x7f28bc06bb10 2024-03-16T19:29:35.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.207+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28bc0681c0 con 0x7f28bc06bb10 2024-03-16T19:29:35.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.208+0000 7f28b8ff9640 1 -- 172.21.15.172:0/1076594736 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6561 (secure 0 0 0) 0x7f28ac07a3c0 con 0x7f28bc06bb10 2024-03-16T19:29:35.189 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:35.189 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1028682606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1028682606},{"type":"v1","addr":"172.21.15.174:6833","nonce":1028682606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6836/902696086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":902696086},{"type":"v1","addr":"172.21.15.174:6836","nonce":902696086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/864851052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":864851052},{"type":"v1","addr":"172.21.15.172:6836","nonce":864851052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/420882750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":420882750},{"type":"v1","addr":"172.21.15.172:6837","nonce":420882750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/2853574742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":2853574742},{"type":"v1","addr":"172.21.15.172:6839","nonce":2853574742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:29:25.674964+0000","modified":"2024-03-16T19:29:31.368977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/2442775894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6835","nonce":2442775894},{"type":"v1","addr":"172.21.15.174:6837","nonce":2442775894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-16T19:29:35.189 INFO:teuthology.orchestra.run.smithi172.stderr:dumped fsmap epoch 8 2024-03-16T19:29:35.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28a806b3b0 msgr2=0x7f28a806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28a806b3b0 0x7f28a806d870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28bc14e6f0 tx=0x7f28b0008040 comp rx=0 tx=0).stop 2024-03-16T19:29:35.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 msgr2=0x7f28bc157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 0x7f28bc157ca0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0098c0 tx=0x7f28ac004730 comp rx=0 tx=0).stop 2024-03-16T19:29:35.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 shutdown_connections 2024-03-16T19:29:35.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f28a806b3b0 0x7f28a806d870 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28bc0cc260 0x7f28bc1f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28bc06bef0 0x7f28bc0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 --2- 172.21.15.172:0/1076594736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28bc06bb10 0x7f28bc157ca0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 >> 172.21.15.172:0/1076594736 conn(0x7f28bc067250 msgr2=0x7f28bc1520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:35.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 shutdown_connections 2024-03-16T19:29:35.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.210+0000 7f28c273a640 1 -- 172.21.15.172:0/1076594736 wait complete. 2024-03-16T19:29:35.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:35.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.322+0000 7f41dfc9d640 1 Processor -- start 2024-03-16T19:29:35.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.322+0000 7f41dfc9d640 1 -- start start 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41dfc9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8152c00 0x7f41d80c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41dfc9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f41d8159220 0x7f41d8159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41dfc9d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f41d815a150 0x7f41d80ce210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41dfc9d640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d806f6b0 con 0x7f41d815a150 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41dfc9d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d80c4f60 con 0x7f41d8159220 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41dfc9d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80c3440 con 0x7f41d8152c00 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41de23a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f41d815a150 0x7f41d80ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41de23a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f41d815a150 0x7f41d80ce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46534/0 (socket says 172.21.15.172:46534) 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41de23a640 1 -- 172.21.15.172:0/2250317590 learned_addr learned my addr 172.21.15.172:0/2250317590 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41dd238640 1 --2- 172.21.15.172:0/2250317590 >> v2:172.21.15.174:3300/0 conn(0x7f41d8159220 0x7f41d8159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41dda39640 1 --2- 172.21.15.172:0/2250317590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8152c00 0x7f41d80c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41de23a640 1 -- 172.21.15.172:0/2250317590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8152c00 msgr2=0x7f41d80c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:35.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41de23a640 1 --2- 172.21.15.172:0/2250317590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8152c00 0x7f41d80c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41de23a640 1 -- 172.21.15.172:0/2250317590 >> v2:172.21.15.174:3300/0 conn(0x7f41d8159220 msgr2=0x7f41d8159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:35.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41de23a640 1 --2- 172.21.15.172:0/2250317590 >> v2:172.21.15.174:3300/0 conn(0x7f41d8159220 0x7f41d8159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.323+0000 7f41de23a640 1 -- 172.21.15.172:0/2250317590 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d80c5440 con 0x7f41d815a150 2024-03-16T19:29:35.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.324+0000 7f41de23a640 1 --2- 172.21.15.172:0/2250317590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 0x7f41d80ce210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f41d8067ba0 tx=0x7f41d002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ba7898fd9f8d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:35.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.324+0000 7f41dca37640 1 -- 172.21.15.172:0/2250317590 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d002cb00 con 0x7f41d815a150 2024-03-16T19:29:35.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.324+0000 7f41dca37640 1 -- 172.21.15.172:0/2250317590 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d002cc60 con 0x7f41d815a150 2024-03-16T19:29:35.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.324+0000 7f41dca37640 1 -- 172.21.15.172:0/2250317590 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d00343f0 con 0x7f41d815a150 2024-03-16T19:29:35.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.324+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2250317590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 msgr2=0x7f41d80ce210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.324+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2250317590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 0x7f41d80ce210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f41d8067ba0 tx=0x7f41d002bf60 comp rx=0 tx=0).stop 2024-03-16T19:29:35.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.325+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2250317590 shutdown_connections 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.325+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2250317590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 0x7f41d80ce210 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.325+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2250317590 >> v2:172.21.15.174:3300/0 conn(0x7f41d8159220 0x7f41d8159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.325+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2250317590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8152c00 0x7f41d80c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.325+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2250317590 >> 172.21.15.172:0/2250317590 conn(0x7f41d8066c20 msgr2=0x7f41d8067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.325+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2250317590 shutdown_connections 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.325+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2250317590 wait complete. 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.326+0000 7f41dfc9d640 1 Processor -- start 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.326+0000 7f41dfc9d640 1 -- start start 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.326+0000 7f41dfc9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8152c00 0x7f41d8153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.326+0000 7f41dfc9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8159220 0x7f41d81f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.326+0000 7f41dfc9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 0x7f41d81f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.326+0000 7f41dfc9d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80c7710 con 0x7f41d815a150 2024-03-16T19:29:35.310 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.326+0000 7f41dfc9d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8156bd0 con 0x7f41d8152c00 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.326+0000 7f41dfc9d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8154060 con 0x7f41d8159220 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41de23a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 0x7f41d81f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41de23a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 0x7f41d81f1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46536/0 (socket says 172.21.15.172:46536) 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41de23a640 1 -- 172.21.15.172:0/2999257494 learned_addr learned my addr 172.21.15.172:0/2999257494 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41dd238640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8159220 0x7f41d81f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41dda39640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8152c00 0x7f41d8153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41de23a640 1 -- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8159220 msgr2=0x7f41d81f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41de23a640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8159220 0x7f41d81f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41de23a640 1 -- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8152c00 msgr2=0x7f41d8153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41de23a640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8152c00 0x7f41d8153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41de23a640 1 -- 172.21.15.172:0/2999257494 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d0009b30 con 0x7f41d815a150 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41dda39640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8152c00 0x7f41d8153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41dd238640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8159220 0x7f41d81f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:35.311 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41de23a640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 0x7f41d81f1ff0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f41d0036040 tx=0x7f41d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:35.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41ce7fc640 1 -- 172.21.15.172:0/2999257494 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0009510 con 0x7f41d815a150 2024-03-16T19:29:35.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41ce7fc640 1 -- 172.21.15.172:0/2999257494 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d003e450 con 0x7f41d815a150 2024-03-16T19:29:35.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d8155ce0 con 0x7f41d815a150 2024-03-16T19:29:35.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41ce7fc640 1 -- 172.21.15.172:0/2999257494 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d003e800 con 0x7f41d815a150 2024-03-16T19:29:35.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.327+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d81562a0 con 0x7f41d815a150 2024-03-16T19:29:35.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.328+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d8156d10 con 0x7f41d815a150 2024-03-16T19:29:35.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.329+0000 7f41ce7fc640 1 -- 172.21.15.172:0/2999257494 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f41d003e960 con 0x7f41d815a150 2024-03-16T19:29:35.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.329+0000 7f41ce7fc640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.329+0000 7f41ce7fc640 1 -- 172.21.15.172:0/2999257494 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41d00ac870 con 0x7f41d815a150 2024-03-16T19:29:35.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.329+0000 7f41dda39640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.329+0000 7f41dda39640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009a40 tx=0x7f41d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:35.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.333+0000 7f41ce7fc640 1 -- 172.21.15.172:0/2999257494 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41d8156d10 con 0x7f41d815a150 2024-03-16T19:29:35.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.507+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41d8156d10 con 0x7f41d815a150 2024-03-16T19:29:35.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.508+0000 7f41ce7fc640 1 -- 172.21.15.172:0/2999257494 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f41d8156d10 con 0x7f41d815a150 2024-03-16T19:29:35.489 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:35.489 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":21,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:27.336190+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:35.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 msgr2=0x7f41bc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009a40 tx=0x7f41d4008040 comp rx=0 tx=0).stop 2024-03-16T19:29:35.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 msgr2=0x7f41d81f1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 0x7f41d81f1ff0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f41d0036040 tx=0x7f41d0004a40 comp rx=0 tx=0).stop 2024-03-16T19:29:35.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 shutdown_connections 2024-03-16T19:29:35.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41d815a150 0x7f41d81f1ff0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f41bc06b3e0 0x7f41bc06d8a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41d8159220 0x7f41d81f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 --2- 172.21.15.172:0/2999257494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8152c00 0x7f41d8153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 >> 172.21.15.172:0/2999257494 conn(0x7f41d8066c20 msgr2=0x7f41d8151fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:35.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 shutdown_connections 2024-03-16T19:29:35.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.510+0000 7f41dfc9d640 1 -- 172.21.15.172:0/2999257494 wait complete. 2024-03-16T19:29:35.502 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:29:14.242742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:29:24.121132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:29:24.651099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T19:29:35.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:29:35.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.624+0000 7ff28526f640 1 Processor -- start 2024-03-16T19:29:35.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.624+0000 7ff28526f640 1 -- start start 2024-03-16T19:29:35.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff28526f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff280154e00 0x7ff28006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff28526f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2800b5f10 0x7ff2800caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff28526f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28015b990 con 0x7ff28006b500 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff28526f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff28015b570 con 0x7ff280154e00 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff28526f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2801551e0 con 0x7ff2800b5f10 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27f7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff28006b500 0x7ff28015df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56290/0 (socket says 172.21.15.172:56290) 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27f7fe640 1 -- 172.21.15.172:0/2392597508 learned_addr learned my addr 172.21.15.172:0/2392597508 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27e7fc640 1 --2- 172.21.15.172:0/2392597508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2800b5f10 0x7ff2800caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27effd640 1 --2- 172.21.15.172:0/2392597508 >> v2:172.21.15.174:3300/0 conn(0x7ff280154e00 0x7ff28006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27dffb640 1 -- 172.21.15.172:0/2392597508 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365706930 0 0) 0x7ff28015b990 con 0x7ff28006b500 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27dffb640 1 -- 172.21.15.172:0/2392597508 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff268003520 con 0x7ff28006b500 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27effd640 1 -- 172.21.15.172:0/2392597508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2800b5f10 msgr2=0x7ff2800caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27effd640 1 --2- 172.21.15.172:0/2392597508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2800b5f10 0x7ff2800caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27dffb640 1 -- 172.21.15.172:0/2392597508 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2585490044 0 0) 0x7ff268003520 con 0x7ff28006b500 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27effd640 1 -- 172.21.15.172:0/2392597508 >> v1:172.21.15.172:6789/0 conn(0x7ff28006b500 legacy=0x7ff28015df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27effd640 1 -- 172.21.15.172:0/2392597508 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800cc090 con 0x7ff280154e00 2024-03-16T19:29:35.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.625+0000 7ff27e7fc640 1 --2- 172.21.15.172:0/2392597508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2800b5f10 0x7ff2800caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:35.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.626+0000 7ff27effd640 1 --2- 172.21.15.172:0/2392597508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff280154e00 0x7ff28006afc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff2700097a0 tx=0x7ff27002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a97297be9ec91391 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:35.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.626+0000 7ff27dffb640 1 -- 172.21.15.172:0/2392597508 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27002f050 con 0x7ff280154e00 2024-03-16T19:29:35.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.626+0000 7ff27dffb640 1 -- 172.21.15.172:0/2392597508 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff270031500 con 0x7ff280154e00 2024-03-16T19:29:35.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.626+0000 7ff27dffb640 1 -- 172.21.15.172:0/2392597508 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2780019f0 con 0x7ff280154e00 2024-03-16T19:29:35.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.627+0000 7ff28526f640 1 -- 172.21.15.172:0/2392597508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff280154e00 msgr2=0x7ff28006afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.627+0000 7ff28526f640 1 --2- 172.21.15.172:0/2392597508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff280154e00 0x7ff28006afc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff2700097a0 tx=0x7ff27002b820 comp rx=0 tx=0).stop 2024-03-16T19:29:35.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.627+0000 7ff28526f640 1 -- 172.21.15.172:0/2392597508 shutdown_connections 2024-03-16T19:29:35.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.627+0000 7ff28526f640 1 --2- 172.21.15.172:0/2392597508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2800b5f10 0x7ff2800caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.627+0000 7ff28526f640 1 --2- 172.21.15.172:0/2392597508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff280154e00 0x7ff28006afc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.627+0000 7ff28526f640 1 -- 172.21.15.172:0/2392597508 >> 172.21.15.172:0/2392597508 conn(0x7ff280066c20 msgr2=0x7ff280067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:35.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.627+0000 7ff28526f640 1 -- 172.21.15.172:0/2392597508 shutdown_connections 2024-03-16T19:29:35.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.627+0000 7ff28526f640 1 -- 172.21.15.172:0/2392597508 wait complete. 2024-03-16T19:29:35.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.628+0000 7ff28526f640 1 Processor -- start 2024-03-16T19:29:35.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.628+0000 7ff28526f640 1 -- start start 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff28526f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff28006b500 0x7ff280157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff28526f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2800b5f10 0x7ff280145a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff28526f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff280154e00 0x7ff28014aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff28526f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800c3930 con 0x7ff280154e00 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff28526f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff280155330 con 0x7ff2800b5f10 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff28526f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff280156140 con 0x7ff28006b500 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff280154e00 0x7ff28014aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff280154e00 0x7ff28014aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46538/0 (socket says 172.21.15.172:46538) 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27f7fe640 1 -- 172.21.15.172:0/2919118274 learned_addr learned my addr 172.21.15.172:0/2919118274 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27e7fc640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2800b5f10 0x7ff280145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27effd640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff28006b500 0x7ff280157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27f7fe640 1 -- 172.21.15.172:0/2919118274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff28006b500 msgr2=0x7ff280157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27f7fe640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff28006b500 0x7ff280157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27f7fe640 1 -- 172.21.15.172:0/2919118274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2800b5f10 msgr2=0x7ff280145a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27f7fe640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2800b5f10 0x7ff280145a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27f7fe640 1 -- 172.21.15.172:0/2919118274 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff274009400 con 0x7ff280154e00 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.629+0000 7ff27f7fe640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff280154e00 0x7ff28014aca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff278001560 tx=0x7ff27800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.630+0000 7ff2677fe640 1 -- 172.21.15.172:0/2919118274 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27800ce90 con 0x7ff280154e00 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.630+0000 7ff2677fe640 1 -- 172.21.15.172:0/2919118274 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff278012400 con 0x7ff280154e00 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.630+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff270009400 con 0x7ff280154e00 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.630+0000 7ff2677fe640 1 -- 172.21.15.172:0/2919118274 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2780127b0 con 0x7ff280154e00 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.630+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2800cef70 con 0x7ff280154e00 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.631+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff280067c30 con 0x7ff280154e00 2024-03-16T19:29:35.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.631+0000 7ff2677fe640 1 -- 172.21.15.172:0/2919118274 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff278012910 con 0x7ff280154e00 2024-03-16T19:29:35.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.632+0000 7ff2677fe640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff26c06b3b0 0x7ff26c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.632+0000 7ff2677fe640 1 -- 172.21.15.172:0/2919118274 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff27808a360 con 0x7ff280154e00 2024-03-16T19:29:35.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.632+0000 7ff27effd640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff26c06b3b0 0x7ff26c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.632+0000 7ff27effd640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff26c06b3b0 0x7ff26c06d870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff270035000 tx=0x7ff270008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:35.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.636+0000 7ff2677fe640 1 -- 172.21.15.172:0/2919118274 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff278055050 con 0x7ff280154e00 2024-03-16T19:29:35.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.808+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2800cc710 con 0x7ff280154e00 2024-03-16T19:29:35.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.808+0000 7ff2677fe640 1 -- 172.21.15.172:0/2919118274 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff278058710 con 0x7ff280154e00 2024-03-16T19:29:35.789 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 1 2024-03-16T19:29:35.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff26c06b3b0 msgr2=0x7ff26c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff26c06b3b0 0x7ff26c06d870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff270035000 tx=0x7ff270008030 comp rx=0 tx=0).stop 2024-03-16T19:29:35.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff280154e00 msgr2=0x7ff28014aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff280154e00 0x7ff28014aca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff278001560 tx=0x7ff27800a430 comp rx=0 tx=0).stop 2024-03-16T19:29:35.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 shutdown_connections 2024-03-16T19:29:35.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff280154e00 0x7ff28014aca0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2800b5f10 0x7ff280145a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff26c06b3b0 0x7ff26c06d870 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 --2- 172.21.15.172:0/2919118274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff28006b500 0x7ff280157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 >> 172.21.15.172:0/2919118274 conn(0x7ff280066c20 msgr2=0x7ff2800cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:35.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 shutdown_connections 2024-03-16T19:29:35.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.810+0000 7ff28526f640 1 -- 172.21.15.172:0/2919118274 wait complete. 2024-03-16T19:29:35.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T19:29:35.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.925+0000 7fd9bf495640 1 Processor -- start 2024-03-16T19:29:35.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.925+0000 7fd9bf495640 1 -- start start 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9bf495640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd9b806c500 0x7fd9b80ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9bf495640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b806f4f0 0x7fd9b80ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9bf495640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b806e010 con 0x7fd9b8158150 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9bf495640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b806f0f0 con 0x7fd9b806c500 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9bf495640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80c8f20 con 0x7fd9b806f4f0 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9bd231640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd9b8158150 0x7fd9b806e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56302/0 (socket says 172.21.15.172:56302) 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9bd231640 1 -- 172.21.15.172:0/1326998608 learned_addr learned my addr 172.21.15.172:0/1326998608 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9bda32640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b806f4f0 0x7fd9b80ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234689509 0 0) 0x7fd9b806e010 con 0x7fd9b8158150 2024-03-16T19:29:35.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9bca30640 1 --2- 172.21.15.172:0/1326998608 >> v2:172.21.15.174:3300/0 conn(0x7fd9b806c500 0x7fd9b80ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a4003520 con 0x7fd9b8158150 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.926+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3242736937 0 0) 0x7fd9a4003520 con 0x7fd9b8158150 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b806f4f0 msgr2=0x7fd9b80ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b806f4f0 0x7fd9b80ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 >> v2:172.21.15.174:3300/0 conn(0x7fd9b806c500 msgr2=0x7fd9b80ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 --2- 172.21.15.172:0/1326998608 >> v2:172.21.15.174:3300/0 conn(0x7fd9b806c500 0x7fd9b80ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b815b580 con 0x7fd9b8158150 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7fd9b40019f0 con 0x7fd9b8158150 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 >> v1:172.21.15.172:6789/0 conn(0x7fd9b8158150 legacy=0x7fd9b806e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9a4003970 0x7fd9a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9a40047c0 0x7fd9a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b806f4f0 0x7fd9a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b806f0f0 con 0x7fd9a4003970 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80c8f20 con 0x7fd9a40047c0 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a400b980 con 0x7fd9b806f4f0 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9bd231640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9a4003970 0x7fd9a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9bca30640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9a40047c0 0x7fd9a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9bda32640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b806f4f0 0x7fd9a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9bd231640 1 -- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b806f4f0 msgr2=0x7fd9a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9bd231640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b806f4f0 0x7fd9a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9bd231640 1 -- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9a40047c0 msgr2=0x7fd9a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9bd231640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9a40047c0 0x7fd9a4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.927+0000 7fd9bd231640 1 -- 172.21.15.172:0/1326998608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4003520 con 0x7fd9a4003970 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.928+0000 7fd9bd231640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9a4003970 0x7fd9a4003d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400a860 tx=0x7fd9a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a79cee6bc16c30e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.928+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4034630 con 0x7fd9a4003970 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.928+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b402f070 con 0x7fd9a4003970 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.928+0000 7fd9affff640 1 -- 172.21.15.172:0/1326998608 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40349d0 con 0x7fd9a4003970 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.928+0000 7fd9bf495640 1 -- 172.21.15.172:0/1326998608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9a4003970 msgr2=0x7fd9a4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.928+0000 7fd9bf495640 1 --2- 172.21.15.172:0/1326998608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9a4003970 0x7fd9a4003d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400a860 tx=0x7fd9a40057f0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.928+0000 7fd9bda32640 1 -- 172.21.15.172:0/1326998608 reap_dead start 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.928+0000 7fd9bf495640 1 -- 172.21.15.172:0/1326998608 shutdown_connections 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.928+0000 7fd9bf495640 1 -- 172.21.15.172:0/1326998608 >> 172.21.15.172:0/1326998608 conn(0x7fd9b8066c20 msgr2=0x7fd9b8067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.929+0000 7fd9bf495640 1 -- 172.21.15.172:0/1326998608 shutdown_connections 2024-03-16T19:29:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.929+0000 7fd9bf495640 1 -- 172.21.15.172:0/1326998608 wait complete. 2024-03-16T19:29:35.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bf495640 1 Processor -- start 2024-03-16T19:29:35.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bf495640 1 -- start start 2024-03-16T19:29:35.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bf495640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9a4003970 0x7fd9b814ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bf495640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b80ce150 0x7fd9b814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bf495640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9b81f4540 0x7fd9b80ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bf495640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b814f980 con 0x7fd9b81f4540 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bf495640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80c5770 con 0x7fd9a4003970 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bf495640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b81f40b0 con 0x7fd9b80ce150 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bda32640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9b81f4540 0x7fd9b80ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bda32640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9b81f4540 0x7fd9b80ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46546/0 (socket says 172.21.15.172:46546) 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bda32640 1 -- 172.21.15.172:0/1038060594 learned_addr learned my addr 172.21.15.172:0/1038060594 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bca30640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b80ce150 0x7fd9b814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.930+0000 7fd9bd231640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9a4003970 0x7fd9b814ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9bda32640 1 -- 172.21.15.172:0/1038060594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b80ce150 msgr2=0x7fd9b814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9bda32640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b80ce150 0x7fd9b814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9bda32640 1 -- 172.21.15.172:0/1038060594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9a4003970 msgr2=0x7fd9b814ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9bda32640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9a4003970 0x7fd9b814ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9bda32640 1 -- 172.21.15.172:0/1038060594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b8150120 con 0x7fd9b81f4540 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9bda32640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9b81f4540 0x7fd9b80ca4e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd9b80ced00 tx=0x7fd9b0011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9adffb640 1 -- 172.21.15.172:0/1038060594 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0004030 con 0x7fd9b81f4540 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9adffb640 1 -- 172.21.15.172:0/1038060594 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b0004190 con 0x7fd9b81f4540 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b814f3c0 con 0x7fd9b81f4540 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9adffb640 1 -- 172.21.15.172:0/1038060594 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0004520 con 0x7fd9b81f4540 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.931+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b81f3a60 con 0x7fd9b81f4540 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.932+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b806bf60 con 0x7fd9b81f4540 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.933+0000 7fd9adffb640 1 -- 172.21.15.172:0/1038060594 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd9b000f740 con 0x7fd9b81f4540 2024-03-16T19:29:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.933+0000 7fd9adffb640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd99c06b3b0 0x7fd99c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.933+0000 7fd9adffb640 1 -- 172.21.15.172:0/1038060594 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9b0089ce0 con 0x7fd9b81f4540 2024-03-16T19:29:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.933+0000 7fd9bd231640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd99c06b3b0 0x7fd99c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.933+0000 7fd9bd231640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd99c06b3b0 0x7fd99c06d870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4034000 tx=0x7fd9b4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:35.937+0000 7fd9adffb640 1 -- 172.21.15.172:0/1038060594 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9b00548b0 con 0x7fd9b81f4540 2024-03-16T19:29:36.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.108+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd9b80681c0 con 0x7fd9b81f4540 2024-03-16T19:29:36.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.108+0000 7fd9adffb640 1 -- 172.21.15.172:0/1038060594 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd9b0000e60 con 0x7fd9b81f4540 2024-03-16T19:29:36.089 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 32 2024-03-16T19:29:36.091 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd99c06b3b0 msgr2=0x7fd99c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.091 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd99c06b3b0 0x7fd99c06d870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4034000 tx=0x7fd9b4009030 comp rx=0 tx=0).stop 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9b81f4540 msgr2=0x7fd9b80ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9b81f4540 0x7fd9b80ca4e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd9b80ced00 tx=0x7fd9b0011c30 comp rx=0 tx=0).stop 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 shutdown_connections 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd9b81f4540 0x7fd9b80ca4e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd99c06b3b0 0x7fd99c06d870 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9b80ce150 0x7fd9b814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 --2- 172.21.15.172:0/1038060594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9a4003970 0x7fd9b814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 >> 172.21.15.172:0/1038060594 conn(0x7fd9b8066c20 msgr2=0x7fd9b80b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 shutdown_connections 2024-03-16T19:29:36.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.110+0000 7fd9bf495640 1 -- 172.21.15.172:0/1038060594 wait complete. 2024-03-16T19:29:36.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T19:29:36.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.225+0000 7fb34dde9640 1 Processor -- start 2024-03-16T19:29:36.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.225+0000 7fb34dde9640 1 -- start start 2024-03-16T19:29:36.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.225+0000 7fb34dde9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 0x7fb34806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb34dde9640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3480cba70 con 0x7fb34815e3b0 2024-03-16T19:29:36.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb34dde9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34806b750 con 0x7fb3480c7560 2024-03-16T19:29:36.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb34dde9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34806b140 con 0x7fb34815dfd0 2024-03-16T19:29:36.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb347fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb34815e3b0 0x7fb34815e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56308/0 (socket says 172.21.15.172:56308) 2024-03-16T19:29:36.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb347fff640 1 -- 172.21.15.172:0/3666968904 learned_addr learned my addr 172.21.15.172:0/3666968904 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:36.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb3477fe640 1 --2- 172.21.15.172:0/3666968904 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 0x7fb34806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb3467fc640 1 -- 172.21.15.172:0/3666968904 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824101580 0 0) 0x7fb340001490 con 0x7fb34815e3b0 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb3467fc640 1 -- 172.21.15.172:0/3666968904 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb330003520 con 0x7fb34815e3b0 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb3467fc640 1 -- 172.21.15.172:0/3666968904 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542888540 0 0) 0x7fb34806b750 con 0x7fb3480c7560 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb3467fc640 1 -- 172.21.15.172:0/3666968904 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb340001490 con 0x7fb3480c7560 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb3477fe640 1 -- 172.21.15.172:0/3666968904 >> v1:172.21.15.174:6789/0 conn(0x7fb3480c7560 legacy=0x7fb3480caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb3477fe640 1 -- 172.21.15.172:0/3666968904 >> v1:172.21.15.172:6789/0 conn(0x7fb34815e3b0 legacy=0x7fb34815e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.226+0000 7fb3477fe640 1 -- 172.21.15.172:0/3666968904 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34806c120 con 0x7fb34815dfd0 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.227+0000 7fb3477fe640 1 --2- 172.21.15.172:0/3666968904 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 0x7fb34806bbe0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb3380097d0 tx=0x7fb33802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e92c94c9b709e00f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.227+0000 7fb3467fc640 1 -- 172.21.15.172:0/3666968904 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338030880 con 0x7fb34815dfd0 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.227+0000 7fb3467fc640 1 -- 172.21.15.172:0/3666968904 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3380309e0 con 0x7fb34815dfd0 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.227+0000 7fb3467fc640 1 -- 172.21.15.172:0/3666968904 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338030d50 con 0x7fb34815dfd0 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.227+0000 7fb34dde9640 1 -- 172.21.15.172:0/3666968904 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 msgr2=0x7fb34806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.227+0000 7fb34dde9640 1 --2- 172.21.15.172:0/3666968904 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 0x7fb34806bbe0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb3380097d0 tx=0x7fb33802b9c0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.228+0000 7fb34dde9640 1 -- 172.21.15.172:0/3666968904 shutdown_connections 2024-03-16T19:29:36.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.228+0000 7fb34dde9640 1 --2- 172.21.15.172:0/3666968904 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 0x7fb34806bbe0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.228+0000 7fb34dde9640 1 -- 172.21.15.172:0/3666968904 >> 172.21.15.172:0/3666968904 conn(0x7fb3480d0340 msgr2=0x7fb3480d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:36.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.228+0000 7fb34dde9640 1 -- 172.21.15.172:0/3666968904 shutdown_connections 2024-03-16T19:29:36.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.228+0000 7fb34dde9640 1 -- 172.21.15.172:0/3666968904 wait complete. 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.229+0000 7fb34dde9640 1 Processor -- start 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.229+0000 7fb34dde9640 1 -- start start 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.229+0000 7fb34dde9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3480c7560 0x7fb34814bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.229+0000 7fb34dde9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 0x7fb348145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb34dde9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb34815e3b0 0x7fb3481459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb34dde9640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3481f4ee0 con 0x7fb3480c7560 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb34dde9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3481f3ff0 con 0x7fb34815e3b0 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb34dde9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3481f42d0 con 0x7fb34815dfd0 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb3477fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3480c7560 0x7fb34814bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb3477fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3480c7560 0x7fb34814bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46558/0 (socket says 172.21.15.172:46558) 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb3477fe640 1 -- 172.21.15.172:0/1304839761 learned_addr learned my addr 172.21.15.172:0/1304839761 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:36.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb347fff640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb34815e3b0 0x7fb3481459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb346ffd640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 0x7fb348145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb3477fe640 1 -- 172.21.15.172:0/1304839761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 msgr2=0x7fb348145460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb3477fe640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 0x7fb348145460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb3477fe640 1 -- 172.21.15.172:0/1304839761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb34815e3b0 msgr2=0x7fb3481459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb3477fe640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb34815e3b0 0x7fb3481459a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb3477fe640 1 -- 172.21.15.172:0/1304839761 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb338009400 con 0x7fb3480c7560 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb3477fe640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3480c7560 0x7fb34814bfd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb33802ba00 tx=0x7fb3380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb32ffff640 1 -- 172.21.15.172:0/1304839761 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33803f070 con 0x7fb3480c7560 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb32ffff640 1 -- 172.21.15.172:0/1304839761 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb338003610 con 0x7fb3480c7560 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3481f45b0 con 0x7fb3480c7560 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb32ffff640 1 -- 172.21.15.172:0/1304839761 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338003980 con 0x7fb3480c7560 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.230+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3481f4b70 con 0x7fb3480c7560 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.231+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3481f4d00 con 0x7fb3480c7560 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.232+0000 7fb32ffff640 1 -- 172.21.15.172:0/1304839761 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb338003ae0 con 0x7fb3480c7560 2024-03-16T19:29:36.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.232+0000 7fb32ffff640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb33406b3b0 0x7fb33406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.232+0000 7fb32ffff640 1 -- 172.21.15.172:0/1304839761 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3380abdc0 con 0x7fb3480c7560 2024-03-16T19:29:36.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.232+0000 7fb346ffd640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb33406b3b0 0x7fb33406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.232+0000 7fb346ffd640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb33406b3b0 0x7fb33406d870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb33c007fd0 tx=0x7fb33c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:36.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.236+0000 7fb32ffff640 1 -- 172.21.15.172:0/1304839761 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb338076ad0 con 0x7fb3480c7560 2024-03-16T19:29:36.393 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.411+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3480c9680 con 0x7fb3480c7560 2024-03-16T19:29:36.393 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.411+0000 7fb32ffff640 1 -- 172.21.15.172:0/1304839761 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb33807a190 con 0x7fb3480c7560 2024-03-16T19:29:36.394 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 64 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb33406b3b0 msgr2=0x7fb33406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb33406b3b0 0x7fb33406d870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb33c007fd0 tx=0x7fb33c009040 comp rx=0 tx=0).stop 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3480c7560 msgr2=0x7fb34814bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3480c7560 0x7fb34814bfd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb33802ba00 tx=0x7fb3380046c0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 shutdown_connections 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb34815e3b0 0x7fb3481459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb33406b3b0 0x7fb33406d870 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb34815dfd0 0x7fb348145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 --2- 172.21.15.172:0/1304839761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3480c7560 0x7fb34814bfd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 >> 172.21.15.172:0/1304839761 conn(0x7fb3480d0340 msgr2=0x7fb34815d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:36.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 shutdown_connections 2024-03-16T19:29:36.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.413+0000 7fb34dde9640 1 -- 172.21.15.172:0/1304839761 wait complete. 2024-03-16T19:29:36.405 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:29:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.524+0000 7f9cda055640 1 Processor -- start 2024-03-16T19:29:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.524+0000 7f9cda055640 1 -- start start 2024-03-16T19:29:36.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.524+0000 7f9cda055640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9cd406bb10 0x7f9cd40cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cda055640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9cd406bef0 0x7f9cd40cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cda055640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd40cc720 con 0x7f9cd40cc260 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cda055640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd40cc9c0 con 0x7f9cd406bb10 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cda055640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd415de30 con 0x7f9cd406bef0 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cd37fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9cd406bb10 0x7f9cd40cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cd3fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9cd40cc260 0x7f9cd4154660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56312/0 (socket says 172.21.15.172:56312) 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cd3fff640 1 -- 172.21.15.172:0/1294237972 learned_addr learned my addr 172.21.15.172:0/1294237972 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cd2ffd640 1 --2- 172.21.15.172:0/1294237972 >> v2:172.21.15.174:3300/0 conn(0x7f9cd406bef0 0x7f9cd40cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cd37fe640 1 -- 172.21.15.172:0/1294237972 >> v2:172.21.15.174:3300/0 conn(0x7f9cd406bef0 msgr2=0x7f9cd40cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cd37fe640 1 --2- 172.21.15.172:0/1294237972 >> v2:172.21.15.174:3300/0 conn(0x7f9cd406bef0 0x7f9cd40cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cd37fe640 1 -- 172.21.15.172:0/1294237972 >> v1:172.21.15.172:6789/0 conn(0x7f9cd40cc260 legacy=0x7f9cd4154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cd27fc640 1 -- 172.21.15.172:0/1294237972 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997278170 0 0) 0x7f9cd40cc720 con 0x7f9cd40cc260 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.525+0000 7f9cd37fe640 1 -- 172.21.15.172:0/1294237972 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd415e110 con 0x7f9cd406bb10 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cd37fe640 1 --2- 172.21.15.172:0/1294237972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 0x7f9cd40cae50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9cd40ca7d0 tx=0x7f9cc402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60939aef72f816e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cd27fc640 1 -- 172.21.15.172:0/1294237972 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4030610 con 0x7f9cd406bb10 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cd27fc640 1 -- 172.21.15.172:0/1294237972 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc4030770 con 0x7f9cd406bb10 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cd27fc640 1 -- 172.21.15.172:0/1294237972 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4030b00 con 0x7f9cd406bb10 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cda055640 1 -- 172.21.15.172:0/1294237972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 msgr2=0x7f9cd40cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cda055640 1 --2- 172.21.15.172:0/1294237972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 0x7f9cd40cae50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9cd40ca7d0 tx=0x7f9cc402b960 comp rx=0 tx=0).stop 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cda055640 1 -- 172.21.15.172:0/1294237972 shutdown_connections 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cda055640 1 --2- 172.21.15.172:0/1294237972 >> v2:172.21.15.174:3300/0 conn(0x7f9cd406bef0 0x7f9cd40cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cda055640 1 --2- 172.21.15.172:0/1294237972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 0x7f9cd40cae50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cda055640 1 -- 172.21.15.172:0/1294237972 >> 172.21.15.172:0/1294237972 conn(0x7f9cd4067250 msgr2=0x7f9cd40d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cda055640 1 -- 172.21.15.172:0/1294237972 shutdown_connections 2024-03-16T19:29:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.526+0000 7f9cda055640 1 -- 172.21.15.172:0/1294237972 wait complete. 2024-03-16T19:29:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.528+0000 7f9cda055640 1 Processor -- start 2024-03-16T19:29:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.528+0000 7f9cda055640 1 -- start start 2024-03-16T19:29:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.528+0000 7f9cda055640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 0x7f9cd40cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.528+0000 7f9cda055640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cd406bef0 0x7f9cd40b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.528+0000 7f9cda055640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cd40cc260 0x7f9cd4145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.528+0000 7f9cda055640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd41458f0 con 0x7f9cd406bb10 2024-03-16T19:29:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.528+0000 7f9cda055640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd41f0b50 con 0x7f9cd406bef0 2024-03-16T19:29:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.528+0000 7f9cda055640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd41f0e30 con 0x7f9cd40cc260 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.528+0000 7f9cd37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 0x7f9cd40cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 0x7f9cd40cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46580/0 (socket says 172.21.15.172:46580) 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd37fe640 1 -- 172.21.15.172:0/2365374404 learned_addr learned my addr 172.21.15.172:0/2365374404 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd2ffd640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cd406bef0 0x7f9cd40b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd3fff640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cd40cc260 0x7f9cd4145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd37fe640 1 -- 172.21.15.172:0/2365374404 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cd40cc260 msgr2=0x7f9cd4145dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd37fe640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cd40cc260 0x7f9cd4145dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd37fe640 1 -- 172.21.15.172:0/2365374404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cd406bef0 msgr2=0x7f9cd40b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd37fe640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cd406bef0 0x7f9cd40b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd37fe640 1 -- 172.21.15.172:0/2365374404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc4009400 con 0x7f9cd406bb10 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cd37fe640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 0x7f9cd40cf3c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009890 tx=0x7f9cc4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cbbfff640 1 -- 172.21.15.172:0/2365374404 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc403e040 con 0x7f9cd406bb10 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cbbfff640 1 -- 172.21.15.172:0/2365374404 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc4039c90 con 0x7f9cd406bb10 2024-03-16T19:29:36.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cbbfff640 1 -- 172.21.15.172:0/2365374404 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4030e10 con 0x7f9cd406bb10 2024-03-16T19:29:36.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd41f1110 con 0x7f9cd406bb10 2024-03-16T19:29:36.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.529+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd41f16d0 con 0x7f9cd406bb10 2024-03-16T19:29:36.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.530+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd40681c0 con 0x7f9cd406bb10 2024-03-16T19:29:36.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.531+0000 7f9cbbfff640 1 -- 172.21.15.172:0/2365374404 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9cc404d3b0 con 0x7f9cd406bb10 2024-03-16T19:29:36.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.531+0000 7f9cbbfff640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9cc006b3b0 0x7f9cc006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.531+0000 7f9cd2ffd640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9cc006b3b0 0x7f9cc006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.531+0000 7f9cbbfff640 1 -- 172.21.15.172:0/2365374404 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cc40ac0e0 con 0x7f9cd406bb10 2024-03-16T19:29:36.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.532+0000 7f9cd2ffd640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9cc006b3b0 0x7f9cc006d870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9cd41512a0 tx=0x7f9cc8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:36.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.535+0000 7f9cbbfff640 1 -- 172.21.15.172:0/2365374404 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cc4076df0 con 0x7f9cd406bb10 2024-03-16T19:29:36.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.743+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cd40ca590 con 0x7f9cd406bb10 2024-03-16T19:29:36.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.744+0000 7f9cbbfff640 1 -- 172.21.15.172:0/2365374404 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6561 (secure 0 0 0) 0x7f9cc407a4b0 con 0x7f9cd406bb10 2024-03-16T19:29:36.725 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:36.725 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1028682606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1028682606},{"type":"v1","addr":"172.21.15.174:6833","nonce":1028682606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6836/902696086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":902696086},{"type":"v1","addr":"172.21.15.174:6836","nonce":902696086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/864851052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":864851052},{"type":"v1","addr":"172.21.15.172:6836","nonce":864851052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/420882750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":420882750},{"type":"v1","addr":"172.21.15.172:6837","nonce":420882750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/2853574742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":2853574742},{"type":"v1","addr":"172.21.15.172:6839","nonce":2853574742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:29:25.674964+0000","modified":"2024-03-16T19:29:31.368977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/2442775894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6835","nonce":2442775894},{"type":"v1","addr":"172.21.15.174:6837","nonce":2442775894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-16T19:29:36.725 INFO:teuthology.orchestra.run.smithi172.stderr:dumped fsmap epoch 8 2024-03-16T19:29:36.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9cc006b3b0 msgr2=0x7f9cc006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9cc006b3b0 0x7f9cc006d870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9cd41512a0 tx=0x7f9cc8009190 comp rx=0 tx=0).stop 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 msgr2=0x7f9cd40cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 0x7f9cd40cf3c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009890 tx=0x7f9cc4004730 comp rx=0 tx=0).stop 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 shutdown_connections 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9cc006b3b0 0x7f9cc006d870 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cd40cc260 0x7f9cd4145dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cd406bef0 0x7f9cd40b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 --2- 172.21.15.172:0/2365374404 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9cd406bb10 0x7f9cd40cf3c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 >> 172.21.15.172:0/2365374404 conn(0x7f9cd4067250 msgr2=0x7f9cd41518d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 shutdown_connections 2024-03-16T19:29:36.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.746+0000 7f9cda055640 1 -- 172.21.15.172:0/2365374404 wait complete. 2024-03-16T19:29:36.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:29:36.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.857+0000 7f37f7439640 1 Processor -- start 2024-03-16T19:29:36.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.857+0000 7f37f7439640 1 -- start start 2024-03-16T19:29:36.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f7439640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f0154cb0 0x7f37f006dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f7439640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37f00c8ba0 0x7f37f00cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f7439640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f00cbba0 con 0x7f37f006fe10 2024-03-16T19:29:36.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f7439640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f00c91f0 con 0x7f37f00c8ba0 2024-03-16T19:29:36.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f7439640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f006c180 con 0x7f37f0154cb0 2024-03-16T19:29:36.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f59d6640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37f00c8ba0 0x7f37f00cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f59d6640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37f00c8ba0 0x7f37f00cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46582/0 (socket says 172.21.15.172:46582) 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f49d4640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f37f006fe10 0x7f37f015de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:53390/0 (socket says 172.21.15.172:53390) 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f51d5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f0154cb0 0x7f37f006dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f59d6640 1 -- 172.21.15.172:0/142688925 learned_addr learned my addr 172.21.15.172:0/142688925 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37e7fff640 1 -- 172.21.15.172:0/142688925 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755243500 0 0) 0x7f37f00cbba0 con 0x7f37f006fe10 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f51d5640 1 -- 172.21.15.172:0/142688925 >> v2:172.21.15.172:3300/0 conn(0x7f37f00c8ba0 msgr2=0x7f37f00cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f51d5640 1 --2- 172.21.15.172:0/142688925 >> v2:172.21.15.172:3300/0 conn(0x7f37f00c8ba0 0x7f37f00cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f51d5640 1 -- 172.21.15.172:0/142688925 >> v1:172.21.15.174:6789/0 conn(0x7f37f006fe10 legacy=0x7f37f015de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f51d5640 1 -- 172.21.15.172:0/142688925 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f006c460 con 0x7f37f0154cb0 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.858+0000 7f37f59d6640 1 --2- 172.21.15.172:0/142688925 >> v2:172.21.15.172:3300/0 conn(0x7f37f00c8ba0 0x7f37f00cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.859+0000 7f37f51d5640 1 --2- 172.21.15.172:0/142688925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f0154cb0 0x7f37f006dc20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37ec0097a0 tx=0x7f37ec02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00cf348a43fc1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.859+0000 7f37e7fff640 1 -- 172.21.15.172:0/142688925 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec02f050 con 0x7f37f0154cb0 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.859+0000 7f37e7fff640 1 -- 172.21.15.172:0/142688925 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec031700 con 0x7f37f0154cb0 2024-03-16T19:29:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.859+0000 7f37e7fff640 1 -- 172.21.15.172:0/142688925 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec031a90 con 0x7f37f0154cb0 2024-03-16T19:29:36.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.860+0000 7f37f7439640 1 -- 172.21.15.172:0/142688925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f0154cb0 msgr2=0x7f37f006dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.860+0000 7f37f7439640 1 --2- 172.21.15.172:0/142688925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f0154cb0 0x7f37f006dc20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37ec0097a0 tx=0x7f37ec02b7a0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.860+0000 7f37f7439640 1 -- 172.21.15.172:0/142688925 shutdown_connections 2024-03-16T19:29:36.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.860+0000 7f37f7439640 1 --2- 172.21.15.172:0/142688925 >> v2:172.21.15.172:3300/0 conn(0x7f37f00c8ba0 0x7f37f00cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.860+0000 7f37f7439640 1 --2- 172.21.15.172:0/142688925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f0154cb0 0x7f37f006dc20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.860+0000 7f37f7439640 1 -- 172.21.15.172:0/142688925 >> 172.21.15.172:0/142688925 conn(0x7f37f0070860 msgr2=0x7f37f0070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.860+0000 7f37f7439640 1 -- 172.21.15.172:0/142688925 shutdown_connections 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.860+0000 7f37f7439640 1 -- 172.21.15.172:0/142688925 wait complete. 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.861+0000 7f37f7439640 1 Processor -- start 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.861+0000 7f37f7439640 1 -- start start 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f7439640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f006fe10 0x7f37f01f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f7439640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f00c8ba0 0x7f37f01f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f7439640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37f0154cb0 0x7f37f00b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f7439640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f00b1770 con 0x7f37f0154cb0 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f7439640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f01f46a0 con 0x7f37f006fe10 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f7439640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f00b3680 con 0x7f37f00c8ba0 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f59d6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37f0154cb0 0x7f37f00b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f59d6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37f0154cb0 0x7f37f00b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46586/0 (socket says 172.21.15.172:46586) 2024-03-16T19:29:36.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f59d6640 1 -- 172.21.15.172:0/843279492 learned_addr learned my addr 172.21.15.172:0/843279492 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f51d5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f006fe10 0x7f37f01f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f49d4640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f00c8ba0 0x7f37f01f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f59d6640 1 -- 172.21.15.172:0/843279492 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f00c8ba0 msgr2=0x7f37f01f0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f59d6640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f00c8ba0 0x7f37f01f0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f59d6640 1 -- 172.21.15.172:0/843279492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f006fe10 msgr2=0x7f37f01f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f59d6640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f006fe10 0x7f37f01f3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f59d6640 1 -- 172.21.15.172:0/843279492 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e8009b30 con 0x7f37f0154cb0 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f49d4640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f00c8ba0 0x7f37f01f0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f51d5640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f006fe10 0x7f37f01f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37f59d6640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37f0154cb0 0x7f37f00b2870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009f10 tx=0x7f37e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.862+0000 7f37e5ffb640 1 -- 172.21.15.172:0/843279492 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e8018070 con 0x7f37f0154cb0 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.863+0000 7f37e5ffb640 1 -- 172.21.15.172:0/843279492 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e800f040 con 0x7f37f0154cb0 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.863+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec009400 con 0x7f37f0154cb0 2024-03-16T19:29:36.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.863+0000 7f37e5ffb640 1 -- 172.21.15.172:0/843279492 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e800ce60 con 0x7f37f0154cb0 2024-03-16T19:29:36.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.863+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f01f40e0 con 0x7f37f0154cb0 2024-03-16T19:29:36.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.864+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f00cb550 con 0x7f37f0154cb0 2024-03-16T19:29:36.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.864+0000 7f37e5ffb640 1 -- 172.21.15.172:0/843279492 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f37e8014480 con 0x7f37f0154cb0 2024-03-16T19:29:36.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.864+0000 7f37e5ffb640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d406b3b0 0x7f37d406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:36.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.864+0000 7f37e5ffb640 1 -- 172.21.15.172:0/843279492 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37e8089e20 con 0x7f37f0154cb0 2024-03-16T19:29:36.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.864+0000 7f37f51d5640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d406b3b0 0x7f37d406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:36.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.865+0000 7f37f51d5640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d406b3b0 0x7f37d406d870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37ec035000 tx=0x7f37ec008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:36.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:36.869+0000 7f37e5ffb640 1 -- 172.21.15.172:0/843279492 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37e8054b40 con 0x7f37f0154cb0 2024-03-16T19:29:37.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.040+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37f006fbf0 con 0x7f37f0154cb0 2024-03-16T19:29:37.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.041+0000 7f37e5ffb640 1 -- 172.21.15.172:0/843279492 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f37e8058200 con 0x7f37f0154cb0 2024-03-16T19:29:37.022 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:37.022 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":21,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:29:27.336190+0000","last_up_change":"2024-03-16T19:29:13.180924+0000","last_in_change":"2024-03-16T19:29:04.178546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6825","nonce":1599824697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6827","nonce":1599824697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6831","nonce":1599824697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":1599824697},{"type":"v1","addr":"172.21.15.172:6829","nonce":1599824697}]},"public_addr":"172.21.15.172:6825/1599824697","cluster_addr":"172.21.15.172:6827/1599824697","heartbeat_back_addr":"172.21.15.172:6831/1599824697","heartbeat_front_addr":"172.21.15.172:6829/1599824697","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:29:37.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d406b3b0 msgr2=0x7f37d406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:37.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d406b3b0 0x7f37d406d870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37ec035000 tx=0x7f37ec008030 comp rx=0 tx=0).stop 2024-03-16T19:29:37.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37f0154cb0 msgr2=0x7f37f00b2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:37.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37f0154cb0 0x7f37f00b2870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009f10 tx=0x7f37e800c490 comp rx=0 tx=0).stop 2024-03-16T19:29:37.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 shutdown_connections 2024-03-16T19:29:37.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37f0154cb0 0x7f37f00b2870 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f37d406b3b0 0x7f37d406d870 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f00c8ba0 0x7f37f01f0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 --2- 172.21.15.172:0/843279492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f006fe10 0x7f37f01f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 >> 172.21.15.172:0/843279492 conn(0x7f37f0070860 msgr2=0x7f37f015d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:37.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 shutdown_connections 2024-03-16T19:29:37.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.043+0000 7f37f7439640 1 -- 172.21.15.172:0/843279492 wait complete. 2024-03-16T19:29:37.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T19:29:37.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.154+0000 7f79a44df640 1 Processor -- start 2024-03-16T19:29:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.154+0000 7f79a44df640 1 -- start start 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.155+0000 7f79a44df640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f799c151b80 0x7f799c158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.155+0000 7f79a44df640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 0x7f799c0c2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.155+0000 7f79a44df640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c06aed0 con 0x7f799c0c2640 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.155+0000 7f79a44df640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799c070d80 con 0x7f799c151b80 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.155+0000 7f79a44df640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799c0c1670 con 0x7f799c0c1880 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.155+0000 7f79a2a7c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f799c0c2640 0x7f799c071ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56320/0 (socket says 172.21.15.172:56320) 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.156+0000 7f79a2a7c640 1 -- 172.21.15.172:0/2696743882 learned_addr learned my addr 172.21.15.172:0/2696743882 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.156+0000 7f79a227b640 1 --2- 172.21.15.172:0/2696743882 >> v2:172.21.15.174:3300/0 conn(0x7f799c151b80 0x7f799c158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.156+0000 7f79a1a7a640 1 --2- 172.21.15.172:0/2696743882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 0x7f799c0c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.156+0000 7f79a1279640 1 -- 172.21.15.172:0/2696743882 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206391932 0 0) 0x7f799c06aed0 con 0x7f799c0c2640 2024-03-16T19:29:37.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.156+0000 7f79a227b640 1 -- 172.21.15.172:0/2696743882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 msgr2=0x7f799c0c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:37.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.156+0000 7f79a227b640 1 --2- 172.21.15.172:0/2696743882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 0x7f799c0c2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.156+0000 7f79a227b640 1 -- 172.21.15.172:0/2696743882 >> v1:172.21.15.172:6789/0 conn(0x7f799c0c2640 legacy=0x7f799c071ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:37.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.156+0000 7f79a227b640 1 -- 172.21.15.172:0/2696743882 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c0c10f0 con 0x7f799c151b80 2024-03-16T19:29:37.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.156+0000 7f79a1a7a640 1 --2- 172.21.15.172:0/2696743882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 0x7f799c0c2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:37.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a227b640 1 --2- 172.21.15.172:0/2696743882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f799c151b80 0x7f799c158230 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f79980097a0 tx=0x7f799802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8df3c7a4c7d47cc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:37.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a1279640 1 -- 172.21.15.172:0/2696743882 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799802f050 con 0x7f799c151b80 2024-03-16T19:29:37.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a1279640 1 -- 172.21.15.172:0/2696743882 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7998031700 con 0x7f799c151b80 2024-03-16T19:29:37.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a1279640 1 -- 172.21.15.172:0/2696743882 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7998031a90 con 0x7f799c151b80 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a44df640 1 -- 172.21.15.172:0/2696743882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f799c151b80 msgr2=0x7f799c158230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a44df640 1 --2- 172.21.15.172:0/2696743882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f799c151b80 0x7f799c158230 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f79980097a0 tx=0x7f799802b7a0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a44df640 1 -- 172.21.15.172:0/2696743882 shutdown_connections 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a44df640 1 --2- 172.21.15.172:0/2696743882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 0x7f799c0c2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a44df640 1 --2- 172.21.15.172:0/2696743882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f799c151b80 0x7f799c158230 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a44df640 1 -- 172.21.15.172:0/2696743882 >> 172.21.15.172:0/2696743882 conn(0x7f799c0c8b30 msgr2=0x7f799c0c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a44df640 1 -- 172.21.15.172:0/2696743882 shutdown_connections 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.157+0000 7f79a44df640 1 -- 172.21.15.172:0/2696743882 wait complete. 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.158+0000 7f79a44df640 1 Processor -- start 2024-03-16T19:29:37.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.158+0000 7f79a44df640 1 -- start start 2024-03-16T19:29:37.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a44df640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 0x7f799c0c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:37.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a44df640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f799c0c2640 0x7f799c0c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:37.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a44df640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f799c151b80 0x7f799c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:37.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a44df640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c0c66e0 con 0x7f799c0c2640 2024-03-16T19:29:37.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a44df640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c1f0bc0 con 0x7f799c151b80 2024-03-16T19:29:37.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a44df640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799c0c51c0 con 0x7f799c0c1880 2024-03-16T19:29:37.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a1a7a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f799c0c2640 0x7f799c0c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:37.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a1a7a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f799c0c2640 0x7f799c0c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46594/0 (socket says 172.21.15.172:46594) 2024-03-16T19:29:37.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a1a7a640 1 -- 172.21.15.172:0/2211479312 learned_addr learned my addr 172.21.15.172:0/2211479312 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a2a7c640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f799c151b80 0x7f799c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.159+0000 7f79a227b640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 0x7f799c0c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f79a1a7a640 1 -- 172.21.15.172:0/2211479312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 msgr2=0x7f799c0c61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f79a1a7a640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 0x7f799c0c61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f79a1a7a640 1 -- 172.21.15.172:0/2211479312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f799c151b80 msgr2=0x7f799c0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f79a1a7a640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f799c151b80 0x7f799c0cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f79a1a7a640 1 -- 172.21.15.172:0/2211479312 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c009400 con 0x7f799c0c2640 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f79a1a7a640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f799c0c2640 0x7f799c0c4c80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f798c00e9c0 tx=0x7f798c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f7992ffd640 1 -- 172.21.15.172:0/2211479312 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c00ca60 con 0x7f799c0c2640 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f7992ffd640 1 -- 172.21.15.172:0/2211479312 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c00cbc0 con 0x7f799c0c2640 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7998009400 con 0x7f799c0c2640 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f7992ffd640 1 -- 172.21.15.172:0/2211479312 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c013470 con 0x7f799c0c2640 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.160+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c1f0570 con 0x7f799c0c2640 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.161+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799c0681c0 con 0x7f799c0c2640 2024-03-16T19:29:37.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.161+0000 7f7992ffd640 1 -- 172.21.15.172:0/2211479312 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f798c0135d0 con 0x7f799c0c2640 2024-03-16T19:29:37.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.162+0000 7f7992ffd640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f798006b3b0 0x7f798006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:37.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.162+0000 7f7992ffd640 1 -- 172.21.15.172:0/2211479312 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f798c089de0 con 0x7f799c0c2640 2024-03-16T19:29:37.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.162+0000 7f79a227b640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f798006b3b0 0x7f798006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:37.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.162+0000 7f79a227b640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f798006b3b0 0x7f798006d870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7998035000 tx=0x7f7998008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:37.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.166+0000 7f7992ffd640 1 -- 172.21.15.172:0/2211479312 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f798c054af0 con 0x7f799c0c2640 2024-03-16T19:29:37.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.372+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f799c0c98f0 con 0x7f799c0c2640 2024-03-16T19:29:37.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.373+0000 7f7992ffd640 1 -- 172.21.15.172:0/2211479312 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6561 (secure 0 0 0) 0x7f798c0581b0 con 0x7f799c0c2640 2024-03-16T19:29:37.354 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:37.354 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1028682606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1028682606},{"type":"v1","addr":"172.21.15.174:6833","nonce":1028682606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6836/902696086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":902696086},{"type":"v1","addr":"172.21.15.174:6836","nonce":902696086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6836/864851052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":864851052},{"type":"v1","addr":"172.21.15.172:6836","nonce":864851052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/420882750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6835","nonce":420882750},{"type":"v1","addr":"172.21.15.172:6837","nonce":420882750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4278,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/2853574742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":2853574742},{"type":"v1","addr":"172.21.15.172:6839","nonce":2853574742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T19:29:25.674964+0000","modified":"2024-03-16T19:29:31.368977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/2442775894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6835","nonce":2442775894},{"type":"v1","addr":"172.21.15.174:6837","nonce":2442775894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-16T19:29:37.354 INFO:teuthology.orchestra.run.smithi172.stderr:dumped fsmap epoch 8 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.374+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f798006b3b0 msgr2=0x7f798006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.374+0000 7f79a44df640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f798006b3b0 0x7f798006d870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7998035000 tx=0x7f7998008030 comp rx=0 tx=0).stop 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.374+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f799c0c2640 msgr2=0x7f799c0c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.374+0000 7f79a44df640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f799c0c2640 0x7f799c0c4c80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f798c00e9c0 tx=0x7f798c00ee80 comp rx=0 tx=0).stop 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.375+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 shutdown_connections 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.375+0000 7f79a44df640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f799c151b80 0x7f799c0cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.375+0000 7f79a44df640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f799c0c2640 0x7f799c0c4c80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.375+0000 7f79a44df640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f798006b3b0 0x7f798006d870 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.375+0000 7f79a44df640 1 --2- 172.21.15.172:0/2211479312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f799c0c1880 0x7f799c0c61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.375+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 >> 172.21.15.172:0/2211479312 conn(0x7f799c0c8b30 msgr2=0x7f799c150df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:37.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.375+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 shutdown_connections 2024-03-16T19:29:37.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:37.375+0000 7f79a44df640 1 -- 172.21.15.172:0/2211479312 wait complete. 2024-03-16T19:29:37.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T19:29:25.674964+0000', 'modified': '2024-03-16T19:29:31.368977+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4297}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4297': {'gid': 4297, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.174:6837/2442775894', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6835', 'nonce': 2442775894}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 2442775894}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4297, 'qdb_cluster': [4297]} 2024-03-16T19:29:37.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-16T19:29:37.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-16T19:29:38.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.101+0000 7f11b2017640 1 Processor -- start 2024-03-16T19:29:38.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.102+0000 7f11b2017640 1 -- start start 2024-03-16T19:29:38.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.102+0000 7f11b2017640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f11ac15c340 0x7f11ac0c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.102+0000 7f11b2017640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ac152e90 con 0x7f11ac152ab0 2024-03-16T19:29:38.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.102+0000 7f11b2017640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ac0c30e0 con 0x7f11ac0c9af0 2024-03-16T19:29:38.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.102+0000 7f11b2017640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ac0c85e0 con 0x7f11ac15c340 2024-03-16T19:29:38.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.102+0000 7f11a2ffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f11ac15c340 0x7f11ac0c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.102+0000 7f11ab7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f11ac152ab0 0x7f11ac15bc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56324/0 (socket says 172.21.15.172:56324) 2024-03-16T19:29:38.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.102+0000 7f11ab7fe640 1 -- 172.21.15.172:0/1754554300 learned_addr learned my addr 172.21.15.172:0/1754554300 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:38.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.103+0000 7f11a2ffd640 1 -- 172.21.15.172:0/1754554300 >> v1:172.21.15.174:6789/0 conn(0x7f11ac0c9af0 legacy=0x7f11ac06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.103+0000 7f11a2ffd640 1 -- 172.21.15.172:0/1754554300 >> v1:172.21.15.172:6789/0 conn(0x7f11ac152ab0 legacy=0x7f11ac15bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.103+0000 7f11a2ffd640 1 -- 172.21.15.172:0/1754554300 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac0c33c0 con 0x7f11ac15c340 2024-03-16T19:29:38.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.103+0000 7f11aaffd640 1 -- 172.21.15.172:0/1754554300 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170802938 0 0) 0x7f119c001110 con 0x7f11ac152ab0 2024-03-16T19:29:38.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.103+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/1754554300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac15c340 0x7f11ac0c8ac0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f11940097a0 tx=0x7f119402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b9b4a08f1f1f7d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11aaffd640 1 -- 172.21.15.172:0/1754554300 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194030610 con 0x7f11ac15c340 2024-03-16T19:29:38.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11aaffd640 1 -- 172.21.15.172:0/1754554300 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1194030770 con 0x7f11ac15c340 2024-03-16T19:29:38.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11aaffd640 1 -- 172.21.15.172:0/1754554300 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194030b00 con 0x7f11ac15c340 2024-03-16T19:29:38.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11b2017640 1 -- 172.21.15.172:0/1754554300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac15c340 msgr2=0x7f11ac0c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11b2017640 1 --2- 172.21.15.172:0/1754554300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac15c340 0x7f11ac0c8ac0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f11940097a0 tx=0x7f119402b960 comp rx=0 tx=0).stop 2024-03-16T19:29:38.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11b2017640 1 -- 172.21.15.172:0/1754554300 shutdown_connections 2024-03-16T19:29:38.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11b2017640 1 --2- 172.21.15.172:0/1754554300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac15c340 0x7f11ac0c8ac0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11b2017640 1 -- 172.21.15.172:0/1754554300 >> 172.21.15.172:0/1754554300 conn(0x7f11ac070810 msgr2=0x7f11ac070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:38.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11b2017640 1 -- 172.21.15.172:0/1754554300 shutdown_connections 2024-03-16T19:29:38.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.104+0000 7f11b2017640 1 -- 172.21.15.172:0/1754554300 wait complete. 2024-03-16T19:29:38.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.105+0000 7f11b2017640 1 Processor -- start 2024-03-16T19:29:38.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11b2017640 1 -- start start 2024-03-16T19:29:38.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11b2017640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11ac0c9af0 0x7f11ac153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11b2017640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac152ab0 0x7f11ac0ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11b2017640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ac159150 0x7f11ac15abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11b2017640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac153c60 con 0x7f11ac152ab0 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11b2017640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac1e2ad0 con 0x7f11ac0c9af0 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11b2017640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac1e1ec0 con 0x7f11ac159150 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11a2ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac152ab0 0x7f11ac0ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11a2ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac152ab0 0x7f11ac0ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46622/0 (socket says 172.21.15.172:46622) 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11a2ffd640 1 -- 172.21.15.172:0/3396688795 learned_addr learned my addr 172.21.15.172:0/3396688795 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11abfff640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ac159150 0x7f11ac15abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.106+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11ac0c9af0 0x7f11ac153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11a2ffd640 1 -- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ac159150 msgr2=0x7f11ac15abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ac159150 0x7f11ac15abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11a2ffd640 1 -- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11ac0c9af0 msgr2=0x7f11ac153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11ac0c9af0 0x7f11ac153720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11a2ffd640 1 -- 172.21.15.172:0/3396688795 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1194009400 con 0x7f11ac152ab0 2024-03-16T19:29:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11abfff640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ac159150 0x7f11ac15abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11ac0c9af0 0x7f11ac153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac152ab0 0x7f11ac0ca690 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f11940098c0 tx=0x7f11940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11a8ff9640 1 -- 172.21.15.172:0/3396688795 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194039c10 con 0x7f11ac152ab0 2024-03-16T19:29:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11a8ff9640 1 -- 172.21.15.172:0/3396688795 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119403f040 con 0x7f11ac152ab0 2024-03-16T19:29:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11a8ff9640 1 -- 172.21.15.172:0/3396688795 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11940433f0 con 0x7f11ac152ab0 2024-03-16T19:29:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11b2017640 1 -- 172.21.15.172:0/3396688795 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac1e21a0 con 0x7f11ac152ab0 2024-03-16T19:29:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.107+0000 7f11b2017640 1 -- 172.21.15.172:0/3396688795 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac1e2760 con 0x7f11ac152ab0 2024-03-16T19:29:38.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.108+0000 7f11a8ff9640 1 -- 172.21.15.172:0/3396688795 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1194043550 con 0x7f11ac152ab0 2024-03-16T19:29:38.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.109+0000 7f11a8ff9640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f118c06b3b0 0x7f118c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.109+0000 7f11a8ff9640 1 -- 172.21.15.172:0/3396688795 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11940ac5e0 con 0x7f11ac152ab0 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.109+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f118c06b3b0 0x7f118c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.109+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f118c06b3b0 0x7f118c06d870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f119c007900 tx=0x7f119c007870 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.110+0000 7f11b4bf0740 1 Processor -- start 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.110+0000 7f11b4bf0740 1 -- start start 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.110+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd44e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 0x55d99bd45350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd43490 0x55d99bd84290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99bcffcd0 con 0x55d99bd43490 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99bcb2f20 con 0x55d99bd8e8b0 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d99bcb0450 con 0x55d99bd44a30 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11ab7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd44e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11abfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd43490 0x55d99bd84290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11ab7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd44e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:60318/0 (socket says 172.21.15.172:60318) 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11ab7fe640 1 -- 172.21.15.172:0/1200795832 learned_addr learned my addr 172.21.15.172:0/1200795832 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/1200795832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 0x55d99bd45350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a2ffd640 1 -- 172.21.15.172:0/1200795832 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 msgr2=0x55d99bd44e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/1200795832 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd44e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a2ffd640 1 -- 172.21.15.172:0/1200795832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd43490 msgr2=0x55d99bd84290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/1200795832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd43490 0x55d99bd84290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a2ffd640 1 -- 172.21.15.172:0/1200795832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d99bc60f20 con 0x55d99bd8e8b0 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/1200795832 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd44e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/1200795832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 0x55d99bd45350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f11940aab60 tx=0x7f1194009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de5c018c5b66ccb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a17fa640 1 -- 172.21.15.172:0/1200795832 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11940792c0 con 0x55d99bd8e8b0 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a17fa640 1 -- 172.21.15.172:0/1200795832 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1194077630 con 0x55d99bd8e8b0 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.111+0000 7f11a17fa640 1 -- 172.21.15.172:0/1200795832 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119407a650 con 0x55d99bd8e8b0 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.112+0000 7f11b4bf0740 1 -- 172.21.15.172:0/1200795832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 msgr2=0x55d99bd45350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.112+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/1200795832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 0x55d99bd45350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f11940aab60 tx=0x7f1194009190 comp rx=0 tx=0).stop 2024-03-16T19:29:38.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.112+0000 7f11b4bf0740 1 -- 172.21.15.172:0/1200795832 shutdown_connections 2024-03-16T19:29:38.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.112+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/1200795832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd43490 0x55d99bd84290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.112+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/1200795832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 0x55d99bd45350 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.112+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/1200795832 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd44e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.112+0000 7f11b4bf0740 1 -- 172.21.15.172:0/1200795832 >> 172.21.15.172:0/1200795832 conn(0x55d99bd411a0 msgr2=0x55d99bd43ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:38.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.112+0000 7f11b4bf0740 1 -- 172.21.15.172:0/1200795832 shutdown_connections 2024-03-16T19:29:38.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.112+0000 7f11b4bf0740 1 -- 172.21.15.172:0/1200795832 wait complete. 2024-03-16T19:29:38.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11b4bf0740 1 -- start start 2024-03-16T19:29:38.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11b4bf0740 1 Processor -- start 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd44a30 0x55d99bd86aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd8e8b0 0x55d99bd69220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 0x55d99bd88520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99bcabc30 con 0x55d99bd44a30 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99bc50d30 con 0x55d99bd681b0 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d99b97f310 con 0x55d99bd8e8b0 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11b4bf0740 10 client.0 _ll_register_callbacks cb 0x55d99bc58d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.113+0000 7f11ab7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd44a30 0x55d99bd86aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11ab7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd44a30 0x55d99bd86aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46636/0 (socket says 172.21.15.172:46636) 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11ab7fe640 1 -- 172.21.15.172:0/4284618700 learned_addr learned my addr 172.21.15.172:0/4284618700 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:38.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd8e8b0 0x55d99bd69220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11abfff640 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 0x55d99bd88520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11ab7fe640 1 -- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd8e8b0 msgr2=0x55d99bd69220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd8e8b0 0x55d99bd69220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11ab7fe640 1 -- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 msgr2=0x55d99bd88520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 0x55d99bd88520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11ab7fe640 1 -- 172.21.15.172:0/4284618700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac1e2760 con 0x55d99bd44a30 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd44a30 0x55d99bd86aa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f119c013ad0 tx=0x7f119c007b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97951fa3a04e36ca server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f118a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f118a7fc640 1 -- 172.21.15.172:0/4284618700 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c012ae0 con 0x55d99bd44a30 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f118a7fc640 1 -- 172.21.15.172:0/4284618700 <== mon.0 v2:172.21.15.172:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f119c01b070 con 0x55d99bd44a30 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11b4bf0740 1 -- 172.21.15.172:0/4284618700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d99b9ab6c0 con 0x55d99bd44a30 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f118a7fc640 1 -- 172.21.15.172:0/4284618700 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f119c012c40 con 0x55d99bd44a30 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11b4bf0740 10 client.4632 fetch_fsmap learned FSMap version 8 2024-03-16T19:29:38.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f11b4bf0740 1 -- 172.21.15.172:0/4284618700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55d99bcaa2b0 con 0x55d99bd44a30 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.114+0000 7f118a7fc640 1 -- 172.21.15.172:0/4284618700 <== mon.0 v2:172.21.15.172:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d99b9ab6c0 con 0x55d99bd44a30 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f118a7fc640 1 -- 172.21.15.172:0/4284618700 <== mon.0 v2:172.21.15.172:3300/0 5 ==== fsmap(e 8) v1 ==== 4464+0+0 (secure 0 0 0) 0x7f119c012070 con 0x55d99bd44a30 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f11b4bf0740 10 client.4632 fetch_fsmap finished waiting for FSMap version 8 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f11b4bf0740 10 client.4632 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f11b4bf0740 20 client.4632 populate_metadata read hostname 'smithi172' 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd63a10 0x55d99bda3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f11b4bf0740 4 client.4632 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f11b4bf0740 1 -- 172.21.15.172:0/4284618700 --> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d99b9fc270 con 0x55d99bd63a10 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd63a10 0x55d99bda3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f118a7fc640 10 client.4632 ms_handle_connect on v2:172.21.15.174:6835/2442775894 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.115+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd63a10 0x55d99bda3f90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75502960ee5a599e server_cookie=f9898a55838f8f50 in_seq=0 out_seq=0 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f118a7fc640 1 -- 172.21.15.172:0/4284618700 <== mds.0 v2:172.21.15.174:6835/2442775894 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d99b9fc270 con 0x55d99bd63a10 2024-03-16T19:29:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f118a7fc640 10 client.4632 handle_command_reply: tid=0 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 client.4632 shutdown 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 -- 172.21.15.172:0/4284618700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd44a30 msgr2=0x55d99bd86aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd44a30 0x55d99bd86aa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f119c013ad0 tx=0x7f119c007b90 comp rx=0 tx=0).stop 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 -- 172.21.15.172:0/4284618700 shutdown_connections 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd63a10 0x55d99bda3f90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 0x55d99bd88520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd8e8b0 0x55d99bd69220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f118a7fc640 0 client.4632 ms_handle_reset on v2:172.21.15.174:6835/2442775894 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/4284618700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd44a30 0x55d99bd86aa0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 -- 172.21.15.172:0/4284618700 >> 172.21.15.172:0/4284618700 conn(0x55d99bd43490 msgr2=0x55d99bd3df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 -- 172.21.15.172:0/4284618700 shutdown_connections 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 1 -- 172.21.15.172:0/4284618700 wait complete. 2024-03-16T19:29:38.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.116+0000 7f11b4bf0740 20 client.4632 trim_cache size 0 max 16384 2024-03-16T19:29:38.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.133+0000 7f11b4bf0740 1 Processor -- start 2024-03-16T19:29:38.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.133+0000 7f11b4bf0740 1 -- start start 2024-03-16T19:29:38.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd46110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 0x55d99bd46650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd8e8b0 0x55d99bd8d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99bc31490 con 0x55d99bd8e8b0 2024-03-16T19:29:38.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99bd7f830 con 0x55d99bd681b0 2024-03-16T19:29:38.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d99bd41860 con 0x55d99bd44a30 2024-03-16T19:29:38.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11abfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd8e8b0 0x55d99bd8d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11abfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd8e8b0 0x55d99bd8d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46644/0 (socket says 172.21.15.172:46644) 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11abfff640 1 -- 172.21.15.172:0/2162789114 learned_addr learned my addr 172.21.15.172:0/2162789114 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 0x55d99bd46650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd46110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11abfff640 1 -- 172.21.15.172:0/2162789114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 msgr2=0x55d99bd46110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11abfff640 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd46110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11abfff640 1 -- 172.21.15.172:0/2162789114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 msgr2=0x55d99bd46650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11abfff640 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 0x55d99bd46650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11abfff640 1 -- 172.21.15.172:0/2162789114 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d99bc46a20 con 0x55d99bd8e8b0 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 0x55d99bd46650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11abfff640 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd8e8b0 0x55d99bd8d820 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f11a40021b0 tx=0x7f11a40015d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14dc29b430fbc2ba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11a0ff9640 1 -- 172.21.15.172:0/2162789114 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4003b80 con 0x55d99bd8e8b0 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11a0ff9640 1 -- 172.21.15.172:0/2162789114 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a4003ce0 con 0x55d99bd8e8b0 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.134+0000 7f11a0ff9640 1 -- 172.21.15.172:0/2162789114 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a40052d0 con 0x55d99bd8e8b0 2024-03-16T19:29:38.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 -- 172.21.15.172:0/2162789114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd8e8b0 msgr2=0x55d99bd8d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd8e8b0 0x55d99bd8d820 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f11a40021b0 tx=0x7f11a40015d0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 -- 172.21.15.172:0/2162789114 shutdown_connections 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd8e8b0 0x55d99bd8d820 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd681b0 0x55d99bd46650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/2162789114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd46110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 -- 172.21.15.172:0/2162789114 >> 172.21.15.172:0/2162789114 conn(0x55d99bd43490 msgr2=0x55d99bd88500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 -- 172.21.15.172:0/2162789114 shutdown_connections 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 -- 172.21.15.172:0/2162789114 wait complete. 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.135+0000 7f11b4bf0740 1 -- start start 2024-03-16T19:29:38.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11b4bf0740 1 Processor -- start 2024-03-16T19:29:38.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd3fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd681b0 0x55d99bd5cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11b4bf0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 0x55d99bdf27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99bd5d6b0 con 0x55d99bd681b0 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99bd400f0 con 0x55d99bd8e8b0 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11b4bf0740 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d99bd85af0 con 0x55d99bd44a30 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11b4bf0740 10 client.0 _ll_register_callbacks cb 0x7f119c004db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11a2ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd681b0 0x55d99bd5cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11a2ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd681b0 0x55d99bd5cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46650/0 (socket says 172.21.15.172:46650) 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11a2ffd640 1 -- 172.21.15.172:0/3748230717 learned_addr learned my addr 172.21.15.172:0/3748230717 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd3fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.136+0000 7f11abfff640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 0x55d99bdf27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11a2ffd640 1 -- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 msgr2=0x55d99bd3fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd3fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11a2ffd640 1 -- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 msgr2=0x55d99bdf27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:38.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 0x55d99bdf27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11a2ffd640 1 -- 172.21.15.172:0/3748230717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d99bc60f20 con 0x55d99bd681b0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11a2ffd640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd681b0 0x55d99bd5cac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f119402bf70 tx=0x7f119403b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e860d0e7b6e699ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f118a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f118a7fc640 1 -- 172.21.15.172:0/3748230717 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194077260 con 0x55d99bd681b0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f118a7fc640 1 -- 172.21.15.172:0/3748230717 <== mon.0 v2:172.21.15.172:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1194098030 con 0x55d99bd681b0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11b4bf0740 1 -- 172.21.15.172:0/3748230717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1194077630 con 0x55d99bd681b0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f118a7fc640 1 -- 172.21.15.172:0/3748230717 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11940a41f0 con 0x55d99bd681b0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11b4bf0740 10 client.4638 fetch_fsmap learned FSMap version 8 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11b4bf0740 1 -- 172.21.15.172:0/3748230717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x7f11ac0c30e0 con 0x55d99bd681b0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f118a7fc640 1 -- 172.21.15.172:0/3748230717 <== mon.0 v2:172.21.15.172:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1194077630 con 0x55d99bd681b0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f118a7fc640 1 -- 172.21.15.172:0/3748230717 <== mon.0 v2:172.21.15.172:3300/0 5 ==== fsmap(e 8) v1 ==== 4464+0+0 (secure 0 0 0) 0x7f1194042070 con 0x55d99bd681b0 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11b4bf0740 10 client.4638 fetch_fsmap finished waiting for FSMap version 8 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11b4bf0740 10 client.4638 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-16T19:29:38.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11b4bf0740 20 client.4638 populate_metadata read hostname 'smithi172' 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 0x55d99bd5fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11b4bf0740 4 client.4638 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11b4bf0740 1 -- 172.21.15.172:0/3748230717 --> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d99b9e8610 con 0x55d99bd5f850 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.137+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 0x55d99bd5fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f118a7fc640 10 client.4638 ms_handle_connect on v2:172.21.15.174:6835/2442775894 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 0x55d99bd5fc30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4d1cb979fd626f5 server_cookie=3225a682c317e19b in_seq=0 out_seq=0 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11ab7fe640 1 -- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 msgr2=0x55d99bd5fc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11ab7fe640 1 -- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 msgr2=0x55d99bd5fc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 0x55d99bd5fc30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 0x55d99bd5fc30 unknown :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 0x55d99bd5fc30 unknown :-1 s=BANNER_CONNECTING pgs=13 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f118a7fc640 10 client.4638 ms_handle_connect on v2:172.21.15.174:6835/2442775894 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11ab7fe640 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 0x55d99bd5fc30 crc :-1 s=READY pgs=13 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4d1cb979fd626f5 server_cookie=3225a682c317e19b in_seq=0 out_seq=1 2024-03-16T19:29:38.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f118a7fc640 1 -- 172.21.15.172:0/3748230717 <== mds.0 v2:172.21.15.174:6835/2442775894 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55d99b9e8610 con 0x55d99bd5f850 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f118a7fc640 10 client.4638 handle_command_reply: tid=0 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11b4bf0740 1 client.4638 shutdown 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11b4bf0740 1 -- 172.21.15.172:0/3748230717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd681b0 msgr2=0x55d99bd5cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd681b0 0x55d99bd5cac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f119402bf70 tx=0x7f119403b070 comp rx=0 tx=0).stop 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11b4bf0740 1 -- 172.21.15.172:0/3748230717 shutdown_connections 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.138+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:6835/2442775894,v1:172.21.15.174:6837/2442775894] conn(0x55d99bd5f850 0x55d99bd5fc30 crc :-1 s=READY pgs=13 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99bd8e8b0 0x55d99bdf27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d99bd681b0 0x55d99bd5cac0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f118a7fc640 0 client.4638 ms_handle_reset on v2:172.21.15.174:6835/2442775894 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b4bf0740 1 --2- 172.21.15.172:0/3748230717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55d99bd44a30 0x55d99bd3fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b4bf0740 1 -- 172.21.15.172:0/3748230717 >> 172.21.15.172:0/3748230717 conn(0x55d99bd43490 msgr2=0x55d99bd3df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b4bf0740 1 -- 172.21.15.172:0/3748230717 shutdown_connections 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b4bf0740 1 -- 172.21.15.172:0/3748230717 wait complete. 2024-03-16T19:29:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b4bf0740 20 client.4638 trim_cache size 0 max 16384 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_fsid": "6c00e600-a879-4eb6-acee-0fdf0f76922a", 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "whoami": 0, 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "id": 4297, 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "want_state": "up:active", 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "state": "up:active", 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "fs_name": "cephfs", 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "rank_uptime": 11.785967736, 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "mdsmap_epoch": 8, 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_epoch": 21, 2024-03-16T19:29:38.127 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_epoch_barrier": 21, 2024-03-16T19:29:38.128 INFO:teuthology.orchestra.run.smithi172.stdout: "uptime": 18.411080956999999 2024-03-16T19:29:38.128 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:38.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 -- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f118c06b3b0 msgr2=0x7f118c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f118c06b3b0 0x7f118c06d870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f119c007900 tx=0x7f119c007870 comp rx=0 tx=0).stop 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 -- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac152ab0 msgr2=0x7f11ac0ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac152ab0 0x7f11ac0ca690 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f11940098c0 tx=0x7f11940046c0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 -- 172.21.15.172:0/3396688795 shutdown_connections 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f118c06b3b0 0x7f118c06d870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ac159150 0x7f11ac15abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11ac152ab0 0x7f11ac0ca690 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 --2- 172.21.15.172:0/3396688795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11ac0c9af0 0x7f11ac153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 -- 172.21.15.172:0/3396688795 >> 172.21.15.172:0/3396688795 conn(0x7f11ac070810 msgr2=0x7f11ac0cb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:38.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 -- 172.21.15.172:0/3396688795 shutdown_connections 2024-03-16T19:29:38.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.139+0000 7f11b2017640 1 -- 172.21.15.172:0/3396688795 wait complete. 2024-03-16T19:29:38.131 INFO:teuthology.run_tasks:Running task kclient... 2024-03-16T19:29:38.152 INFO:tasks.kclient:Mounting kernel clients... 2024-03-16T19:29:38.152 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-16T19:29:38.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:38.152 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list 2024-03-16T19:29:38.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:38.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link delete ceph-brx 2024-03-16T19:29:38.237 INFO:teuthology.orchestra.run.smithi172.stderr:Cannot find device "ceph-brx" 2024-03-16T19:29:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T19:29:38.238 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-16T19:29:38.238 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T19:29:38.238 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T19:29:38.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-16T19:29:38.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.399+0000 7f55579ac640 1 Processor -- start 2024-03-16T19:29:38.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.399+0000 7f55579ac640 1 -- start start 2024-03-16T19:29:38.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f55579ac640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5550152b20 0x7f555006cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f55579ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 0x7f555006b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f55579ac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555015cd20 con 0x7f5550071be0 2024-03-16T19:29:38.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f55579ac640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555006c060 con 0x7f5550152b20 2024-03-16T19:29:38.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f55579ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5550152f00 con 0x7f555006aed0 2024-03-16T19:29:38.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f5555748640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5550152b20 0x7f555006cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f5555748640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5550152b20 0x7f555006cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46654/0 (socket says 172.21.15.172:46654) 2024-03-16T19:29:38.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f5555748640 1 -- 172.21.15.172:0/3909822146 learned_addr learned my addr 172.21.15.172:0/3909822146 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:38.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f5554f47640 1 --2- 172.21.15.172:0/3909822146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 0x7f555006b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f5555748640 1 -- 172.21.15.172:0/3909822146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 msgr2=0x7f555006b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:38.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f5555748640 1 --2- 172.21.15.172:0/3909822146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 0x7f555006b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f5555748640 1 -- 172.21.15.172:0/3909822146 >> v1:172.21.15.174:6789/0 conn(0x7f5550071be0 legacy=0x7f5550158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.400+0000 7f5555748640 1 -- 172.21.15.172:0/3909822146 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555015be70 con 0x7f5550152b20 2024-03-16T19:29:38.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.401+0000 7f5555748640 1 --2- 172.21.15.172:0/3909822146 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550152b20 0x7f555006cc00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f554c009e50 tx=0x7f554c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbec0a4e1baec77 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.401+0000 7f5547fff640 1 -- 172.21.15.172:0/3909822146 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c02cb00 con 0x7f5550152b20 2024-03-16T19:29:38.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.401+0000 7f5547fff640 1 -- 172.21.15.172:0/3909822146 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c02cc60 con 0x7f5550152b20 2024-03-16T19:29:38.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.401+0000 7f5547fff640 1 -- 172.21.15.172:0/3909822146 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c0343f0 con 0x7f5550152b20 2024-03-16T19:29:38.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.402+0000 7f55579ac640 1 -- 172.21.15.172:0/3909822146 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550152b20 msgr2=0x7f555006cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.402+0000 7f55579ac640 1 --2- 172.21.15.172:0/3909822146 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550152b20 0x7f555006cc00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f554c009e50 tx=0x7f554c02bf60 comp rx=0 tx=0).stop 2024-03-16T19:29:38.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.402+0000 7f55579ac640 1 -- 172.21.15.172:0/3909822146 shutdown_connections 2024-03-16T19:29:38.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.402+0000 7f55579ac640 1 --2- 172.21.15.172:0/3909822146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 0x7f555006b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.402+0000 7f55579ac640 1 --2- 172.21.15.172:0/3909822146 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550152b20 0x7f555006cc00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.402+0000 7f55579ac640 1 -- 172.21.15.172:0/3909822146 >> 172.21.15.172:0/3909822146 conn(0x7f555006e8b0 msgr2=0x7f555006ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:38.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.402+0000 7f55579ac640 1 -- 172.21.15.172:0/3909822146 shutdown_connections 2024-03-16T19:29:38.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.402+0000 7f55579ac640 1 -- 172.21.15.172:0/3909822146 wait complete. 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.403+0000 7f55579ac640 1 Processor -- start 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.403+0000 7f55579ac640 1 -- start start 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f55579ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 0x7f55501f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f55579ac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550071be0 0x7f555015b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f55579ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5550152b20 0x7f555014d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f55579ac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55501f2d10 con 0x7f5550071be0 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f55579ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5550156900 con 0x7f5550152b20 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f55579ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55501eed00 con 0x7f555006aed0 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5554f47640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550071be0 0x7f555015b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5554f47640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550071be0 0x7f555015b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46660/0 (socket says 172.21.15.172:46660) 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5554f47640 1 -- 172.21.15.172:0/2762399221 learned_addr learned my addr 172.21.15.172:0/2762399221 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5555f49640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5550152b20 0x7f555014d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5555748640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 0x7f55501f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5554f47640 1 -- 172.21.15.172:0/2762399221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 msgr2=0x7f55501f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5554f47640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 0x7f55501f27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5554f47640 1 -- 172.21.15.172:0/2762399221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5550152b20 msgr2=0x7f555014d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5554f47640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5550152b20 0x7f555014d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5554f47640 1 -- 172.21.15.172:0/2762399221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c009b30 con 0x7f5550071be0 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5554f47640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550071be0 0x7f555015b410 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f554000e9e0 tx=0x7f554000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5545ffb640 1 -- 172.21.15.172:0/2762399221 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540009670 con 0x7f5550071be0 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5545ffb640 1 -- 172.21.15.172:0/2762399221 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554000c9d0 con 0x7f5550071be0 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f5545ffb640 1 -- 172.21.15.172:0/2762399221 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554000cd80 con 0x7f5550071be0 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.404+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55501eefe0 con 0x7f5550071be0 2024-03-16T19:29:38.387 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.405+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55501ef5a0 con 0x7f5550071be0 2024-03-16T19:29:38.388 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.406+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55500688d0 con 0x7f5550071be0 2024-03-16T19:29:38.388 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.406+0000 7f5545ffb640 1 -- 172.21.15.172:0/2762399221 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5540013470 con 0x7f5550071be0 2024-03-16T19:29:38.388 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.406+0000 7f5545ffb640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f553406b3b0 0x7f553406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:38.388 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.406+0000 7f5545ffb640 1 -- 172.21.15.172:0/2762399221 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5540089d90 con 0x7f5550071be0 2024-03-16T19:29:38.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.406+0000 7f5555748640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f553406b3b0 0x7f553406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:38.389 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.407+0000 7f5555748640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f553406b3b0 0x7f553406d870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f554c035000 tx=0x7f554c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:38.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.411+0000 7f5545ffb640 1 -- 172.21.15.172:0/2762399221 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5540054aa0 con 0x7f5550071be0 2024-03-16T19:29:38.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.619+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f55501ef790 con 0x7f5550071be0 2024-03-16T19:29:38.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.619+0000 7f5545ffb640 1 -- 172.21.15.172:0/2762399221 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5540017070 con 0x7f5550071be0 2024-03-16T19:29:38.600 INFO:teuthology.orchestra.run.smithi172.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f553406b3b0 msgr2=0x7f553406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f553406b3b0 0x7f553406d870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f554c035000 tx=0x7f554c030000 comp rx=0 tx=0).stop 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550071be0 msgr2=0x7f555015b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550071be0 0x7f555015b410 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f554000e9e0 tx=0x7f554000eea0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 shutdown_connections 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5550152b20 0x7f555014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5550071be0 0x7f555015b410 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f553406b3b0 0x7f553406d870 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 --2- 172.21.15.172:0/2762399221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f555006aed0 0x7f55501f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 >> 172.21.15.172:0/2762399221 conn(0x7f555006e8b0 msgr2=0x7f5550158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 shutdown_connections 2024-03-16T19:29:38.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:38.621+0000 7f55579ac640 1 -- 172.21.15.172:0/2762399221 wait complete. 2024-03-16T19:29:38.613 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-16T19:29:38.613 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T19:29:38.613 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi172.front.sepia.ceph.com 2024-03-16T19:29:38.613 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-16T19:29:38.613 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T19:29:38.613 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-16T19:29:38.613 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T19:29:38.613 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T19:29:38.613 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-16T19:29:38.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:38.614 DEBUG:teuthology.orchestra.run.smithi172:> ip addr 2024-03-16T19:29:38.628 INFO:teuthology.orchestra.run.smithi172.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T19:29:38.628 INFO:teuthology.orchestra.run.smithi172.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T19:29:38.628 INFO:teuthology.orchestra.run.smithi172.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T19:29:38.628 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 ::1/128 scope host 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:d9:17:2c brd ff:ff:ff:ff:ff:ff 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: altname enp7s0f0 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:d9:17:2d brd ff:ff:ff:ff:ff:ff 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: altname enp7s0f1 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:88:7f:90 brd ff:ff:ff:ff:ff:ff 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: link/ether 0c:c4:7a:88:7f:91 brd ff:ff:ff:ff:ff:ff 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: inet 172.21.15.172/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft 42719sec preferred_lft 42719sec 2024-03-16T19:29:38.629 INFO:teuthology.orchestra.run.smithi172.stdout: inet6 fe80::ec4:7aff:fe88:7f91/64 scope link 2024-03-16T19:29:38.630 INFO:teuthology.orchestra.run.smithi172.stdout: valid_lft forever preferred_lft forever 2024-03-16T19:29:38.630 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T19:29:38.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:38.630 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2024-03-16T19:29:38.630 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2024-03-16T19:29:38.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link add name ceph-brx type bridge 2024-03-16T19:29:38.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip addr flush dev ceph-brx 2024-03-16T19:29:38.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link set ceph-brx up 2024-03-16T19:29:38.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T19:29:38.631 DEBUG:teuthology.orchestra.run.smithi172:> ') 2024-03-16T19:29:38.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:38.778 DEBUG:teuthology.orchestra.run.smithi172:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T19:29:38.804 INFO:teuthology.orchestra.run.smithi172.stdout:1 2024-03-16T19:29:38.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:38.805 DEBUG:teuthology.orchestra.run.smithi172:> route 2024-03-16T19:29:38.877 INFO:teuthology.orchestra.run.smithi172.stdout:Kernel IP routing table 2024-03-16T19:29:38.877 INFO:teuthology.orchestra.run.smithi172.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T19:29:38.877 INFO:teuthology.orchestra.run.smithi172.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-16T19:29:38.877 INFO:teuthology.orchestra.run.smithi172.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-16T19:29:38.877 INFO:teuthology.orchestra.run.smithi172.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T19:29:38.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:38.877 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2024-03-16T19:29:38.878 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2024-03-16T19:29:38.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-16T19:29:38.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-16T19:29:38.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-16T19:29:38.878 DEBUG:teuthology.orchestra.run.smithi172:> ') 2024-03-16T19:29:39.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:39.138 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list 2024-03-16T19:29:39.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:39.153 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list-id 2024-03-16T19:29:39.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:39.208 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2024-03-16T19:29:39.208 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2024-03-16T19:29:39.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T19:29:39.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-16T19:29:39.208 DEBUG:teuthology.orchestra.run.smithi172:> ') 2024-03-16T19:29:39.292 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-16T19:29:39.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:39.292 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2024-03-16T19:29:39.292 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2024-03-16T19:29:39.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-16T19:29:39.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T19:29:39.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-16T19:29:39.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-16T19:29:39.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-16T19:29:39.293 DEBUG:teuthology.orchestra.run.smithi172:> ') 2024-03-16T19:29:39.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:39.442 DEBUG:teuthology.orchestra.run.smithi172:> (cd / && exec bash -c ' 2024-03-16T19:29:39.442 DEBUG:teuthology.orchestra.run.smithi172:> set -e 2024-03-16T19:29:39.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link set brx.0 up 2024-03-16T19:29:39.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T19:29:39.442 DEBUG:teuthology.orchestra.run.smithi172:> ') 2024-03-16T19:29:39.484 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T19:29:39.485 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-16T19:29:39.539 INFO:teuthology.orchestra.run.smithi172.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-16T19:29:39.540 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T19:29:39.540 DEBUG:teuthology.orchestra.run.smithi172:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-16T19:29:39.595 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-16T19:29:39.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:39.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-16T19:29:39.681 INFO:teuthology.orchestra.run.smithi172.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-16T19:29:39.681 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-16T19:29:39.788 INFO:teuthology.orchestra.run.smithi172.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-16T19:29:39.789 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-16T19:29:39.789 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: resolved to: "172.21.15.172:3300,172.21.15.174:3300,172.21.15.174:3301" 2024-03-16T19:29:39.789 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: trying mount with new device syntax: 0@6c00e600-a879-4eb6-acee-0fdf0f76922a.cephfs=/ 2024-03-16T19:29:39.789 INFO:teuthology.orchestra.run.smithi172.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.172:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2024-03-16T19:29:39.790 INFO:teuthology.orchestra.run.smithi172.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-16T19:29:39.790 INFO:teuthology.orchestra.run.smithi172.stdout: You just mounted a file system that supports labels which does not 2024-03-16T19:29:39.790 INFO:teuthology.orchestra.run.smithi172.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-16T19:29:39.790 INFO:teuthology.orchestra.run.smithi172.stdout: applications will generate AVC messages and not be allowed access to 2024-03-16T19:29:39.790 INFO:teuthology.orchestra.run.smithi172.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-16T19:29:39.792 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-16T19:29:39.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:39.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-16T19:29:39.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:39.826 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T19:29:40.290 INFO:teuthology.orchestra.run.smithi172.stdout:client4588 2024-03-16T19:29:40.290 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:29:40.290 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T19:29:40.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:40.306 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T19:29:40.368 INFO:teuthology.orchestra.run.smithi172.stdout:client4588 2024-03-16T19:29:40.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:40.368 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T19:29:40.431 INFO:teuthology.orchestra.run.smithi172.stdout:client4588 2024-03-16T19:29:40.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:29:40.431 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/6c00e600-a879-4eb6-acee-0fdf0f76922a.client4588/status') 2024-03-16T19:29:40.704 INFO:tasks.ceph.mds.f.smithi174.stderr:2024-03-16T19:29:40.723+0000 7efca60d8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-16T19:29:40.994 INFO:teuthology.orchestra.run.smithi172.stdout:instance: client.4588 (3)172.21.15.172:0/2627359036 2024-03-16T19:29:40.995 INFO:teuthology.orchestra.run.smithi172.stdout:blocklisted: false 2024-03-16T19:29:40.995 INFO:teuthology.orchestra.run.smithi172.stderr:0+1 records in 2024-03-16T19:29:40.995 INFO:teuthology.orchestra.run.smithi172.stderr:0+1 records out 2024-03-16T19:29:40.995 INFO:teuthology.orchestra.run.smithi172.stderr:71 bytes copied, 2.315e-05 s, 3.1 MB/s 2024-03-16T19:29:40.996 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-16T19:29:41.010 INFO:tasks.thrashosds:config is {'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-16T19:29:41.010 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-lusov-qdb-squid'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb-squid', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}} 2024-03-16T19:29:41.010 INFO:tasks.thrashosds:config is {'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-03-16T19:29:41.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-16T19:29:41.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.140+0000 7f2177dac640 1 Processor -- start 2024-03-16T19:29:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.140+0000 7f2177dac640 1 -- start start 2024-03-16T19:29:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.140+0000 7f2177dac640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2170158150 0x7f217006e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2177dac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006f4f0 0x7f2170165270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2177dac640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f217006e010 con 0x7f217006c500 2024-03-16T19:29:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2177dac640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21700c8f20 con 0x7f2170158150 2024-03-16T19:29:41.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2177dac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217015b580 con 0x7f217006f4f0 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2175347640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f217006c500 0x7f21700ce0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:55102/0 (socket says 172.21.15.172:55102) 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2175347640 1 -- 172.21.15.172:0/3582159098 learned_addr learned my addr 172.21.15.172:0/3582159098 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2176349640 1 --2- 172.21.15.172:0/3582159098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006f4f0 0x7f2170165270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2175b48640 1 --2- 172.21.15.172:0/3582159098 >> v2:172.21.15.174:3300/0 conn(0x7f2170158150 0x7f217006e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2174b46640 1 -- 172.21.15.172:0/3582159098 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133356054 0 0) 0x7f217006e010 con 0x7f217006c500 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2174b46640 1 -- 172.21.15.172:0/3582159098 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f215c003520 con 0x7f217006c500 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2176349640 1 -- 172.21.15.172:0/3582159098 >> v2:172.21.15.174:3300/0 conn(0x7f2170158150 msgr2=0x7f217006e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2176349640 1 --2- 172.21.15.172:0/3582159098 >> v2:172.21.15.174:3300/0 conn(0x7f2170158150 0x7f217006e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2176349640 1 -- 172.21.15.172:0/3582159098 >> v1:172.21.15.172:6789/0 conn(0x7f217006c500 legacy=0x7f21700ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2176349640 1 -- 172.21.15.172:0/3582159098 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217006d810 con 0x7f217006f4f0 2024-03-16T19:29:41.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.141+0000 7f2175b48640 1 --2- 172.21.15.172:0/3582159098 >> v2:172.21.15.174:3300/0 conn(0x7f2170158150 0x7f217006e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.142+0000 7f2176349640 1 --2- 172.21.15.172:0/3582159098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006f4f0 0x7f2170165270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2168009d90 tx=0x7f216802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2e98cc1ba8dc162 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.142+0000 7f2174b46640 1 -- 172.21.15.172:0/3582159098 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216802cbc0 con 0x7f217006f4f0 2024-03-16T19:29:41.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.142+0000 7f2174b46640 1 -- 172.21.15.172:0/3582159098 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216802cd20 con 0x7f217006f4f0 2024-03-16T19:29:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.142+0000 7f2174b46640 1 -- 172.21.15.172:0/3582159098 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168034640 con 0x7f217006f4f0 2024-03-16T19:29:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.143+0000 7f2177dac640 1 -- 172.21.15.172:0/3582159098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006f4f0 msgr2=0x7f2170165270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.143+0000 7f2177dac640 1 --2- 172.21.15.172:0/3582159098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006f4f0 0x7f2170165270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2168009d90 tx=0x7f216802be10 comp rx=0 tx=0).stop 2024-03-16T19:29:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.143+0000 7f2177dac640 1 -- 172.21.15.172:0/3582159098 shutdown_connections 2024-03-16T19:29:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.143+0000 7f2177dac640 1 --2- 172.21.15.172:0/3582159098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006f4f0 0x7f2170165270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.143+0000 7f2177dac640 1 --2- 172.21.15.172:0/3582159098 >> v2:172.21.15.174:3300/0 conn(0x7f2170158150 0x7f217006e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.143+0000 7f2177dac640 1 -- 172.21.15.172:0/3582159098 >> 172.21.15.172:0/3582159098 conn(0x7f2170066c20 msgr2=0x7f2170067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.143+0000 7f2177dac640 1 -- 172.21.15.172:0/3582159098 shutdown_connections 2024-03-16T19:29:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.143+0000 7f2177dac640 1 -- 172.21.15.172:0/3582159098 wait complete. 2024-03-16T19:29:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.144+0000 7f2177dac640 1 Processor -- start 2024-03-16T19:29:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.144+0000 7f2177dac640 1 -- start start 2024-03-16T19:29:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2177dac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006c500 0x7f21700c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2177dac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f217006f4f0 0x7f21701e74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2177dac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2170158150 0x7f21700ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2177dac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21700c6660 con 0x7f2170158150 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2177dac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21700c6180 con 0x7f217006f4f0 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2177dac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21701f0680 con 0x7f217006c500 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2176349640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2170158150 0x7f21700ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2175b48640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006c500 0x7f21700c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2176349640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2170158150 0x7f21700ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34832/0 (socket says 172.21.15.172:34832) 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2176349640 1 -- 172.21.15.172:0/1760260337 learned_addr learned my addr 172.21.15.172:0/1760260337 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2175347640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f217006f4f0 0x7f21701e74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2176349640 1 -- 172.21.15.172:0/1760260337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006c500 msgr2=0x7f21700c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2176349640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006c500 0x7f21700c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2176349640 1 -- 172.21.15.172:0/1760260337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f217006f4f0 msgr2=0x7f21701e74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2176349640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f217006f4f0 0x7f21701e74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2176349640 1 -- 172.21.15.172:0/1760260337 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c009400 con 0x7f2170158150 2024-03-16T19:29:41.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2175b48640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006c500 0x7f21700c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2176349640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2170158150 0x7f21700ca140 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2168009d90 tx=0x7f2168004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f21627fc640 1 -- 172.21.15.172:0/1760260337 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168007000 con 0x7f2170158150 2024-03-16T19:29:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f21627fc640 1 -- 172.21.15.172:0/1760260337 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216802ce90 con 0x7f2170158150 2024-03-16T19:29:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f21627fc640 1 -- 172.21.15.172:0/1760260337 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680408b0 con 0x7f2170158150 2024-03-16T19:29:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2168009a70 con 0x7f2170158150 2024-03-16T19:29:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.145+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21701f1650 con 0x7f2170158150 2024-03-16T19:29:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.146+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217006bf60 con 0x7f2170158150 2024-03-16T19:29:41.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.147+0000 7f21627fc640 1 -- 172.21.15.172:0/1760260337 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f2168040a10 con 0x7f2170158150 2024-03-16T19:29:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.147+0000 7f21627fc640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f215406b3b0 0x7f215406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.147+0000 7f21627fc640 1 -- 172.21.15.172:0/1760260337 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21680ac700 con 0x7f2170158150 2024-03-16T19:29:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.147+0000 7f2175b48640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f215406b3b0 0x7f215406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.148+0000 7f2175b48640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f215406b3b0 0x7f215406d870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f216c011380 tx=0x7f216c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:41.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.151+0000 7f21627fc640 1 -- 172.21.15.172:0/1760260337 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2168077440 con 0x7f2170158150 2024-03-16T19:29:41.302 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.321+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) v1 -- 0x7f21700681c0 con 0x7f2170158150 2024-03-16T19:29:41.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.324+0000 7f21627fc640 1 -- 172.21.15.172:0/1760260337 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v10)=0 v10) v1 ==== 136+0+0 (secure 0 0 0) 0x7f2168009450 con 0x7f2170158150 2024-03-16T19:29:41.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f215406b3b0 msgr2=0x7f215406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f215406b3b0 0x7f215406d870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f216c011380 tx=0x7f216c009190 comp rx=0 tx=0).stop 2024-03-16T19:29:41.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2170158150 msgr2=0x7f21700ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2170158150 0x7f21700ca140 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2168009d90 tx=0x7f2168004a40 comp rx=0 tx=0).stop 2024-03-16T19:29:41.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 shutdown_connections 2024-03-16T19:29:41.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2170158150 0x7f21700ca140 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f217006f4f0 0x7f21701e74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f215406b3b0 0x7f215406d870 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 --2- 172.21.15.172:0/1760260337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f217006c500 0x7f21700c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 >> 172.21.15.172:0/1760260337 conn(0x7f2170066c20 msgr2=0x7f2170158840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:41.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 shutdown_connections 2024-03-16T19:29:41.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.327+0000 7f2177dac640 1 -- 172.21.15.172:0/1760260337 wait complete. 2024-03-16T19:29:41.319 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-16T19:29:41.319 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-16T19:29:41.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:29:41.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.440+0000 7f86167b6640 1 Processor -- start 2024-03-16T19:29:41.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.440+0000 7f86167b6640 1 -- start start 2024-03-16T19:29:41.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f86167b6640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f861006bb10 0x7f86100cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f86167b6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 0x7f86100cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f86167b6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86100cc720 con 0x7f861006bef0 2024-03-16T19:29:41.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f86167b6640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86100cc9c0 con 0x7f861006bb10 2024-03-16T19:29:41.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f86167b6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861015de30 con 0x7f86100cc260 2024-03-16T19:29:41.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f860ffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f861006bb10 0x7f86100cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f860f7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f861006bef0 0x7f8610154660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:55110/0 (socket says 172.21.15.172:55110) 2024-03-16T19:29:41.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f860f7fe640 1 -- 172.21.15.172:0/1172737707 learned_addr learned my addr 172.21.15.172:0/1172737707 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:41.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f8614d53640 1 --2- 172.21.15.172:0/1172737707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 0x7f86100cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f860ffff640 1 -- 172.21.15.172:0/1172737707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 msgr2=0x7f86100cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:41.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f860ffff640 1 --2- 172.21.15.172:0/1172737707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 0x7f86100cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f860ffff640 1 -- 172.21.15.172:0/1172737707 >> v1:172.21.15.172:6789/0 conn(0x7f861006bef0 legacy=0x7f8610154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f860ffff640 1 -- 172.21.15.172:0/1172737707 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861015e110 con 0x7f861006bb10 2024-03-16T19:29:41.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.441+0000 7f860effd640 1 -- 172.21.15.172:0/1172737707 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549715469 0 0) 0x7f86100cc720 con 0x7f861006bef0 2024-03-16T19:29:41.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f860ffff640 1 --2- 172.21.15.172:0/1172737707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 0x7f86100cae50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f86000097a0 tx=0x7f860002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6ead9078d1eed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:41.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f860effd640 1 -- 172.21.15.172:0/1172737707 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600030610 con 0x7f861006bb10 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f860effd640 1 -- 172.21.15.172:0/1172737707 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8600030770 con 0x7f861006bb10 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f860effd640 1 -- 172.21.15.172:0/1172737707 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600030b00 con 0x7f861006bb10 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f86167b6640 1 -- 172.21.15.172:0/1172737707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 msgr2=0x7f86100cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f86167b6640 1 --2- 172.21.15.172:0/1172737707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 0x7f86100cae50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f86000097a0 tx=0x7f860002b960 comp rx=0 tx=0).stop 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f86167b6640 1 -- 172.21.15.172:0/1172737707 shutdown_connections 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f86167b6640 1 --2- 172.21.15.172:0/1172737707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 0x7f86100cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f86167b6640 1 --2- 172.21.15.172:0/1172737707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 0x7f86100cae50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f86167b6640 1 -- 172.21.15.172:0/1172737707 >> 172.21.15.172:0/1172737707 conn(0x7f8610067250 msgr2=0x7f86100d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.442+0000 7f86167b6640 1 -- 172.21.15.172:0/1172737707 shutdown_connections 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.443+0000 7f86167b6640 1 -- 172.21.15.172:0/1172737707 wait complete. 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.444+0000 7f86167b6640 1 Processor -- start 2024-03-16T19:29:41.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.444+0000 7f86167b6640 1 -- start start 2024-03-16T19:29:41.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.444+0000 7f86167b6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 0x7f86101e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.444+0000 7f86167b6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861006bef0 0x7f86100cf380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.444+0000 7f86167b6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 0x7f86101f0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.444+0000 7f86167b6640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861015bfb0 con 0x7f861006bb10 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.444+0000 7f86167b6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861015c7f0 con 0x7f861006bef0 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.444+0000 7f86167b6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86101f3c80 con 0x7f86100cc260 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 0x7f86101e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 0x7f86101e78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34864/0 (socket says 172.21.15.172:34864) 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860ffff640 1 -- 172.21.15.172:0/1792514717 learned_addr learned my addr 172.21.15.172:0/1792514717 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f8614d53640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 0x7f86101f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860f7fe640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861006bef0 0x7f86100cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860ffff640 1 -- 172.21.15.172:0/1792514717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 msgr2=0x7f86101f0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860ffff640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 0x7f86101f0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860ffff640 1 -- 172.21.15.172:0/1792514717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861006bef0 msgr2=0x7f86100cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860ffff640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861006bef0 0x7f86100cf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860ffff640 1 -- 172.21.15.172:0/1792514717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8600009400 con 0x7f861006bb10 2024-03-16T19:29:41.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860ffff640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 0x7f86101e78d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f86000098c0 tx=0x7f8600004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:41.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860cff9640 1 -- 172.21.15.172:0/1792514717 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860003e040 con 0x7f861006bb10 2024-03-16T19:29:41.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860cff9640 1 -- 172.21.15.172:0/1792514717 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8600039cb0 con 0x7f861006bb10 2024-03-16T19:29:41.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86101f3f60 con 0x7f861006bb10 2024-03-16T19:29:41.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f860cff9640 1 -- 172.21.15.172:0/1792514717 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600044630 con 0x7f861006bb10 2024-03-16T19:29:41.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.445+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86101f4520 con 0x7f861006bb10 2024-03-16T19:29:41.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.446+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86101e7e10 con 0x7f861006bb10 2024-03-16T19:29:41.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.447+0000 7f860cff9640 1 -- 172.21.15.172:0/1792514717 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f860004d3b0 con 0x7f861006bb10 2024-03-16T19:29:41.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.447+0000 7f860cff9640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f85fc06b3b0 0x7f85fc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.447+0000 7f860f7fe640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f85fc06b3b0 0x7f85fc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.447+0000 7f860cff9640 1 -- 172.21.15.172:0/1792514717 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86000abfa0 con 0x7f861006bb10 2024-03-16T19:29:41.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.448+0000 7f860f7fe640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f85fc06b3b0 0x7f85fc06d870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f861015b940 tx=0x7f8604007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:41.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.452+0000 7f860cff9640 1 -- 172.21.15.172:0/1792514717 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8600076cb0 con 0x7f861006bb10 2024-03-16T19:29:41.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.621+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f861015e9b0 con 0x7f85fc06b3b0 2024-03-16T19:29:41.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.626+0000 7f860cff9640 1 -- 172.21.15.172:0/1792514717 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7f8604004c70 con 0x7f85fc06b3b0 2024-03-16T19:29:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:41.611 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:29:41.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.629+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f85fc06b3b0 msgr2=0x7f85fc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.629+0000 7f86167b6640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f85fc06b3b0 0x7f85fc06d870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f861015b940 tx=0x7f8604007d50 comp rx=0 tx=0).stop 2024-03-16T19:29:41.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 msgr2=0x7f86101e78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 0x7f86101e78d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f86000098c0 tx=0x7f8600004730 comp rx=0 tx=0).stop 2024-03-16T19:29:41.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 shutdown_connections 2024-03-16T19:29:41.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f85fc06b3b0 0x7f85fc06d870 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86100cc260 0x7f86101f0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861006bef0 0x7f86100cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 --2- 172.21.15.172:0/1792514717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f861006bb10 0x7f86101e78d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 >> 172.21.15.172:0/1792514717 conn(0x7f8610067250 msgr2=0x7f8610152050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:41.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 shutdown_connections 2024-03-16T19:29:41.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.630+0000 7f86167b6640 1 -- 172.21.15.172:0/1792514717 wait complete. 2024-03-16T19:29:41.621 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-16T19:29:40.202800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219048,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755672,"statfs":{"total":773094113280,"available":772869808128,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002817"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344501+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:29:27.344501+0000","last_peered":"2024-03-16T19:29:27.344501+0000","last_clean":"2024-03-16T19:29:27.344501+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:29:27.344501+0000","last_undegraded":"2024-03-16T19:29:27.344501+0000","last_fullsized":"2024-03-16T19:29:27.344501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339631+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:29:27.339631+0000","last_peered":"2024-03-16T19:29:27.339631+0000","last_clean":"2024-03-16T19:29:27.339631+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:29:27.339631+0000","last_undegraded":"2024-03-16T19:29:27.339631+0000","last_fullsized":"2024-03-16T19:29:27.339631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062822+0000","last_change":"2024-03-16T19:29:26.347979+0000","last_active":"2024-03-16T19:29:28.062822+0000","last_peered":"2024-03-16T19:29:28.062822+0000","last_clean":"2024-03-16T19:29:28.062822+0000","last_became_active":"2024-03-16T19:29:26.347711+0000","last_became_peered":"2024-03-16T19:29:26.347711+0000","last_unstale":"2024-03-16T19:29:28.062822+0000","last_undegraded":"2024-03-16T19:29:28.062822+0000","last_fullsized":"2024-03-16T19:29:28.062822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:08.017182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343558+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:29:27.343558+0000","last_peered":"2024-03-16T19:29:27.343558+0000","last_clean":"2024-03-16T19:29:27.343558+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:29:27.343558+0000","last_undegraded":"2024-03-16T19:29:27.343558+0000","last_fullsized":"2024-03-16T19:29:27.343558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344270+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:29:27.344270+0000","last_peered":"2024-03-16T19:29:27.344270+0000","last_clean":"2024-03-16T19:29:27.344270+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:29:27.344270+0000","last_undegraded":"2024-03-16T19:29:27.344270+0000","last_fullsized":"2024-03-16T19:29:27.344270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343104+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:29:27.343104+0000","last_peered":"2024-03-16T19:29:27.343104+0000","last_clean":"2024-03-16T19:29:27.343104+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:29:27.343104+0000","last_undegraded":"2024-03-16T19:29:27.343104+0000","last_fullsized":"2024-03-16T19:29:27.343104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344269+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:29:27.344269+0000","last_peered":"2024-03-16T19:29:27.344269+0000","last_clean":"2024-03-16T19:29:27.344269+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:29:27.344269+0000","last_undegraded":"2024-03-16T19:29:27.344269+0000","last_fullsized":"2024-03-16T19:29:27.344269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561737+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:29:27.561737+0000","last_peered":"2024-03-16T19:29:27.561737+0000","last_clean":"2024-03-16T19:29:27.561737+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:29:27.561737+0000","last_undegraded":"2024-03-16T19:29:27.561737+0000","last_fullsized":"2024-03-16T19:29:27.561737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344497+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:29:27.344497+0000","last_peered":"2024-03-16T19:29:27.344497+0000","last_clean":"2024-03-16T19:29:27.344497+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:29:27.344497+0000","last_undegraded":"2024-03-16T19:29:27.344497+0000","last_fullsized":"2024-03-16T19:29:27.344497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063286+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:29:28.063286+0000","last_peered":"2024-03-16T19:29:28.063286+0000","last_clean":"2024-03-16T19:29:28.063286+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:29:28.063286+0000","last_undegraded":"2024-03-16T19:29:28.063286+0000","last_fullsized":"2024-03-16T19:29:28.063286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062996+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:29:28.062996+0000","last_peered":"2024-03-16T19:29:28.062996+0000","last_clean":"2024-03-16T19:29:28.062996+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:29:28.062996+0000","last_undegraded":"2024-03-16T19:29:28.062996+0000","last_fullsized":"2024-03-16T19:29:28.062996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063177+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:29:28.063177+0000","last_peered":"2024-03-16T19:29:28.063177+0000","last_clean":"2024-03-16T19:29:28.063177+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:29:28.063177+0000","last_undegraded":"2024-03-16T19:29:28.063177+0000","last_fullsized":"2024-03-16T19:29:28.063177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561930+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:29:27.561930+0000","last_peered":"2024-03-16T19:29:27.561930+0000","last_clean":"2024-03-16T19:29:27.561930+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:29:27.561930+0000","last_undegraded":"2024-03-16T19:29:27.561930+0000","last_fullsized":"2024-03-16T19:29:27.561930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062462+0000","last_change":"2024-03-16T19:29:26.349644+0000","last_active":"2024-03-16T19:29:28.062462+0000","last_peered":"2024-03-16T19:29:28.062462+0000","last_clean":"2024-03-16T19:29:28.062462+0000","last_became_active":"2024-03-16T19:29:26.349397+0000","last_became_peered":"2024-03-16T19:29:26.349397+0000","last_unstale":"2024-03-16T19:29:28.062462+0000","last_undegraded":"2024-03-16T19:29:28.062462+0000","last_fullsized":"2024-03-16T19:29:28.062462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:47:20.566393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339397+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:29:27.339397+0000","last_peered":"2024-03-16T19:29:27.339397+0000","last_clean":"2024-03-16T19:29:27.339397+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:29:27.339397+0000","last_undegraded":"2024-03-16T19:29:27.339397+0000","last_fullsized":"2024-03-16T19:29:27.339397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344311+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:29:27.344311+0000","last_peered":"2024-03-16T19:29:27.344311+0000","last_clean":"2024-03-16T19:29:27.344311+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:29:27.344311+0000","last_undegraded":"2024-03-16T19:29:27.344311+0000","last_fullsized":"2024-03-16T19:29:27.344311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062569+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:29:28.062569+0000","last_peered":"2024-03-16T19:29:28.062569+0000","last_clean":"2024-03-16T19:29:28.062569+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:29:28.062569+0000","last_undegraded":"2024-03-16T19:29:28.062569+0000","last_fullsized":"2024-03-16T19:29:28.062569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344648+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:29:27.344648+0000","last_peered":"2024-03-16T19:29:27.344648+0000","last_clean":"2024-03-16T19:29:27.344648+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:29:27.344648+0000","last_undegraded":"2024-03-16T19:29:27.344648+0000","last_fullsized":"2024-03-16T19:29:27.344648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063112+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:29:28.063112+0000","last_peered":"2024-03-16T19:29:28.063112+0000","last_clean":"2024-03-16T19:29:28.063112+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:29:28.063112+0000","last_undegraded":"2024-03-16T19:29:28.063112+0000","last_fullsized":"2024-03-16T19:29:28.063112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062498+0000","last_change":"2024-03-16T19:29:26.357191+0000","last_active":"2024-03-16T19:29:28.062498+0000","last_peered":"2024-03-16T19:29:28.062498+0000","last_clean":"2024-03-16T19:29:28.062498+0000","last_became_active":"2024-03-16T19:29:26.356947+0000","last_became_peered":"2024-03-16T19:29:26.356947+0000","last_unstale":"2024-03-16T19:29:28.062498+0000","last_undegraded":"2024-03-16T19:29:28.062498+0000","last_fullsized":"2024-03-16T19:29:28.062498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:40.884164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343265+0000","last_change":"2024-03-16T19:29:26.358307+0000","last_active":"2024-03-16T19:29:27.343265+0000","last_peered":"2024-03-16T19:29:27.343265+0000","last_clean":"2024-03-16T19:29:27.343265+0000","last_became_active":"2024-03-16T19:29:26.357970+0000","last_became_peered":"2024-03-16T19:29:26.357970+0000","last_unstale":"2024-03-16T19:29:27.343265+0000","last_undegraded":"2024-03-16T19:29:27.343265+0000","last_fullsized":"2024-03-16T19:29:27.343265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:33.976868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561749+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:29:27.561749+0000","last_peered":"2024-03-16T19:29:27.561749+0000","last_clean":"2024-03-16T19:29:27.561749+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:29:27.561749+0000","last_undegraded":"2024-03-16T19:29:27.561749+0000","last_fullsized":"2024-03-16T19:29:27.561749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343286+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:29:27.343286+0000","last_peered":"2024-03-16T19:29:27.343286+0000","last_clean":"2024-03-16T19:29:27.343286+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:29:27.343286+0000","last_undegraded":"2024-03-16T19:29:27.343286+0000","last_fullsized":"2024-03-16T19:29:27.343286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562192+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:29:27.562192+0000","last_peered":"2024-03-16T19:29:27.562192+0000","last_clean":"2024-03-16T19:29:27.562192+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:29:27.562192+0000","last_undegraded":"2024-03-16T19:29:27.562192+0000","last_fullsized":"2024-03-16T19:29:27.562192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063443+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:29:28.063443+0000","last_peered":"2024-03-16T19:29:28.063443+0000","last_clean":"2024-03-16T19:29:28.063443+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:29:28.063443+0000","last_undegraded":"2024-03-16T19:29:28.063443+0000","last_fullsized":"2024-03-16T19:29:28.063443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339564+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:29:27.339564+0000","last_peered":"2024-03-16T19:29:27.339564+0000","last_clean":"2024-03-16T19:29:27.339564+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:29:27.339564+0000","last_undegraded":"2024-03-16T19:29:27.339564+0000","last_fullsized":"2024-03-16T19:29:27.339564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562214+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:29:27.562214+0000","last_peered":"2024-03-16T19:29:27.562214+0000","last_clean":"2024-03-16T19:29:27.562214+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:29:27.562214+0000","last_undegraded":"2024-03-16T19:29:27.562214+0000","last_fullsized":"2024-03-16T19:29:27.562214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062637+0000","last_change":"2024-03-16T19:29:26.349443+0000","last_active":"2024-03-16T19:29:28.062637+0000","last_peered":"2024-03-16T19:29:28.062637+0000","last_clean":"2024-03-16T19:29:28.062637+0000","last_became_active":"2024-03-16T19:29:26.349181+0000","last_became_peered":"2024-03-16T19:29:26.349181+0000","last_unstale":"2024-03-16T19:29:28.062637+0000","last_undegraded":"2024-03-16T19:29:28.062637+0000","last_fullsized":"2024-03-16T19:29:28.062637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:16.483346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343433+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:29:27.343433+0000","last_peered":"2024-03-16T19:29:27.343433+0000","last_clean":"2024-03-16T19:29:27.343433+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:29:27.343433+0000","last_undegraded":"2024-03-16T19:29:27.343433+0000","last_fullsized":"2024-03-16T19:29:27.343433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062168+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:29:28.062168+0000","last_peered":"2024-03-16T19:29:28.062168+0000","last_clean":"2024-03-16T19:29:28.062168+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:29:28.062168+0000","last_undegraded":"2024-03-16T19:29:28.062168+0000","last_fullsized":"2024-03-16T19:29:28.062168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062728+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:29:28.062728+0000","last_peered":"2024-03-16T19:29:28.062728+0000","last_clean":"2024-03-16T19:29:28.062728+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:29:28.062728+0000","last_undegraded":"2024-03-16T19:29:28.062728+0000","last_fullsized":"2024-03-16T19:29:28.062728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063484+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:29:28.063484+0000","last_peered":"2024-03-16T19:29:28.063484+0000","last_clean":"2024-03-16T19:29:28.063484+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:29:28.063484+0000","last_undegraded":"2024-03-16T19:29:28.063484+0000","last_fullsized":"2024-03-16T19:29:28.063484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.215645+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:29:28.215645+0000","last_peered":"2024-03-16T19:29:28.215645+0000","last_clean":"2024-03-16T19:29:28.215645+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:29:28.215645+0000","last_undegraded":"2024-03-16T19:29:28.215645+0000","last_fullsized":"2024-03-16T19:29:28.215645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062765+0000","last_change":"2024-03-16T19:29:26.353288+0000","last_active":"2024-03-16T19:29:28.062765+0000","last_peered":"2024-03-16T19:29:28.062765+0000","last_clean":"2024-03-16T19:29:28.062765+0000","last_became_active":"2024-03-16T19:29:26.352973+0000","last_became_peered":"2024-03-16T19:29:26.352973+0000","last_unstale":"2024-03-16T19:29:28.062765+0000","last_undegraded":"2024-03-16T19:29:28.062765+0000","last_fullsized":"2024-03-16T19:29:28.062765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:22:41.505003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.210595+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:29:28.210595+0000","last_peered":"2024-03-16T19:29:28.210595+0000","last_clean":"2024-03-16T19:29:28.210595+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:29:28.210595+0000","last_undegraded":"2024-03-16T19:29:28.210595+0000","last_fullsized":"2024-03-16T19:29:28.210595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561856+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:29:27.561856+0000","last_peered":"2024-03-16T19:29:27.561856+0000","last_clean":"2024-03-16T19:29:27.561856+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:29:27.561856+0000","last_undegraded":"2024-03-16T19:29:27.561856+0000","last_fullsized":"2024-03-16T19:29:27.561856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561778+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:29:27.561778+0000","last_peered":"2024-03-16T19:29:27.561778+0000","last_clean":"2024-03-16T19:29:27.561778+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:29:27.561778+0000","last_undegraded":"2024-03-16T19:29:27.561778+0000","last_fullsized":"2024-03-16T19:29:27.561778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343316+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:29:27.343316+0000","last_peered":"2024-03-16T19:29:27.343316+0000","last_clean":"2024-03-16T19:29:27.343316+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:29:27.343316+0000","last_undegraded":"2024-03-16T19:29:27.343316+0000","last_fullsized":"2024-03-16T19:29:27.343316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339401+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:29:27.339401+0000","last_peered":"2024-03-16T19:29:27.339401+0000","last_clean":"2024-03-16T19:29:27.339401+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:29:27.339401+0000","last_undegraded":"2024-03-16T19:29:27.339401+0000","last_fullsized":"2024-03-16T19:29:27.339401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063312+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:29:28.063312+0000","last_peered":"2024-03-16T19:29:28.063312+0000","last_clean":"2024-03-16T19:29:28.063312+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:29:28.063312+0000","last_undegraded":"2024-03-16T19:29:28.063312+0000","last_fullsized":"2024-03-16T19:29:28.063312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344324+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:29:27.344324+0000","last_peered":"2024-03-16T19:29:27.344324+0000","last_clean":"2024-03-16T19:29:27.344324+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:29:27.344324+0000","last_undegraded":"2024-03-16T19:29:27.344324+0000","last_fullsized":"2024-03-16T19:29:27.344324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343744+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:29:27.343744+0000","last_peered":"2024-03-16T19:29:27.343744+0000","last_clean":"2024-03-16T19:29:27.343744+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:29:27.343744+0000","last_undegraded":"2024-03-16T19:29:27.343744+0000","last_fullsized":"2024-03-16T19:29:27.343744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062922+0000","last_change":"2024-03-16T19:29:26.353918+0000","last_active":"2024-03-16T19:29:28.062922+0000","last_peered":"2024-03-16T19:29:28.062922+0000","last_clean":"2024-03-16T19:29:28.062922+0000","last_became_active":"2024-03-16T19:29:26.353463+0000","last_became_peered":"2024-03-16T19:29:26.353463+0000","last_unstale":"2024-03-16T19:29:28.062922+0000","last_undegraded":"2024-03-16T19:29:28.062922+0000","last_fullsized":"2024-03-16T19:29:28.062922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:36.035692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.362815+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:29:31.362815+0000","last_peered":"2024-03-16T19:29:31.362815+0000","last_clean":"2024-03-16T19:29:31.362815+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:29:31.362815+0000","last_undegraded":"2024-03-16T19:29:31.362815+0000","last_fullsized":"2024-03-16T19:29:31.362815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063360+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:29:28.063360+0000","last_peered":"2024-03-16T19:29:28.063360+0000","last_clean":"2024-03-16T19:29:28.063360+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:29:28.063360+0000","last_undegraded":"2024-03-16T19:29:28.063360+0000","last_fullsized":"2024-03-16T19:29:28.063360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365125+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:29:31.365125+0000","last_peered":"2024-03-16T19:29:31.365125+0000","last_clean":"2024-03-16T19:29:31.365125+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:29:31.365125+0000","last_undegraded":"2024-03-16T19:29:31.365125+0000","last_fullsized":"2024-03-16T19:29:31.365125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344465+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:29:27.344465+0000","last_peered":"2024-03-16T19:29:27.344465+0000","last_clean":"2024-03-16T19:29:27.344465+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:29:27.344465+0000","last_undegraded":"2024-03-16T19:29:27.344465+0000","last_fullsized":"2024-03-16T19:29:27.344465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365001+0000","last_change":"2024-03-16T19:29:25.336296+0000","last_active":"2024-03-16T19:29:31.365001+0000","last_peered":"2024-03-16T19:29:31.365001+0000","last_clean":"2024-03-16T19:29:31.365001+0000","last_became_active":"2024-03-16T19:29:25.333046+0000","last_became_peered":"2024-03-16T19:29:25.333046+0000","last_unstale":"2024-03-16T19:29:31.365001+0000","last_undegraded":"2024-03-16T19:29:31.365001+0000","last_fullsized":"2024-03-16T19:29:31.365001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:33.538342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062507+0000","last_change":"2024-03-16T19:29:26.354709+0000","last_active":"2024-03-16T19:29:28.062507+0000","last_peered":"2024-03-16T19:29:28.062507+0000","last_clean":"2024-03-16T19:29:28.062507+0000","last_became_active":"2024-03-16T19:29:26.354377+0000","last_became_peered":"2024-03-16T19:29:26.354377+0000","last_unstale":"2024-03-16T19:29:28.062507+0000","last_undegraded":"2024-03-16T19:29:28.062507+0000","last_fullsized":"2024-03-16T19:29:28.062507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:41:56.277270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343553+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:29:27.343553+0000","last_peered":"2024-03-16T19:29:27.343553+0000","last_clean":"2024-03-16T19:29:27.343553+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:29:27.343553+0000","last_undegraded":"2024-03-16T19:29:27.343553+0000","last_fullsized":"2024-03-16T19:29:27.343553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344364+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:29:27.344364+0000","last_peered":"2024-03-16T19:29:27.344364+0000","last_clean":"2024-03-16T19:29:27.344364+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:29:27.344364+0000","last_undegraded":"2024-03-16T19:29:27.344364+0000","last_fullsized":"2024-03-16T19:29:27.344364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062737+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:29:28.062737+0000","last_peered":"2024-03-16T19:29:28.062737+0000","last_clean":"2024-03-16T19:29:28.062737+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:29:28.062737+0000","last_undegraded":"2024-03-16T19:29:28.062737+0000","last_fullsized":"2024-03-16T19:29:28.062737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062954+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:29:28.062954+0000","last_peered":"2024-03-16T19:29:28.062954+0000","last_clean":"2024-03-16T19:29:28.062954+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:29:28.062954+0000","last_undegraded":"2024-03-16T19:29:28.062954+0000","last_fullsized":"2024-03-16T19:29:28.062954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365607+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:29:31.365607+0000","last_peered":"2024-03-16T19:29:31.365607+0000","last_clean":"2024-03-16T19:29:31.365607+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:29:31.365607+0000","last_undegraded":"2024-03-16T19:29:31.365607+0000","last_fullsized":"2024-03-16T19:29:31.365607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344168+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:29:27.344168+0000","last_peered":"2024-03-16T19:29:27.344168+0000","last_clean":"2024-03-16T19:29:27.344168+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:29:27.344168+0000","last_undegraded":"2024-03-16T19:29:27.344168+0000","last_fullsized":"2024-03-16T19:29:27.344168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.211252+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:29:28.211252+0000","last_peered":"2024-03-16T19:29:28.211252+0000","last_clean":"2024-03-16T19:29:28.211252+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:29:28.211252+0000","last_undegraded":"2024-03-16T19:29:28.211252+0000","last_fullsized":"2024-03-16T19:29:28.211252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344198+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:29:27.344198+0000","last_peered":"2024-03-16T19:29:27.344198+0000","last_clean":"2024-03-16T19:29:27.344198+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:29:27.344198+0000","last_undegraded":"2024-03-16T19:29:27.344198+0000","last_fullsized":"2024-03-16T19:29:27.344198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.362435+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:29:31.362435+0000","last_peered":"2024-03-16T19:29:31.362435+0000","last_clean":"2024-03-16T19:29:31.362435+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:29:31.362435+0000","last_undegraded":"2024-03-16T19:29:31.362435+0000","last_fullsized":"2024-03-16T19:29:31.362435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344664+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:29:27.344664+0000","last_peered":"2024-03-16T19:29:27.344664+0000","last_clean":"2024-03-16T19:29:27.344664+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:29:27.344664+0000","last_undegraded":"2024-03-16T19:29:27.344664+0000","last_fullsized":"2024-03-16T19:29:27.344664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.219180+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:29:28.219180+0000","last_peered":"2024-03-16T19:29:28.219180+0000","last_clean":"2024-03-16T19:29:28.219180+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:29:28.219180+0000","last_undegraded":"2024-03-16T19:29:28.219180+0000","last_fullsized":"2024-03-16T19:29:28.219180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344128+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:29:27.344128+0000","last_peered":"2024-03-16T19:29:27.344128+0000","last_clean":"2024-03-16T19:29:27.344128+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:29:27.344128+0000","last_undegraded":"2024-03-16T19:29:27.344128+0000","last_fullsized":"2024-03-16T19:29:27.344128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062516+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:29:28.062516+0000","last_peered":"2024-03-16T19:29:28.062516+0000","last_clean":"2024-03-16T19:29:28.062516+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:29:28.062516+0000","last_undegraded":"2024-03-16T19:29:28.062516+0000","last_fullsized":"2024-03-16T19:29:28.062516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344351+0000","last_change":"2024-03-16T19:29:26.364742+0000","last_active":"2024-03-16T19:29:27.344351+0000","last_peered":"2024-03-16T19:29:27.344351+0000","last_clean":"2024-03-16T19:29:27.344351+0000","last_became_active":"2024-03-16T19:29:26.364328+0000","last_became_peered":"2024-03-16T19:29:26.364328+0000","last_unstale":"2024-03-16T19:29:27.344351+0000","last_undegraded":"2024-03-16T19:29:27.344351+0000","last_fullsized":"2024-03-16T19:29:27.344351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:18.877774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343226+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:29:27.343226+0000","last_peered":"2024-03-16T19:29:27.343226+0000","last_clean":"2024-03-16T19:29:27.343226+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:29:27.343226+0000","last_undegraded":"2024-03-16T19:29:27.343226+0000","last_fullsized":"2024-03-16T19:29:27.343226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561997+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:29:27.561997+0000","last_peered":"2024-03-16T19:29:27.561997+0000","last_clean":"2024-03-16T19:29:27.561997+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:29:27.561997+0000","last_undegraded":"2024-03-16T19:29:27.561997+0000","last_fullsized":"2024-03-16T19:29:27.561997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343035+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:29:27.343035+0000","last_peered":"2024-03-16T19:29:27.343035+0000","last_clean":"2024-03-16T19:29:27.343035+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:29:27.343035+0000","last_undegraded":"2024-03-16T19:29:27.343035+0000","last_fullsized":"2024-03-16T19:29:27.343035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062305+0000","last_change":"2024-03-16T19:29:26.352692+0000","last_active":"2024-03-16T19:29:28.062305+0000","last_peered":"2024-03-16T19:29:28.062305+0000","last_clean":"2024-03-16T19:29:28.062305+0000","last_became_active":"2024-03-16T19:29:26.352419+0000","last_became_peered":"2024-03-16T19:29:26.352419+0000","last_unstale":"2024-03-16T19:29:28.062305+0000","last_undegraded":"2024-03-16T19:29:28.062305+0000","last_fullsized":"2024-03-16T19:29:28.062305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:33.056538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062749+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:29:28.062749+0000","last_peered":"2024-03-16T19:29:28.062749+0000","last_clean":"2024-03-16T19:29:28.062749+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:29:28.062749+0000","last_undegraded":"2024-03-16T19:29:28.062749+0000","last_fullsized":"2024-03-16T19:29:28.062749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339393+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:29:27.339393+0000","last_peered":"2024-03-16T19:29:27.339393+0000","last_clean":"2024-03-16T19:29:27.339393+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:29:27.339393+0000","last_undegraded":"2024-03-16T19:29:27.339393+0000","last_fullsized":"2024-03-16T19:29:27.339393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344073+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:29:27.344073+0000","last_peered":"2024-03-16T19:29:27.344073+0000","last_clean":"2024-03-16T19:29:27.344073+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:29:27.344073+0000","last_undegraded":"2024-03-16T19:29:27.344073+0000","last_fullsized":"2024-03-16T19:29:27.344073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063637+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:29:28.063637+0000","last_peered":"2024-03-16T19:29:28.063637+0000","last_clean":"2024-03-16T19:29:28.063637+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:29:28.063637+0000","last_undegraded":"2024-03-16T19:29:28.063637+0000","last_fullsized":"2024-03-16T19:29:28.063637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339499+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:29:27.339499+0000","last_peered":"2024-03-16T19:29:27.339499+0000","last_clean":"2024-03-16T19:29:27.339499+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:29:27.339499+0000","last_undegraded":"2024-03-16T19:29:27.339499+0000","last_fullsized":"2024-03-16T19:29:27.339499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343975+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:29:27.343975+0000","last_peered":"2024-03-16T19:29:27.343975+0000","last_clean":"2024-03-16T19:29:27.343975+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:29:27.343975+0000","last_undegraded":"2024-03-16T19:29:27.343975+0000","last_fullsized":"2024-03-16T19:29:27.343975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344573+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:29:27.344573+0000","last_peered":"2024-03-16T19:29:27.344573+0000","last_clean":"2024-03-16T19:29:27.344573+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:29:27.344573+0000","last_undegraded":"2024-03-16T19:29:27.344573+0000","last_fullsized":"2024-03-16T19:29:27.344573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339540+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:29:27.339540+0000","last_peered":"2024-03-16T19:29:27.339540+0000","last_clean":"2024-03-16T19:29:27.339540+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:29:27.339540+0000","last_undegraded":"2024-03-16T19:29:27.339540+0000","last_fullsized":"2024-03-16T19:29:27.339540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.219205+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:29:28.219205+0000","last_peered":"2024-03-16T19:29:28.219205+0000","last_clean":"2024-03-16T19:29:28.219205+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:29:28.219205+0000","last_undegraded":"2024-03-16T19:29:28.219205+0000","last_fullsized":"2024-03-16T19:29:28.219205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562225+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:29:27.562225+0000","last_peered":"2024-03-16T19:29:27.562225+0000","last_clean":"2024-03-16T19:29:27.562225+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:29:27.562225+0000","last_undegraded":"2024-03-16T19:29:27.562225+0000","last_fullsized":"2024-03-16T19:29:27.562225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.360976+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:29:31.360976+0000","last_peered":"2024-03-16T19:29:31.360976+0000","last_clean":"2024-03-16T19:29:31.360976+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:29:31.360976+0000","last_undegraded":"2024-03-16T19:29:31.360976+0000","last_fullsized":"2024-03-16T19:29:31.360976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062725+0000","last_change":"2024-03-16T19:29:26.350351+0000","last_active":"2024-03-16T19:29:28.062725+0000","last_peered":"2024-03-16T19:29:28.062725+0000","last_clean":"2024-03-16T19:29:28.062725+0000","last_became_active":"2024-03-16T19:29:26.350074+0000","last_became_peered":"2024-03-16T19:29:26.350074+0000","last_unstale":"2024-03-16T19:29:28.062725+0000","last_undegraded":"2024-03-16T19:29:28.062725+0000","last_fullsized":"2024-03-16T19:29:28.062725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:49.416395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.342931+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:29:27.342931+0000","last_peered":"2024-03-16T19:29:27.342931+0000","last_clean":"2024-03-16T19:29:27.342931+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:29:27.342931+0000","last_undegraded":"2024-03-16T19:29:27.342931+0000","last_fullsized":"2024-03-16T19:29:27.342931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063379+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:29:28.063379+0000","last_peered":"2024-03-16T19:29:28.063379+0000","last_clean":"2024-03-16T19:29:28.063379+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:29:28.063379+0000","last_undegraded":"2024-03-16T19:29:28.063379+0000","last_fullsized":"2024-03-16T19:29:28.063379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339690+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:29:27.339690+0000","last_peered":"2024-03-16T19:29:27.339690+0000","last_clean":"2024-03-16T19:29:27.339690+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:29:27.339690+0000","last_undegraded":"2024-03-16T19:29:27.339690+0000","last_fullsized":"2024-03-16T19:29:27.339690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561913+0000","last_change":"2024-03-16T19:29:26.347388+0000","last_active":"2024-03-16T19:29:27.561913+0000","last_peered":"2024-03-16T19:29:27.561913+0000","last_clean":"2024-03-16T19:29:27.561913+0000","last_became_active":"2024-03-16T19:29:26.346582+0000","last_became_peered":"2024-03-16T19:29:26.346582+0000","last_unstale":"2024-03-16T19:29:27.561913+0000","last_undegraded":"2024-03-16T19:29:27.561913+0000","last_fullsized":"2024-03-16T19:29:27.561913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:05.294895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339497+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:29:27.339497+0000","last_peered":"2024-03-16T19:29:27.339497+0000","last_clean":"2024-03-16T19:29:27.339497+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:29:27.339497+0000","last_undegraded":"2024-03-16T19:29:27.339497+0000","last_fullsized":"2024-03-16T19:29:27.339497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561968+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:29:27.561968+0000","last_peered":"2024-03-16T19:29:27.561968+0000","last_clean":"2024-03-16T19:29:27.561968+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:29:27.561968+0000","last_undegraded":"2024-03-16T19:29:27.561968+0000","last_fullsized":"2024-03-16T19:29:27.561968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365383+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:29:31.365383+0000","last_peered":"2024-03-16T19:29:31.365383+0000","last_clean":"2024-03-16T19:29:31.365383+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:29:31.365383+0000","last_undegraded":"2024-03-16T19:29:31.365383+0000","last_fullsized":"2024-03-16T19:29:31.365383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062837+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:29:28.062837+0000","last_peered":"2024-03-16T19:29:28.062837+0000","last_clean":"2024-03-16T19:29:28.062837+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:29:28.062837+0000","last_undegraded":"2024-03-16T19:29:28.062837+0000","last_fullsized":"2024-03-16T19:29:28.062837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343809+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:29:27.343809+0000","last_peered":"2024-03-16T19:29:27.343809+0000","last_clean":"2024-03-16T19:29:27.343809+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:29:27.343809+0000","last_undegraded":"2024-03-16T19:29:27.343809+0000","last_fullsized":"2024-03-16T19:29:27.343809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.360603+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:29:31.360603+0000","last_peered":"2024-03-16T19:29:31.360603+0000","last_clean":"2024-03-16T19:29:31.360603+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:29:31.360603+0000","last_undegraded":"2024-03-16T19:29:31.360603+0000","last_fullsized":"2024-03-16T19:29:31.360603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344208+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:29:27.344208+0000","last_peered":"2024-03-16T19:29:27.344208+0000","last_clean":"2024-03-16T19:29:27.344208+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:29:27.344208+0000","last_undegraded":"2024-03-16T19:29:27.344208+0000","last_fullsized":"2024-03-16T19:29:27.344208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365202+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:29:31.365202+0000","last_peered":"2024-03-16T19:29:31.365202+0000","last_clean":"2024-03-16T19:29:31.365202+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:29:31.365202+0000","last_undegraded":"2024-03-16T19:29:31.365202+0000","last_fullsized":"2024-03-16T19:29:31.365202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062891+0000","last_change":"2024-03-16T19:29:26.354303+0000","last_active":"2024-03-16T19:29:28.062891+0000","last_peered":"2024-03-16T19:29:28.062891+0000","last_clean":"2024-03-16T19:29:28.062891+0000","last_became_active":"2024-03-16T19:29:26.353804+0000","last_became_peered":"2024-03-16T19:29:26.353804+0000","last_unstale":"2024-03-16T19:29:28.062891+0000","last_undegraded":"2024-03-16T19:29:28.062891+0000","last_fullsized":"2024-03-16T19:29:28.062891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:10:45.368536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.215673+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:29:28.215673+0000","last_peered":"2024-03-16T19:29:28.215673+0000","last_clean":"2024-03-16T19:29:28.215673+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:29:28.215673+0000","last_undegraded":"2024-03-16T19:29:28.215673+0000","last_fullsized":"2024-03-16T19:29:28.215673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562176+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:29:27.562176+0000","last_peered":"2024-03-16T19:29:27.562176+0000","last_clean":"2024-03-16T19:29:27.562176+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:29:27.562176+0000","last_undegraded":"2024-03-16T19:29:27.562176+0000","last_fullsized":"2024-03-16T19:29:27.562176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365525+0000","last_change":"2024-03-16T19:29:25.342597+0000","last_active":"2024-03-16T19:29:31.365525+0000","last_peered":"2024-03-16T19:29:31.365525+0000","last_clean":"2024-03-16T19:29:31.365525+0000","last_became_active":"2024-03-16T19:29:25.342320+0000","last_became_peered":"2024-03-16T19:29:25.342320+0000","last_unstale":"2024-03-16T19:29:31.365525+0000","last_undegraded":"2024-03-16T19:29:31.365525+0000","last_fullsized":"2024-03-16T19:29:31.365525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:44:41.274396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:29:41.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:29:41.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.781+0000 7fbf0b047640 1 Processor -- start 2024-03-16T19:29:41.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.781+0000 7fbf0b047640 1 -- start start 2024-03-16T19:29:41.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.782+0000 7fbf0b047640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0415aab0 0x7fbf041626e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.782+0000 7fbf0b047640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf04151610 con 0x7fbf04151230 2024-03-16T19:29:41.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.782+0000 7fbf0b047640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf040c6a30 con 0x7fbf040c9650 2024-03-16T19:29:41.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.782+0000 7fbf0b047640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0406bb30 con 0x7fbf0415aab0 2024-03-16T19:29:41.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.782+0000 7fbf08de3640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fbf04151230 0x7fbf0406bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:55124/0 (socket says 172.21.15.172:55124) 2024-03-16T19:29:41.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.782+0000 7fbf08de3640 1 -- 172.21.15.172:0/3754249776 learned_addr learned my addr 172.21.15.172:0/3754249776 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:41.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.782+0000 7fbf095e4640 1 --2- 172.21.15.172:0/3754249776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0415aab0 0x7fbf041626e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.782+0000 7fbefb7fe640 1 -- 172.21.15.172:0/3754249776 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911672287 0 0) 0x7fbf04151610 con 0x7fbf04151230 2024-03-16T19:29:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.783+0000 7fbefb7fe640 1 -- 172.21.15.172:0/3754249776 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec003520 con 0x7fbf04151230 2024-03-16T19:29:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.783+0000 7fbefb7fe640 1 -- 172.21.15.172:0/3754249776 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821065391 0 0) 0x7fbf040c6a30 con 0x7fbf040c9650 2024-03-16T19:29:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.783+0000 7fbefb7fe640 1 -- 172.21.15.172:0/3754249776 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf04151610 con 0x7fbf040c9650 2024-03-16T19:29:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.783+0000 7fbf095e4640 1 -- 172.21.15.172:0/3754249776 >> v1:172.21.15.174:6789/0 conn(0x7fbf040c9650 legacy=0x7fbf0415a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.783+0000 7fbf095e4640 1 -- 172.21.15.172:0/3754249776 >> v1:172.21.15.172:6789/0 conn(0x7fbf04151230 legacy=0x7fbf0406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.783+0000 7fbf095e4640 1 -- 172.21.15.172:0/3754249776 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf040c2440 con 0x7fbf0415aab0 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.783+0000 7fbf095e4640 1 --2- 172.21.15.172:0/3754249776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0415aab0 0x7fbf041626e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009dc0 tx=0x7fbefc02bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1511eae94263bb43 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbefb7fe640 1 -- 172.21.15.172:0/3754249776 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc02cd50 con 0x7fbf0415aab0 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbefb7fe640 1 -- 172.21.15.172:0/3754249776 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc034440 con 0x7fbf0415aab0 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbefb7fe640 1 -- 172.21.15.172:0/3754249776 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0347d0 con 0x7fbf0415aab0 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbf0b047640 1 -- 172.21.15.172:0/3754249776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0415aab0 msgr2=0x7fbf041626e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbf0b047640 1 --2- 172.21.15.172:0/3754249776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0415aab0 0x7fbf041626e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009dc0 tx=0x7fbefc02bfa0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbf0b047640 1 -- 172.21.15.172:0/3754249776 shutdown_connections 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbf0b047640 1 --2- 172.21.15.172:0/3754249776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0415aab0 0x7fbf041626e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbf0b047640 1 -- 172.21.15.172:0/3754249776 >> 172.21.15.172:0/3754249776 conn(0x7fbf040c8b30 msgr2=0x7fbf040c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbf0b047640 1 -- 172.21.15.172:0/3754249776 shutdown_connections 2024-03-16T19:29:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.784+0000 7fbf0b047640 1 -- 172.21.15.172:0/3754249776 wait complete. 2024-03-16T19:29:41.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.785+0000 7fbf0b047640 1 Processor -- start 2024-03-16T19:29:41.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.785+0000 7fbf0b047640 1 -- start start 2024-03-16T19:29:41.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf0b047640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf040c9650 0x7fbf04144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf0b047640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf04151230 0x7fbf04147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf0b047640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf0415aab0 0x7fbf040c6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf0b047640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0414ad80 con 0x7fbf0415aab0 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf0b047640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040c7a30 con 0x7fbf04151230 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf0b047640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf04148270 con 0x7fbf040c9650 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf095e4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf0415aab0 0x7fbf040c6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf095e4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf0415aab0 0x7fbf040c6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34878/0 (socket says 172.21.15.172:34878) 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf095e4640 1 -- 172.21.15.172:0/3377326130 learned_addr learned my addr 172.21.15.172:0/3377326130 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf08de3640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf040c9650 0x7fbf04144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbefbfff640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf04151230 0x7fbf04147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.786+0000 7fbf095e4640 1 -- 172.21.15.172:0/3377326130 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf040c9650 msgr2=0x7fbf04144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbf095e4640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf040c9650 0x7fbf04144c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbf095e4640 1 -- 172.21.15.172:0/3377326130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf04151230 msgr2=0x7fbf04147950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbf095e4640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf04151230 0x7fbf04147950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbf095e4640 1 -- 172.21.15.172:0/3377326130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc009a70 con 0x7fbf0415aab0 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbf095e4640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf0415aab0 0x7fbf040c6cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbefc037040 tx=0x7fbefc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbef97fa640 1 -- 172.21.15.172:0/3377326130 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc040450 con 0x7fbf0415aab0 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbef97fa640 1 -- 172.21.15.172:0/3377326130 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc0405b0 con 0x7fbf0415aab0 2024-03-16T19:29:41.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf04148550 con 0x7fbf0415aab0 2024-03-16T19:29:41.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbef97fa640 1 -- 172.21.15.172:0/3377326130 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc040960 con 0x7fbf0415aab0 2024-03-16T19:29:41.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.787+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf04148b10 con 0x7fbf0415aab0 2024-03-16T19:29:41.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.788+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0406daf0 con 0x7fbf0415aab0 2024-03-16T19:29:41.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.788+0000 7fbef97fa640 1 -- 172.21.15.172:0/3377326130 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbefc040ac0 con 0x7fbf0415aab0 2024-03-16T19:29:41.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.789+0000 7fbef97fa640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbef006b3b0 0x7fbef006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:41.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.789+0000 7fbf08de3640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbef006b3b0 0x7fbef006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:41.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.789+0000 7fbef97fa640 1 -- 172.21.15.172:0/3377326130 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbefc0ac7d0 con 0x7fbf0415aab0 2024-03-16T19:29:41.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.789+0000 7fbf08de3640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbef006b3b0 0x7fbef006d870 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbf00000f60 tx=0x7fbf00007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:41.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.793+0000 7fbef97fa640 1 -- 172.21.15.172:0/3377326130 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbefc0774e0 con 0x7fbf0415aab0 2024-03-16T19:29:41.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.959+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbf040c2960 con 0x7fbef006b3b0 2024-03-16T19:29:41.946 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.965+0000 7fbef97fa640 1 -- 172.21.15.172:0/3377326130 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7fbf000014f0 con 0x7fbef006b3b0 2024-03-16T19:29:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:41.950 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:29:41.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbef006b3b0 msgr2=0x7fbef006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbef006b3b0 0x7fbef006d870 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbf00000f60 tx=0x7fbf00007d70 comp rx=0 tx=0).stop 2024-03-16T19:29:41.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf0415aab0 msgr2=0x7fbf040c6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:41.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf0415aab0 0x7fbf040c6cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbefc037040 tx=0x7fbefc004a40 comp rx=0 tx=0).stop 2024-03-16T19:29:41.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 shutdown_connections 2024-03-16T19:29:41.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf0415aab0 0x7fbf040c6cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf04151230 0x7fbf04147950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbef006b3b0 0x7fbef006d870 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 --2- 172.21.15.172:0/3377326130 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf040c9650 0x7fbf04144c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:41.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 >> 172.21.15.172:0/3377326130 conn(0x7fbf040c8b30 msgr2=0x7fbf04150420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:41.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 shutdown_connections 2024-03-16T19:29:41.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:41.968+0000 7fbf0b047640 1 -- 172.21.15.172:0/3377326130 wait complete. 2024-03-16T19:29:41.961 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-16T19:29:40.202800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219048,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755672,"statfs":{"total":773094113280,"available":772869808128,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002817"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344501+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:29:27.344501+0000","last_peered":"2024-03-16T19:29:27.344501+0000","last_clean":"2024-03-16T19:29:27.344501+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:29:27.344501+0000","last_undegraded":"2024-03-16T19:29:27.344501+0000","last_fullsized":"2024-03-16T19:29:27.344501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339631+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:29:27.339631+0000","last_peered":"2024-03-16T19:29:27.339631+0000","last_clean":"2024-03-16T19:29:27.339631+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:29:27.339631+0000","last_undegraded":"2024-03-16T19:29:27.339631+0000","last_fullsized":"2024-03-16T19:29:27.339631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062822+0000","last_change":"2024-03-16T19:29:26.347979+0000","last_active":"2024-03-16T19:29:28.062822+0000","last_peered":"2024-03-16T19:29:28.062822+0000","last_clean":"2024-03-16T19:29:28.062822+0000","last_became_active":"2024-03-16T19:29:26.347711+0000","last_became_peered":"2024-03-16T19:29:26.347711+0000","last_unstale":"2024-03-16T19:29:28.062822+0000","last_undegraded":"2024-03-16T19:29:28.062822+0000","last_fullsized":"2024-03-16T19:29:28.062822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:08.017182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343558+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:29:27.343558+0000","last_peered":"2024-03-16T19:29:27.343558+0000","last_clean":"2024-03-16T19:29:27.343558+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:29:27.343558+0000","last_undegraded":"2024-03-16T19:29:27.343558+0000","last_fullsized":"2024-03-16T19:29:27.343558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344270+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:29:27.344270+0000","last_peered":"2024-03-16T19:29:27.344270+0000","last_clean":"2024-03-16T19:29:27.344270+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:29:27.344270+0000","last_undegraded":"2024-03-16T19:29:27.344270+0000","last_fullsized":"2024-03-16T19:29:27.344270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343104+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:29:27.343104+0000","last_peered":"2024-03-16T19:29:27.343104+0000","last_clean":"2024-03-16T19:29:27.343104+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:29:27.343104+0000","last_undegraded":"2024-03-16T19:29:27.343104+0000","last_fullsized":"2024-03-16T19:29:27.343104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344269+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:29:27.344269+0000","last_peered":"2024-03-16T19:29:27.344269+0000","last_clean":"2024-03-16T19:29:27.344269+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:29:27.344269+0000","last_undegraded":"2024-03-16T19:29:27.344269+0000","last_fullsized":"2024-03-16T19:29:27.344269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561737+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:29:27.561737+0000","last_peered":"2024-03-16T19:29:27.561737+0000","last_clean":"2024-03-16T19:29:27.561737+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:29:27.561737+0000","last_undegraded":"2024-03-16T19:29:27.561737+0000","last_fullsized":"2024-03-16T19:29:27.561737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344497+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:29:27.344497+0000","last_peered":"2024-03-16T19:29:27.344497+0000","last_clean":"2024-03-16T19:29:27.344497+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:29:27.344497+0000","last_undegraded":"2024-03-16T19:29:27.344497+0000","last_fullsized":"2024-03-16T19:29:27.344497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063286+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:29:28.063286+0000","last_peered":"2024-03-16T19:29:28.063286+0000","last_clean":"2024-03-16T19:29:28.063286+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:29:28.063286+0000","last_undegraded":"2024-03-16T19:29:28.063286+0000","last_fullsized":"2024-03-16T19:29:28.063286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062996+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:29:28.062996+0000","last_peered":"2024-03-16T19:29:28.062996+0000","last_clean":"2024-03-16T19:29:28.062996+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:29:28.062996+0000","last_undegraded":"2024-03-16T19:29:28.062996+0000","last_fullsized":"2024-03-16T19:29:28.062996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063177+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:29:28.063177+0000","last_peered":"2024-03-16T19:29:28.063177+0000","last_clean":"2024-03-16T19:29:28.063177+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:29:28.063177+0000","last_undegraded":"2024-03-16T19:29:28.063177+0000","last_fullsized":"2024-03-16T19:29:28.063177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561930+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:29:27.561930+0000","last_peered":"2024-03-16T19:29:27.561930+0000","last_clean":"2024-03-16T19:29:27.561930+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:29:27.561930+0000","last_undegraded":"2024-03-16T19:29:27.561930+0000","last_fullsized":"2024-03-16T19:29:27.561930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062462+0000","last_change":"2024-03-16T19:29:26.349644+0000","last_active":"2024-03-16T19:29:28.062462+0000","last_peered":"2024-03-16T19:29:28.062462+0000","last_clean":"2024-03-16T19:29:28.062462+0000","last_became_active":"2024-03-16T19:29:26.349397+0000","last_became_peered":"2024-03-16T19:29:26.349397+0000","last_unstale":"2024-03-16T19:29:28.062462+0000","last_undegraded":"2024-03-16T19:29:28.062462+0000","last_fullsized":"2024-03-16T19:29:28.062462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:47:20.566393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339397+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:29:27.339397+0000","last_peered":"2024-03-16T19:29:27.339397+0000","last_clean":"2024-03-16T19:29:27.339397+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:29:27.339397+0000","last_undegraded":"2024-03-16T19:29:27.339397+0000","last_fullsized":"2024-03-16T19:29:27.339397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344311+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:29:27.344311+0000","last_peered":"2024-03-16T19:29:27.344311+0000","last_clean":"2024-03-16T19:29:27.344311+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:29:27.344311+0000","last_undegraded":"2024-03-16T19:29:27.344311+0000","last_fullsized":"2024-03-16T19:29:27.344311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062569+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:29:28.062569+0000","last_peered":"2024-03-16T19:29:28.062569+0000","last_clean":"2024-03-16T19:29:28.062569+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:29:28.062569+0000","last_undegraded":"2024-03-16T19:29:28.062569+0000","last_fullsized":"2024-03-16T19:29:28.062569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344648+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:29:27.344648+0000","last_peered":"2024-03-16T19:29:27.344648+0000","last_clean":"2024-03-16T19:29:27.344648+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:29:27.344648+0000","last_undegraded":"2024-03-16T19:29:27.344648+0000","last_fullsized":"2024-03-16T19:29:27.344648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063112+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:29:28.063112+0000","last_peered":"2024-03-16T19:29:28.063112+0000","last_clean":"2024-03-16T19:29:28.063112+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:29:28.063112+0000","last_undegraded":"2024-03-16T19:29:28.063112+0000","last_fullsized":"2024-03-16T19:29:28.063112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062498+0000","last_change":"2024-03-16T19:29:26.357191+0000","last_active":"2024-03-16T19:29:28.062498+0000","last_peered":"2024-03-16T19:29:28.062498+0000","last_clean":"2024-03-16T19:29:28.062498+0000","last_became_active":"2024-03-16T19:29:26.356947+0000","last_became_peered":"2024-03-16T19:29:26.356947+0000","last_unstale":"2024-03-16T19:29:28.062498+0000","last_undegraded":"2024-03-16T19:29:28.062498+0000","last_fullsized":"2024-03-16T19:29:28.062498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:40.884164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343265+0000","last_change":"2024-03-16T19:29:26.358307+0000","last_active":"2024-03-16T19:29:27.343265+0000","last_peered":"2024-03-16T19:29:27.343265+0000","last_clean":"2024-03-16T19:29:27.343265+0000","last_became_active":"2024-03-16T19:29:26.357970+0000","last_became_peered":"2024-03-16T19:29:26.357970+0000","last_unstale":"2024-03-16T19:29:27.343265+0000","last_undegraded":"2024-03-16T19:29:27.343265+0000","last_fullsized":"2024-03-16T19:29:27.343265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:33.976868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561749+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:29:27.561749+0000","last_peered":"2024-03-16T19:29:27.561749+0000","last_clean":"2024-03-16T19:29:27.561749+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:29:27.561749+0000","last_undegraded":"2024-03-16T19:29:27.561749+0000","last_fullsized":"2024-03-16T19:29:27.561749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343286+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:29:27.343286+0000","last_peered":"2024-03-16T19:29:27.343286+0000","last_clean":"2024-03-16T19:29:27.343286+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:29:27.343286+0000","last_undegraded":"2024-03-16T19:29:27.343286+0000","last_fullsized":"2024-03-16T19:29:27.343286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562192+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:29:27.562192+0000","last_peered":"2024-03-16T19:29:27.562192+0000","last_clean":"2024-03-16T19:29:27.562192+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:29:27.562192+0000","last_undegraded":"2024-03-16T19:29:27.562192+0000","last_fullsized":"2024-03-16T19:29:27.562192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063443+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:29:28.063443+0000","last_peered":"2024-03-16T19:29:28.063443+0000","last_clean":"2024-03-16T19:29:28.063443+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:29:28.063443+0000","last_undegraded":"2024-03-16T19:29:28.063443+0000","last_fullsized":"2024-03-16T19:29:28.063443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339564+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:29:27.339564+0000","last_peered":"2024-03-16T19:29:27.339564+0000","last_clean":"2024-03-16T19:29:27.339564+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:29:27.339564+0000","last_undegraded":"2024-03-16T19:29:27.339564+0000","last_fullsized":"2024-03-16T19:29:27.339564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562214+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:29:27.562214+0000","last_peered":"2024-03-16T19:29:27.562214+0000","last_clean":"2024-03-16T19:29:27.562214+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:29:27.562214+0000","last_undegraded":"2024-03-16T19:29:27.562214+0000","last_fullsized":"2024-03-16T19:29:27.562214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062637+0000","last_change":"2024-03-16T19:29:26.349443+0000","last_active":"2024-03-16T19:29:28.062637+0000","last_peered":"2024-03-16T19:29:28.062637+0000","last_clean":"2024-03-16T19:29:28.062637+0000","last_became_active":"2024-03-16T19:29:26.349181+0000","last_became_peered":"2024-03-16T19:29:26.349181+0000","last_unstale":"2024-03-16T19:29:28.062637+0000","last_undegraded":"2024-03-16T19:29:28.062637+0000","last_fullsized":"2024-03-16T19:29:28.062637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:33:16.483346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343433+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:29:27.343433+0000","last_peered":"2024-03-16T19:29:27.343433+0000","last_clean":"2024-03-16T19:29:27.343433+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:29:27.343433+0000","last_undegraded":"2024-03-16T19:29:27.343433+0000","last_fullsized":"2024-03-16T19:29:27.343433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062168+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:29:28.062168+0000","last_peered":"2024-03-16T19:29:28.062168+0000","last_clean":"2024-03-16T19:29:28.062168+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:29:28.062168+0000","last_undegraded":"2024-03-16T19:29:28.062168+0000","last_fullsized":"2024-03-16T19:29:28.062168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062728+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:29:28.062728+0000","last_peered":"2024-03-16T19:29:28.062728+0000","last_clean":"2024-03-16T19:29:28.062728+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:29:28.062728+0000","last_undegraded":"2024-03-16T19:29:28.062728+0000","last_fullsized":"2024-03-16T19:29:28.062728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063484+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:29:28.063484+0000","last_peered":"2024-03-16T19:29:28.063484+0000","last_clean":"2024-03-16T19:29:28.063484+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:29:28.063484+0000","last_undegraded":"2024-03-16T19:29:28.063484+0000","last_fullsized":"2024-03-16T19:29:28.063484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.215645+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:29:28.215645+0000","last_peered":"2024-03-16T19:29:28.215645+0000","last_clean":"2024-03-16T19:29:28.215645+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:29:28.215645+0000","last_undegraded":"2024-03-16T19:29:28.215645+0000","last_fullsized":"2024-03-16T19:29:28.215645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062765+0000","last_change":"2024-03-16T19:29:26.353288+0000","last_active":"2024-03-16T19:29:28.062765+0000","last_peered":"2024-03-16T19:29:28.062765+0000","last_clean":"2024-03-16T19:29:28.062765+0000","last_became_active":"2024-03-16T19:29:26.352973+0000","last_became_peered":"2024-03-16T19:29:26.352973+0000","last_unstale":"2024-03-16T19:29:28.062765+0000","last_undegraded":"2024-03-16T19:29:28.062765+0000","last_fullsized":"2024-03-16T19:29:28.062765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:22:41.505003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.210595+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:29:28.210595+0000","last_peered":"2024-03-16T19:29:28.210595+0000","last_clean":"2024-03-16T19:29:28.210595+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:29:28.210595+0000","last_undegraded":"2024-03-16T19:29:28.210595+0000","last_fullsized":"2024-03-16T19:29:28.210595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561856+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:29:27.561856+0000","last_peered":"2024-03-16T19:29:27.561856+0000","last_clean":"2024-03-16T19:29:27.561856+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:29:27.561856+0000","last_undegraded":"2024-03-16T19:29:27.561856+0000","last_fullsized":"2024-03-16T19:29:27.561856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561778+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:29:27.561778+0000","last_peered":"2024-03-16T19:29:27.561778+0000","last_clean":"2024-03-16T19:29:27.561778+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:29:27.561778+0000","last_undegraded":"2024-03-16T19:29:27.561778+0000","last_fullsized":"2024-03-16T19:29:27.561778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343316+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:29:27.343316+0000","last_peered":"2024-03-16T19:29:27.343316+0000","last_clean":"2024-03-16T19:29:27.343316+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:29:27.343316+0000","last_undegraded":"2024-03-16T19:29:27.343316+0000","last_fullsized":"2024-03-16T19:29:27.343316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339401+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:29:27.339401+0000","last_peered":"2024-03-16T19:29:27.339401+0000","last_clean":"2024-03-16T19:29:27.339401+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:29:27.339401+0000","last_undegraded":"2024-03-16T19:29:27.339401+0000","last_fullsized":"2024-03-16T19:29:27.339401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063312+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:29:28.063312+0000","last_peered":"2024-03-16T19:29:28.063312+0000","last_clean":"2024-03-16T19:29:28.063312+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:29:28.063312+0000","last_undegraded":"2024-03-16T19:29:28.063312+0000","last_fullsized":"2024-03-16T19:29:28.063312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344324+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:29:27.344324+0000","last_peered":"2024-03-16T19:29:27.344324+0000","last_clean":"2024-03-16T19:29:27.344324+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:29:27.344324+0000","last_undegraded":"2024-03-16T19:29:27.344324+0000","last_fullsized":"2024-03-16T19:29:27.344324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343744+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:29:27.343744+0000","last_peered":"2024-03-16T19:29:27.343744+0000","last_clean":"2024-03-16T19:29:27.343744+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:29:27.343744+0000","last_undegraded":"2024-03-16T19:29:27.343744+0000","last_fullsized":"2024-03-16T19:29:27.343744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062922+0000","last_change":"2024-03-16T19:29:26.353918+0000","last_active":"2024-03-16T19:29:28.062922+0000","last_peered":"2024-03-16T19:29:28.062922+0000","last_clean":"2024-03-16T19:29:28.062922+0000","last_became_active":"2024-03-16T19:29:26.353463+0000","last_became_peered":"2024-03-16T19:29:26.353463+0000","last_unstale":"2024-03-16T19:29:28.062922+0000","last_undegraded":"2024-03-16T19:29:28.062922+0000","last_fullsized":"2024-03-16T19:29:28.062922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:36.035692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.362815+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:29:31.362815+0000","last_peered":"2024-03-16T19:29:31.362815+0000","last_clean":"2024-03-16T19:29:31.362815+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:29:31.362815+0000","last_undegraded":"2024-03-16T19:29:31.362815+0000","last_fullsized":"2024-03-16T19:29:31.362815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063360+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:29:28.063360+0000","last_peered":"2024-03-16T19:29:28.063360+0000","last_clean":"2024-03-16T19:29:28.063360+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:29:28.063360+0000","last_undegraded":"2024-03-16T19:29:28.063360+0000","last_fullsized":"2024-03-16T19:29:28.063360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365125+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:29:31.365125+0000","last_peered":"2024-03-16T19:29:31.365125+0000","last_clean":"2024-03-16T19:29:31.365125+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:29:31.365125+0000","last_undegraded":"2024-03-16T19:29:31.365125+0000","last_fullsized":"2024-03-16T19:29:31.365125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344465+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:29:27.344465+0000","last_peered":"2024-03-16T19:29:27.344465+0000","last_clean":"2024-03-16T19:29:27.344465+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:29:27.344465+0000","last_undegraded":"2024-03-16T19:29:27.344465+0000","last_fullsized":"2024-03-16T19:29:27.344465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365001+0000","last_change":"2024-03-16T19:29:25.336296+0000","last_active":"2024-03-16T19:29:31.365001+0000","last_peered":"2024-03-16T19:29:31.365001+0000","last_clean":"2024-03-16T19:29:31.365001+0000","last_became_active":"2024-03-16T19:29:25.333046+0000","last_became_peered":"2024-03-16T19:29:25.333046+0000","last_unstale":"2024-03-16T19:29:31.365001+0000","last_undegraded":"2024-03-16T19:29:31.365001+0000","last_fullsized":"2024-03-16T19:29:31.365001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:33.538342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062507+0000","last_change":"2024-03-16T19:29:26.354709+0000","last_active":"2024-03-16T19:29:28.062507+0000","last_peered":"2024-03-16T19:29:28.062507+0000","last_clean":"2024-03-16T19:29:28.062507+0000","last_became_active":"2024-03-16T19:29:26.354377+0000","last_became_peered":"2024-03-16T19:29:26.354377+0000","last_unstale":"2024-03-16T19:29:28.062507+0000","last_undegraded":"2024-03-16T19:29:28.062507+0000","last_fullsized":"2024-03-16T19:29:28.062507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:41:56.277270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343553+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:29:27.343553+0000","last_peered":"2024-03-16T19:29:27.343553+0000","last_clean":"2024-03-16T19:29:27.343553+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:29:27.343553+0000","last_undegraded":"2024-03-16T19:29:27.343553+0000","last_fullsized":"2024-03-16T19:29:27.343553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344364+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:29:27.344364+0000","last_peered":"2024-03-16T19:29:27.344364+0000","last_clean":"2024-03-16T19:29:27.344364+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:29:27.344364+0000","last_undegraded":"2024-03-16T19:29:27.344364+0000","last_fullsized":"2024-03-16T19:29:27.344364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062737+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:29:28.062737+0000","last_peered":"2024-03-16T19:29:28.062737+0000","last_clean":"2024-03-16T19:29:28.062737+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:29:28.062737+0000","last_undegraded":"2024-03-16T19:29:28.062737+0000","last_fullsized":"2024-03-16T19:29:28.062737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062954+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:29:28.062954+0000","last_peered":"2024-03-16T19:29:28.062954+0000","last_clean":"2024-03-16T19:29:28.062954+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:29:28.062954+0000","last_undegraded":"2024-03-16T19:29:28.062954+0000","last_fullsized":"2024-03-16T19:29:28.062954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365607+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:29:31.365607+0000","last_peered":"2024-03-16T19:29:31.365607+0000","last_clean":"2024-03-16T19:29:31.365607+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:29:31.365607+0000","last_undegraded":"2024-03-16T19:29:31.365607+0000","last_fullsized":"2024-03-16T19:29:31.365607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344168+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:29:27.344168+0000","last_peered":"2024-03-16T19:29:27.344168+0000","last_clean":"2024-03-16T19:29:27.344168+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:29:27.344168+0000","last_undegraded":"2024-03-16T19:29:27.344168+0000","last_fullsized":"2024-03-16T19:29:27.344168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.211252+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:29:28.211252+0000","last_peered":"2024-03-16T19:29:28.211252+0000","last_clean":"2024-03-16T19:29:28.211252+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:29:28.211252+0000","last_undegraded":"2024-03-16T19:29:28.211252+0000","last_fullsized":"2024-03-16T19:29:28.211252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344198+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:29:27.344198+0000","last_peered":"2024-03-16T19:29:27.344198+0000","last_clean":"2024-03-16T19:29:27.344198+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:29:27.344198+0000","last_undegraded":"2024-03-16T19:29:27.344198+0000","last_fullsized":"2024-03-16T19:29:27.344198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.362435+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:29:31.362435+0000","last_peered":"2024-03-16T19:29:31.362435+0000","last_clean":"2024-03-16T19:29:31.362435+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:29:31.362435+0000","last_undegraded":"2024-03-16T19:29:31.362435+0000","last_fullsized":"2024-03-16T19:29:31.362435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344664+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:29:27.344664+0000","last_peered":"2024-03-16T19:29:27.344664+0000","last_clean":"2024-03-16T19:29:27.344664+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:29:27.344664+0000","last_undegraded":"2024-03-16T19:29:27.344664+0000","last_fullsized":"2024-03-16T19:29:27.344664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.219180+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:29:28.219180+0000","last_peered":"2024-03-16T19:29:28.219180+0000","last_clean":"2024-03-16T19:29:28.219180+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:29:28.219180+0000","last_undegraded":"2024-03-16T19:29:28.219180+0000","last_fullsized":"2024-03-16T19:29:28.219180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344128+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:29:27.344128+0000","last_peered":"2024-03-16T19:29:27.344128+0000","last_clean":"2024-03-16T19:29:27.344128+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:29:27.344128+0000","last_undegraded":"2024-03-16T19:29:27.344128+0000","last_fullsized":"2024-03-16T19:29:27.344128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062516+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:29:28.062516+0000","last_peered":"2024-03-16T19:29:28.062516+0000","last_clean":"2024-03-16T19:29:28.062516+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:29:28.062516+0000","last_undegraded":"2024-03-16T19:29:28.062516+0000","last_fullsized":"2024-03-16T19:29:28.062516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344351+0000","last_change":"2024-03-16T19:29:26.364742+0000","last_active":"2024-03-16T19:29:27.344351+0000","last_peered":"2024-03-16T19:29:27.344351+0000","last_clean":"2024-03-16T19:29:27.344351+0000","last_became_active":"2024-03-16T19:29:26.364328+0000","last_became_peered":"2024-03-16T19:29:26.364328+0000","last_unstale":"2024-03-16T19:29:27.344351+0000","last_undegraded":"2024-03-16T19:29:27.344351+0000","last_fullsized":"2024-03-16T19:29:27.344351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:18.877774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343098+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:29:27.343098+0000","last_peered":"2024-03-16T19:29:27.343098+0000","last_clean":"2024-03-16T19:29:27.343098+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:29:27.343098+0000","last_undegraded":"2024-03-16T19:29:27.343098+0000","last_fullsized":"2024-03-16T19:29:27.343098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343226+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:29:27.343226+0000","last_peered":"2024-03-16T19:29:27.343226+0000","last_clean":"2024-03-16T19:29:27.343226+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:29:27.343226+0000","last_undegraded":"2024-03-16T19:29:27.343226+0000","last_fullsized":"2024-03-16T19:29:27.343226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561997+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:29:27.561997+0000","last_peered":"2024-03-16T19:29:27.561997+0000","last_clean":"2024-03-16T19:29:27.561997+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:29:27.561997+0000","last_undegraded":"2024-03-16T19:29:27.561997+0000","last_fullsized":"2024-03-16T19:29:27.561997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343035+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:29:27.343035+0000","last_peered":"2024-03-16T19:29:27.343035+0000","last_clean":"2024-03-16T19:29:27.343035+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:29:27.343035+0000","last_undegraded":"2024-03-16T19:29:27.343035+0000","last_fullsized":"2024-03-16T19:29:27.343035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062305+0000","last_change":"2024-03-16T19:29:26.352692+0000","last_active":"2024-03-16T19:29:28.062305+0000","last_peered":"2024-03-16T19:29:28.062305+0000","last_clean":"2024-03-16T19:29:28.062305+0000","last_became_active":"2024-03-16T19:29:26.352419+0000","last_became_peered":"2024-03-16T19:29:26.352419+0000","last_unstale":"2024-03-16T19:29:28.062305+0000","last_undegraded":"2024-03-16T19:29:28.062305+0000","last_fullsized":"2024-03-16T19:29:28.062305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:33.056538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062749+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:29:28.062749+0000","last_peered":"2024-03-16T19:29:28.062749+0000","last_clean":"2024-03-16T19:29:28.062749+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:29:28.062749+0000","last_undegraded":"2024-03-16T19:29:28.062749+0000","last_fullsized":"2024-03-16T19:29:28.062749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339393+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:29:27.339393+0000","last_peered":"2024-03-16T19:29:27.339393+0000","last_clean":"2024-03-16T19:29:27.339393+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:29:27.339393+0000","last_undegraded":"2024-03-16T19:29:27.339393+0000","last_fullsized":"2024-03-16T19:29:27.339393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344073+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:29:27.344073+0000","last_peered":"2024-03-16T19:29:27.344073+0000","last_clean":"2024-03-16T19:29:27.344073+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:29:27.344073+0000","last_undegraded":"2024-03-16T19:29:27.344073+0000","last_fullsized":"2024-03-16T19:29:27.344073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063637+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:29:28.063637+0000","last_peered":"2024-03-16T19:29:28.063637+0000","last_clean":"2024-03-16T19:29:28.063637+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:29:28.063637+0000","last_undegraded":"2024-03-16T19:29:28.063637+0000","last_fullsized":"2024-03-16T19:29:28.063637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339499+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:29:27.339499+0000","last_peered":"2024-03-16T19:29:27.339499+0000","last_clean":"2024-03-16T19:29:27.339499+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:29:27.339499+0000","last_undegraded":"2024-03-16T19:29:27.339499+0000","last_fullsized":"2024-03-16T19:29:27.339499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343975+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:29:27.343975+0000","last_peered":"2024-03-16T19:29:27.343975+0000","last_clean":"2024-03-16T19:29:27.343975+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:29:27.343975+0000","last_undegraded":"2024-03-16T19:29:27.343975+0000","last_fullsized":"2024-03-16T19:29:27.343975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344573+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:29:27.344573+0000","last_peered":"2024-03-16T19:29:27.344573+0000","last_clean":"2024-03-16T19:29:27.344573+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:29:27.344573+0000","last_undegraded":"2024-03-16T19:29:27.344573+0000","last_fullsized":"2024-03-16T19:29:27.344573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339540+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:29:27.339540+0000","last_peered":"2024-03-16T19:29:27.339540+0000","last_clean":"2024-03-16T19:29:27.339540+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:29:27.339540+0000","last_undegraded":"2024-03-16T19:29:27.339540+0000","last_fullsized":"2024-03-16T19:29:27.339540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.219205+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:29:28.219205+0000","last_peered":"2024-03-16T19:29:28.219205+0000","last_clean":"2024-03-16T19:29:28.219205+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:29:28.219205+0000","last_undegraded":"2024-03-16T19:29:28.219205+0000","last_fullsized":"2024-03-16T19:29:28.219205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562225+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:29:27.562225+0000","last_peered":"2024-03-16T19:29:27.562225+0000","last_clean":"2024-03-16T19:29:27.562225+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:29:27.562225+0000","last_undegraded":"2024-03-16T19:29:27.562225+0000","last_fullsized":"2024-03-16T19:29:27.562225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.360976+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:29:31.360976+0000","last_peered":"2024-03-16T19:29:31.360976+0000","last_clean":"2024-03-16T19:29:31.360976+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:29:31.360976+0000","last_undegraded":"2024-03-16T19:29:31.360976+0000","last_fullsized":"2024-03-16T19:29:31.360976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062725+0000","last_change":"2024-03-16T19:29:26.350351+0000","last_active":"2024-03-16T19:29:28.062725+0000","last_peered":"2024-03-16T19:29:28.062725+0000","last_clean":"2024-03-16T19:29:28.062725+0000","last_became_active":"2024-03-16T19:29:26.350074+0000","last_became_peered":"2024-03-16T19:29:26.350074+0000","last_unstale":"2024-03-16T19:29:28.062725+0000","last_undegraded":"2024-03-16T19:29:28.062725+0000","last_fullsized":"2024-03-16T19:29:28.062725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:49.416395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.342931+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:29:27.342931+0000","last_peered":"2024-03-16T19:29:27.342931+0000","last_clean":"2024-03-16T19:29:27.342931+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:29:27.342931+0000","last_undegraded":"2024-03-16T19:29:27.342931+0000","last_fullsized":"2024-03-16T19:29:27.342931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.063379+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:29:28.063379+0000","last_peered":"2024-03-16T19:29:28.063379+0000","last_clean":"2024-03-16T19:29:28.063379+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:29:28.063379+0000","last_undegraded":"2024-03-16T19:29:28.063379+0000","last_fullsized":"2024-03-16T19:29:28.063379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339690+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:29:27.339690+0000","last_peered":"2024-03-16T19:29:27.339690+0000","last_clean":"2024-03-16T19:29:27.339690+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:29:27.339690+0000","last_undegraded":"2024-03-16T19:29:27.339690+0000","last_fullsized":"2024-03-16T19:29:27.339690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561913+0000","last_change":"2024-03-16T19:29:26.347388+0000","last_active":"2024-03-16T19:29:27.561913+0000","last_peered":"2024-03-16T19:29:27.561913+0000","last_clean":"2024-03-16T19:29:27.561913+0000","last_became_active":"2024-03-16T19:29:26.346582+0000","last_became_peered":"2024-03-16T19:29:26.346582+0000","last_unstale":"2024-03-16T19:29:27.561913+0000","last_undegraded":"2024-03-16T19:29:27.561913+0000","last_fullsized":"2024-03-16T19:29:27.561913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:05.294895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.339497+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:29:27.339497+0000","last_peered":"2024-03-16T19:29:27.339497+0000","last_clean":"2024-03-16T19:29:27.339497+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:29:27.339497+0000","last_undegraded":"2024-03-16T19:29:27.339497+0000","last_fullsized":"2024-03-16T19:29:27.339497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.561968+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:29:27.561968+0000","last_peered":"2024-03-16T19:29:27.561968+0000","last_clean":"2024-03-16T19:29:27.561968+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:29:27.561968+0000","last_undegraded":"2024-03-16T19:29:27.561968+0000","last_fullsized":"2024-03-16T19:29:27.561968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365383+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:29:31.365383+0000","last_peered":"2024-03-16T19:29:31.365383+0000","last_clean":"2024-03-16T19:29:31.365383+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:29:31.365383+0000","last_undegraded":"2024-03-16T19:29:31.365383+0000","last_fullsized":"2024-03-16T19:29:31.365383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062837+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:29:28.062837+0000","last_peered":"2024-03-16T19:29:28.062837+0000","last_clean":"2024-03-16T19:29:28.062837+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:29:28.062837+0000","last_undegraded":"2024-03-16T19:29:28.062837+0000","last_fullsized":"2024-03-16T19:29:28.062837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.343809+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:29:27.343809+0000","last_peered":"2024-03-16T19:29:27.343809+0000","last_clean":"2024-03-16T19:29:27.343809+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:29:27.343809+0000","last_undegraded":"2024-03-16T19:29:27.343809+0000","last_fullsized":"2024-03-16T19:29:27.343809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.360603+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:29:31.360603+0000","last_peered":"2024-03-16T19:29:31.360603+0000","last_clean":"2024-03-16T19:29:31.360603+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:29:31.360603+0000","last_undegraded":"2024-03-16T19:29:31.360603+0000","last_fullsized":"2024-03-16T19:29:31.360603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.344208+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:29:27.344208+0000","last_peered":"2024-03-16T19:29:27.344208+0000","last_clean":"2024-03-16T19:29:27.344208+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:29:27.344208+0000","last_undegraded":"2024-03-16T19:29:27.344208+0000","last_fullsized":"2024-03-16T19:29:27.344208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365202+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:29:31.365202+0000","last_peered":"2024-03-16T19:29:31.365202+0000","last_clean":"2024-03-16T19:29:31.365202+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:29:31.365202+0000","last_undegraded":"2024-03-16T19:29:31.365202+0000","last_fullsized":"2024-03-16T19:29:31.365202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.062891+0000","last_change":"2024-03-16T19:29:26.354303+0000","last_active":"2024-03-16T19:29:28.062891+0000","last_peered":"2024-03-16T19:29:28.062891+0000","last_clean":"2024-03-16T19:29:28.062891+0000","last_became_active":"2024-03-16T19:29:26.353804+0000","last_became_peered":"2024-03-16T19:29:26.353804+0000","last_unstale":"2024-03-16T19:29:28.062891+0000","last_undegraded":"2024-03-16T19:29:28.062891+0000","last_fullsized":"2024-03-16T19:29:28.062891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:10:45.368536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:28.215673+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:29:28.215673+0000","last_peered":"2024-03-16T19:29:28.215673+0000","last_clean":"2024-03-16T19:29:28.215673+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:29:28.215673+0000","last_undegraded":"2024-03-16T19:29:28.215673+0000","last_fullsized":"2024-03-16T19:29:28.215673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:27.562176+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:29:27.562176+0000","last_peered":"2024-03-16T19:29:27.562176+0000","last_clean":"2024-03-16T19:29:27.562176+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:29:27.562176+0000","last_undegraded":"2024-03-16T19:29:27.562176+0000","last_fullsized":"2024-03-16T19:29:27.562176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-16T19:29:31.365525+0000","last_change":"2024-03-16T19:29:25.342597+0000","last_active":"2024-03-16T19:29:31.365525+0000","last_peered":"2024-03-16T19:29:31.365525+0000","last_clean":"2024-03-16T19:29:31.365525+0000","last_became_active":"2024-03-16T19:29:25.342320+0000","last_became_peered":"2024-03-16T19:29:25.342320+0000","last_unstale":"2024-03-16T19:29:31.365525+0000","last_undegraded":"2024-03-16T19:29:31.365525+0000","last_fullsized":"2024-03-16T19:29:31.365525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:44:41.274396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:29:41.964 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-16T19:29:41.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-03-16T19:29:42.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.120+0000 7f903ad3a640 1 Processor -- start 2024-03-16T19:29:42.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.120+0000 7f903ad3a640 1 -- start start 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.121+0000 7f903ad3a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9034152840 0x7f903406e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.121+0000 7f903ad3a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f90340cb2b0 0x7f9034162fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.121+0000 7f903ad3a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90341590a0 con 0x7f90340c3610 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.121+0000 7f903ad3a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90340c4260 con 0x7f90340cb2b0 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.121+0000 7f903ad3a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90340c31c0 con 0x7f9034152840 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.121+0000 7f9033fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f90340c3610 0x7f90340caba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:55140/0 (socket says 172.21.15.172:55140) 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.121+0000 7f9033fff640 1 -- 172.21.15.172:0/797808739 learned_addr learned my addr 172.21.15.172:0/797808739 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.121+0000 7f9038ad6640 1 --2- 172.21.15.172:0/797808739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9034152840 0x7f903406e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.121+0000 7f90392d7640 1 --2- 172.21.15.172:0/797808739 >> v2:172.21.15.174:3300/0 conn(0x7f90340cb2b0 0x7f9034162fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.122+0000 7f9038ad6640 1 -- 172.21.15.172:0/797808739 >> v2:172.21.15.174:3300/0 conn(0x7f90340cb2b0 msgr2=0x7f9034162fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.122+0000 7f9038ad6640 1 --2- 172.21.15.172:0/797808739 >> v2:172.21.15.174:3300/0 conn(0x7f90340cb2b0 0x7f9034162fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.122+0000 7f9038ad6640 1 -- 172.21.15.172:0/797808739 >> v1:172.21.15.172:6789/0 conn(0x7f90340c3610 legacy=0x7f90340caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.122+0000 7f90337fe640 1 -- 172.21.15.172:0/797808739 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805329013 0 0) 0x7f90341590a0 con 0x7f90340c3610 2024-03-16T19:29:42.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.122+0000 7f9038ad6640 1 -- 172.21.15.172:0/797808739 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90341595d0 con 0x7f9034152840 2024-03-16T19:29:42.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.122+0000 7f9038ad6640 1 --2- 172.21.15.172:0/797808739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9034152840 0x7f903406e330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f90240097a0 tx=0x7f902402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d27002f7a5e8ea63 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.122+0000 7f90337fe640 1 -- 172.21.15.172:0/797808739 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9024031830 con 0x7f9034152840 2024-03-16T19:29:42.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f90337fe640 1 -- 172.21.15.172:0/797808739 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9024031990 con 0x7f9034152840 2024-03-16T19:29:42.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f90337fe640 1 -- 172.21.15.172:0/797808739 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9024031d20 con 0x7f9034152840 2024-03-16T19:29:42.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f903ad3a640 1 -- 172.21.15.172:0/797808739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9034152840 msgr2=0x7f903406e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f903ad3a640 1 --2- 172.21.15.172:0/797808739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9034152840 0x7f903406e330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f90240097a0 tx=0x7f902402b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f903ad3a640 1 -- 172.21.15.172:0/797808739 shutdown_connections 2024-03-16T19:29:42.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f903ad3a640 1 --2- 172.21.15.172:0/797808739 >> v2:172.21.15.174:3300/0 conn(0x7f90340cb2b0 0x7f9034162fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f903ad3a640 1 --2- 172.21.15.172:0/797808739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9034152840 0x7f903406e330 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f903ad3a640 1 -- 172.21.15.172:0/797808739 >> 172.21.15.172:0/797808739 conn(0x7f9034070860 msgr2=0x7f9034070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:42.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f903ad3a640 1 -- 172.21.15.172:0/797808739 shutdown_connections 2024-03-16T19:29:42.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.123+0000 7f903ad3a640 1 -- 172.21.15.172:0/797808739 wait complete. 2024-03-16T19:29:42.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.124+0000 7f903ad3a640 1 Processor -- start 2024-03-16T19:29:42.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.124+0000 7f903ad3a640 1 -- start start 2024-03-16T19:29:42.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f903ad3a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90340c3610 0x7f90341ee150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f903ad3a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90340cb2b0 0x7f90340c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f903ad3a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9034152840 0x7f9034156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f903ad3a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90340c85a0 con 0x7f90340c3610 2024-03-16T19:29:42.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f903ad3a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90340c7bd0 con 0x7f9034152840 2024-03-16T19:29:42.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f903ad3a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903414ed60 con 0x7f90340cb2b0 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f9038ad6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90340c3610 0x7f90341ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f9038ad6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90340c3610 0x7f90341ee150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34886/0 (socket says 172.21.15.172:34886) 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f9038ad6640 1 -- 172.21.15.172:0/986208215 learned_addr learned my addr 172.21.15.172:0/986208215 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f9033fff640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90340cb2b0 0x7f90340c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f90392d7640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9034152840 0x7f9034156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f9038ad6640 1 -- 172.21.15.172:0/986208215 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90340cb2b0 msgr2=0x7f90340c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f9038ad6640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90340cb2b0 0x7f90340c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f9038ad6640 1 -- 172.21.15.172:0/986208215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9034152840 msgr2=0x7f9034156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f9038ad6640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9034152840 0x7f9034156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.125+0000 7f9038ad6640 1 -- 172.21.15.172:0/986208215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9024009400 con 0x7f90340c3610 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.126+0000 7f9038ad6640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90340c3610 0x7f90341ee150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9024000c00 tx=0x7f90240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.126+0000 7f90317fa640 1 -- 172.21.15.172:0/986208215 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9024004230 con 0x7f90340c3610 2024-03-16T19:29:42.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.126+0000 7f90317fa640 1 -- 172.21.15.172:0/986208215 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90240043b0 con 0x7f90340c3610 2024-03-16T19:29:42.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.126+0000 7f90317fa640 1 -- 172.21.15.172:0/986208215 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9024004520 con 0x7f90340c3610 2024-03-16T19:29:42.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.126+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903414e7c0 con 0x7f90340c3610 2024-03-16T19:29:42.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.126+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90340c5cb0 con 0x7f90340c3610 2024-03-16T19:29:42.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.127+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90340c79a0 con 0x7f90340c3610 2024-03-16T19:29:42.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.127+0000 7f90317fa640 1 -- 172.21.15.172:0/986208215 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f902403a870 con 0x7f90340c3610 2024-03-16T19:29:42.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.127+0000 7f90317fa640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f902006b3e0 0x7f902006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.128+0000 7f90317fa640 1 -- 172.21.15.172:0/986208215 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90240abc30 con 0x7f90340c3610 2024-03-16T19:29:42.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.128+0000 7f9033fff640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f902006b3e0 0x7f902006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.128+0000 7f9033fff640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f902006b3e0 0x7f902006d8a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f90340c9a10 tx=0x7f9028007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.132+0000 7f90317fa640 1 -- 172.21.15.172:0/986208215 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90340c79a0 con 0x7f90340c3610 2024-03-16T19:29:42.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.305+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f90340c79a0 con 0x7f90340c3610 2024-03-16T19:29:42.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.306+0000 7f90317fa640 1 -- 172.21.15.172:0/986208215 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v21) v1 ==== 56+0+3255 (secure 0 0 0) 0x7f90340c79a0 con 0x7f90340c3610 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 21 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:fsid 6c00e600-a879-4eb6-acee-0fdf0f76922a 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:created 2024-03-16T19:29:02.010581+0000 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:modified 2024-03-16T19:29:27.336190+0000 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:crush_version 4 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:full_ratio 0.95 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:backfillfull_ratio 0.9 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:nearfull_ratio 0.85 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:require_min_compat_client luminous 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:min_compat_client jewel 2024-03-16T19:29:42.288 INFO:teuthology.orchestra.run.smithi172.stdout:require_osd_release squid 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:stretch_mode_enabled false 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:pool 2 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:pool 3 'cephfs_data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:max_osd 8 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] [v2:172.21.15.172:6802/491754535,v1:172.21.15.172:6803/491754535] exists,up 460ed9e4-eacf-4544-9199-23ca7208b663 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] [v2:172.21.15.172:6810/3098410089,v1:172.21.15.172:6811/3098410089] exists,up b84ebb50-079d-460f-bf05-71040f17500a 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] [v2:172.21.15.172:6826/1599824697,v1:172.21.15.172:6827/1599824697] exists,up 6774c42c-5373-4543-8f8c-ed7b82c12057 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] [v2:172.21.15.172:6818/1613555619,v1:172.21.15.172:6819/1613555619] exists,up b96838ef-d991-43ab-b2d7-16deab58ca78 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] [v2:172.21.15.174:6810/967866068,v1:172.21.15.174:6811/967866068] exists,up f5c78559-d22e-4847-be3f-2f597d7f6789 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] [v2:172.21.15.174:6818/3985312519,v1:172.21.15.174:6820/3985312519] exists,up e1f4797f-31c3-4292-a4a1-88fa0ab64f9a 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] [v2:172.21.15.174:6823/3676139844,v1:172.21.15.174:6824/3676139844] exists,up ea3e888e-c4c6-444a-8ebe-e26cddb2084b 2024-03-16T19:29:42.289 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] [v2:172.21.15.174:6802/339941177,v1:172.21.15.174:6803/339941177] exists,up bb0252a5-3aa1-43f0-be38-37413819720b 2024-03-16T19:29:42.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f902006b3e0 msgr2=0x7f902006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f902006b3e0 0x7f902006d8a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f90340c9a10 tx=0x7f9028007f00 comp rx=0 tx=0).stop 2024-03-16T19:29:42.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90340c3610 msgr2=0x7f90341ee150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90340c3610 0x7f90341ee150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9024000c00 tx=0x7f90240046c0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 shutdown_connections 2024-03-16T19:29:42.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9034152840 0x7f9034156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f902006b3e0 0x7f902006d8a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90340cb2b0 0x7f90340c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 --2- 172.21.15.172:0/986208215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90340c3610 0x7f90341ee150 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 >> 172.21.15.172:0/986208215 conn(0x7f9034070860 msgr2=0x7f9034153e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:42.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 shutdown_connections 2024-03-16T19:29:42.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.308+0000 7f903ad3a640 1 -- 172.21.15.172:0/986208215 wait complete. 2024-03-16T19:29:42.300 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] [v2:172.21.15.172:6802/491754535,v1:172.21.15.172:6803/491754535] exists,up 460ed9e4-eacf-4544-9199-23ca7208b663', 'osd.1 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] [v2:172.21.15.172:6810/3098410089,v1:172.21.15.172:6811/3098410089] exists,up b84ebb50-079d-460f-bf05-71040f17500a', 'osd.2 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] [v2:172.21.15.172:6826/1599824697,v1:172.21.15.172:6827/1599824697] exists,up 6774c42c-5373-4543-8f8c-ed7b82c12057', 'osd.3 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] [v2:172.21.15.172:6818/1613555619,v1:172.21.15.172:6819/1613555619] exists,up b96838ef-d991-43ab-b2d7-16deab58ca78', 'osd.4 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] [v2:172.21.15.174:6810/967866068,v1:172.21.15.174:6811/967866068] exists,up f5c78559-d22e-4847-be3f-2f597d7f6789', 'osd.5 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] [v2:172.21.15.174:6818/3985312519,v1:172.21.15.174:6820/3985312519] exists,up e1f4797f-31c3-4292-a4a1-88fa0ab64f9a', 'osd.6 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] [v2:172.21.15.174:6823/3676139844,v1:172.21.15.174:6824/3676139844] exists,up ea3e888e-c4c6-444a-8ebe-e26cddb2084b', 'osd.7 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] [v2:172.21.15.174:6802/339941177,v1:172.21.15.174:6803/339941177] exists,up bb0252a5-3aa1-43f0-be38-37413819720b'] 2024-03-16T19:29:42.301 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-03-16T19:29:42.405 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:42.405 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "mon.a", 2024-03-16T19:29:42.405 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster": "ceph", 2024-03-16T19:29:42.405 INFO:teuthology.orchestra.run.smithi172.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-16T19:29:42.405 INFO:teuthology.orchestra.run.smithi172.stdout: "admin_socket_mode": "", 2024-03-16T19:29:42.405 INFO:teuthology.orchestra.run.smithi172.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-03-16T19:29:42.405 INFO:teuthology.orchestra.run.smithi172.stdout: "auth_client_required": "cephx, none", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "auth_cluster_required": "cephx", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "auth_debug": "true", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "auth_service_required": "cephx", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "auth_supported": "cephx", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_aio": "true", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_aio_poll_ms": "250", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_aio_reap_max": "16", 2024-03-16T19:29:42.406 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_async_discard": "false", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_block_size": "4096", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_debug_aio": "false", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_debug_inflight_ios": "false", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_enable_discard": "false", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_flock_retry": "3", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_inject_crash": "0", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_ioring": "false", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_ioring_hipri": "false", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-16T19:29:42.407 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bdev_type": "", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_alloc_size": "1048576", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_allocator": "hybrid", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_buffered_io": "true", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_check_for_zeros": "false", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_compact_log_sync": "false", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-16T19:29:42.408 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_max_log_runway": "4194304", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_max_prefetch": "1048576", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_min_flush_size": "524288", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_min_log_runway": "1048576", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_replay_recovery": "false", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_sync_write": "false", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_allocation_from_file": "true", 2024-03-16T19:29:42.409 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_allocator": "hybrid", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_create": "true", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_db_create": "false", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_db_path": "", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_db_size": "0", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_path": "", 2024-03-16T19:29:42.410 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_preallocate_file": "false", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_size": "107374182400", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_wal_create": "false", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_wal_path": "", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_block_wal_size": "100663296", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_bluefs": "true", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-16T19:29:42.411 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_autotune": "true", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_size": "0", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_type": "2q", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_clone_cow": "true", 2024-03-16T19:29:42.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_compression_mode": "none", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_csum_type": "crc32c", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_freelist": "false", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-16T19:29:42.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_small_allocations": "0", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_default_buffered_read": "true", 2024-03-16T19:29:42.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_default_buffered_write": "false", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fail_eio": "false", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-16T19:29:42.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_on_mount": "false", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_on_umount": "false", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_ignore_data_csum": "false", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-16T19:29:42.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_log_op_age": "5.000000", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_max_alloc_size": "0", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_max_blob_size": "0", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_max_deferred_txc": "32", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_min_alloc_size": "0", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_nid_prealloc": "1024", 2024-03-16T19:29:42.417 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_retry_disk_reads": "3", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_rocksdb_cf": "true", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_spdk_mem": "512", 2024-03-16T19:29:42.418 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_tracing": "false", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-16T19:29:42.419 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_zero_block_detection": "false", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-16T19:29:42.420 INFO:teuthology.orchestra.run.smithi172.stdout: "cephx_cluster_require_signatures": "false", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "cephx_cluster_require_version": "2", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "cephx_require_signatures": "false", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "cephx_require_version": "2", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "cephx_service_require_signatures": "false", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "cephx_service_require_version": "2", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "cephx_sign_messages": "true", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "chdir": "", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_acl_type": "", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_asio_thread_count": "2", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_cache_mid": "0.750000", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_cache_size": "16384", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_caps_release_delay": "5", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_check_pool_perm": "true", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_debug_force_sync_read": "false", 2024-03-16T19:29:42.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_debug_getattr_caps": "false", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_debug_inject_tick_delay": "0", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_die_on_failed_remount": "false", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_dirsize_rbytes": "true", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_force_lazyio": "false", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_fs": "", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_inject_release_failure": "false", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_max_inline_size": "4096", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_mds_namespace": "", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_metadata": "", 2024-03-16T19:29:42.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_mount_gid": "-1", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_mount_timeout": "300", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_mount_uid": "-1", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_mountpoint": "/", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_notify_timeout": "10", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_oc": "true", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_oc_max_dirty": "104857600", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_oc_max_objects": "1000", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_oc_size": "209715200", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_oc_target_dirty": "8388608", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_permissions": "true", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_quota": "true", 2024-03-16T19:29:42.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_quota_df": "true", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_readahead_max_bytes": "0", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_readahead_max_periods": "4", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_readahead_min": "131072", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_reconnect_stale": "false", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_shutdown_timeout": "30", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_snapdir": ".snap", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_tick_interval": "1", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_trace": "", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_try_dentry_invalidate": "false", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_use_faked_inos": "false", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_use_random_mds": "false", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "clog_to_graylog": "false", 2024-03-16T19:29:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "clog_to_graylog_port": "12201", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "clog_to_monitors": "default=true", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "clog_to_syslog": "false", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "clog_to_syslog_level": "info", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "-", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_network": "", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_network_interface": "", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "compressor_zlib_isal": "false", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "compressor_zlib_level": "5", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "compressor_zlib_winsize": "-15", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "compressor_zstd_level": "1", 2024-03-16T19:29:42.425 INFO:teuthology.orchestra.run.smithi172.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "crimson_alien_op_num_threads": "6", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "crimson_seastar_cpu_cores": "", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_location": "", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_location_hook": "", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_location_hook_timeout": "10", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "daemonize": "false", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "daos_pool": "tank", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-16T19:29:42.426 INFO:teuthology.orchestra.run.smithi172.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_alienstore": "0/5", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_allow_any_pool_priority": "false", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_asok": "1/5", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_asok_assert_abort": "false", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_asserts_on_shutdown": "true", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_auth": "20/20", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_bdev": "1/3", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_bluefs": "1/5", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_bluestore": "1/5", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_buffer": "0/1", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_ceph_exporter": "1/5", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_cephfs_mirror": "0/5", 2024-03-16T19:29:42.427 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_cephsqlite": "0/5", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_client": "0/5", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_compressor": "1/5", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_context": "0/1", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_crimson_interrupt": "0/5", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_crush": "1/1", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_crypto": "1/5", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_cyanstore": "0/5", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_deliberately_leak_memory": "false", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_disable_randomized_ping": "false", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_dpdk": "1/5", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_eventtrace": "1/5", 2024-03-16T19:29:42.428 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_filer": "0/1", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_filestore": "1/3", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_finisher": "1/1", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_fuse": "1/5", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_heartbeat_testing_span": "0", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_heartbeatmap": "1/5", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_javaclient": "1/5", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_journal": "1/3", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_journaler": "0/5", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_kstore": "1/5", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_lockdep": "0/1", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mclock": "1/5", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mds": "1/5", 2024-03-16T19:29:42.429 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mds_balancer": "1/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mds_locker": "1/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mds_log": "1/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mds_log_expire": "1/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mds_migrator": "1/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mds_quiesce": "3/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_memstore": "1/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mgr": "2/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mgrc": "1/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_mon": "20/20", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_monc": "0/10", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_ms": "1/1", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_none": "0/5", 2024-03-16T19:29:42.430 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_objclass": "0/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_objectcacher": "0/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_objecter": "0/1", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_optracker": "0/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_osd": "1/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_paxos": "20/20", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_perfcounter": "1/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_prioritycache": "1/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rados": "0/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rbd": "0/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rbd_mirror": "0/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rbd_pwl": "0/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rbd_replay": "0/5", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_refs": "0/0", 2024-03-16T19:29:42.431 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_reserver": "1/1", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rgw": "1/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rgw_access": "1/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rgw_datacache": "1/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rgw_dbstore": "1/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rgw_flight": "1/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rgw_sync": "1/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_rocksdb": "4/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore": "0/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_backref": "0/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_cache": "0/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_cleaner": "0/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_device": "0/5", 2024-03-16T19:29:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_epm": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_journal": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_lba": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_odata": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_omap": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_onode": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_t": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_seastore_tm": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_striper": "0/1", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_test": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_throttle": "1/1", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_timer": "0/1", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_tp": "0/5", 2024-03-16T19:29:42.433 INFO:teuthology.orchestra.run.smithi172.stdout: "debug_trace": "1/5", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "device_failure_prediction_mode": "none", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "err_to_graylog": "false", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "err_to_journald": "false", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "err_to_stderr": "true", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "err_to_syslog": "false", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event_tracing": "false", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "exporter_addr": "0.0.0.0", 2024-03-16T19:29:42.434 INFO:teuthology.orchestra.run.smithi172.stdout: "exporter_http_port": "9926", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "exporter_prio_limit": "5", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "exporter_sort_metrics": "true", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "exporter_stats_period": "5", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "fake_statfs_for_testing": "0", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "fatal_signal_handlers": "true", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "filer_max_purge_ops": "10", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "filer_max_truncate_ops": "128", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_apply_finisher_threads": "1", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_blackhole": "false", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_btrfs_clone_range": "true", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_btrfs_snap": "true", 2024-03-16T19:29:42.435 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_caller_concurrency": "10", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_collect_device_partition_information": "true", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_commit_timeout": "600.000000", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_debug_inject_read_err": "false", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_debug_omap_check": "false", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_debug_verify_split": "false", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_dump_file": "", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_fadvise": "true", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_fail_eio": "true", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_fd_cache_shards": "16", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_fd_cache_size": "128", 2024-03-16T19:29:42.436 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_fiemap": "false", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_fiemap_threshold": "4096", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_inject_stall": "0", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_journal_parallel": "false", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_journal_trailing": "false", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_journal_writeahead": "false", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_kill_at": "0", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-16T19:29:42.437 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_inline_xattrs": "0", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_xattr_value_size": "0", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_merge_threshold": "-10", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_odsync_write": "false", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_omap_backend": "rocksdb", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_omap_backend_path": "", 2024-03-16T19:29:42.438 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_op_thread_timeout": "60", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_op_threads": "2", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_punch_hole": "false", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-16T19:29:42.439 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_queue_max_ops": "50", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_seek_data_hole": "false", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_sloppy_crc": "false", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_splice": "false", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_split_multiple": "2", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_split_rand_factor": "20", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_update_to": "1000", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_enable": "true", 2024-03-16T19:29:42.440 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_xfs_extsize": "false", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "filestore_zfs_snap": "false", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "fio_dir": "/tmp/fio", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "fsid": "6c00e600-a879-4eb6-acee-0fdf0f76922a", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_allow_other": "true", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_atomic_o_trunc": "true", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_big_writes": "true", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_debug": "false", 2024-03-16T19:29:42.441 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_default_permissions": "false", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_disable_pagecache": "false", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_max_write": "0", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_multithreaded": "true", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_require_active_mds": "true", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_set_user_groups": "true", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_splice_move": "true", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_splice_read": "true", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_splice_write": "true", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "fuse_use_invalidate_cb": "true", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "gss_target_name": "ceph", 2024-03-16T19:29:42.442 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_file": "", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_inject_failure": "0", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_interval": "5", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "host": "smithi172", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-16T19:29:42.443 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "inject_early_sigterm": "false", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "jaeger_agent_port": "6799", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "jaeger_tracing_enable": "false", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_aio": "true", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_align_min_size": "65536", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_block_align": "true", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_block_size": "4096", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_dio": "true", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_discard": "false", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_force_aio": "false", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_ignore_corruption": "false", 2024-03-16T19:29:42.444 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_max_write_bytes": "10485760", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_max_write_entries": "100", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_replay_from": "0", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_write_header_frequency": "0", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journal_zero_on_create": "false", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journaler_prefetch_periods": "10", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journaler_prezero_periods": "5", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "journaler_write_head_interval": "15", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "keyfile": "", 2024-03-16T19:29:42.445 INFO:teuthology.orchestra.run.smithi172.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_backend": "rocksdb", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_default_stripe_size": "65536", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_fsck_on_mount": "false", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_max_bytes": "67108864", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_max_ops": "512", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_nid_prealloc": "1024", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_onode_map_size": "1024", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_sync_submit_transaction": "false", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "kstore_sync_transaction": "false", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "librados_thread_count": "2", 2024-03-16T19:29:42.446 INFO:teuthology.orchestra.run.smithi172.stdout: "lockdep": "false", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "lockdep_force_backtrace": "false", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_coarse_timestamps": "true", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_flush_on_exit": "false", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_graylog_host": "127.0.0.1", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_graylog_port": "12201", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_max_new": "1000", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_max_recent": "10000", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_stderr_prefix": "", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_stop_at_utilization": "0.970000", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_to_file": "true", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_to_graylog": "false", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_to_journald": "false", 2024-03-16T19:29:42.447 INFO:teuthology.orchestra.run.smithi172.stdout: "log_to_stderr": "false", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "log_to_syslog": "false", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "max_rotating_auth_attempts": "10", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_action_on_write_error": "1", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_alternate_name_max": "8192", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_asio_thread_count": "2", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_export_pin": "true", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_fragment_dirs": "true", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_fragment_interval": "5", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_interval": "10", 2024-03-16T19:29:42.448 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_max": "-1", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_max_until": "-1", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_merge_size": "50", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_midchunk": "0.300000", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_min_start": "0.200000", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_minchunk": "0.001000", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_mode": "0", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_need_max": "1.200000", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_need_min": "0.800000", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_overload_epochs": "2", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_split_bits": "3", 2024-03-16T19:29:42.449 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_split_size": "10000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_target_decay": "10.000000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_beacon_grace": "15.000000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_beacon_interval": "4.000000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache_mid": "0.700000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache_release_free_interval": "10", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache_reservation": "0.050000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache_trim_interval": "1", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache_trim_threshold": "262144", 2024-03-16T19:29:42.450 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_client_prealloc_inos": "1000", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_connect_bootstrapping": "false", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_damage_table_max_entries": "10000", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_debug_auth_pins": "false", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_debug_frag": "false", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_debug_scatterstat": "false", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_debug_subtrees": "false", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_decay_halflife": "5.000000", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_default_dir_hash": "2", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_defer_session_stale": "true", 2024-03-16T19:29:42.451 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_deny_all_reconnect": "false", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_dir_keys_per_op": "16384", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_dir_max_commit_size": "10", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_dir_max_entries": "0", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_dir_prefetch": "true", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_dump_cache_on_map": "false", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_early_reply": "true", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_enable_op_tracker": "true", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_enforce_unique_name": "true", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-16T19:29:42.452 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_export_ephemeral_random": "true", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_health_summarize_threshold": "10", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_inject_health_dummy": "false", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_inject_migrator_session_race": "false", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-16T19:29:42.453 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_join_fs": "", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_journal_format": "1", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_create_at": "0", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_export_at": "0", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_import_at": "0", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_journal_at": "0", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_journal_expire_at": "0", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_journal_replay_at": "0", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_link_at": "0", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_mdstable_at": "0", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_openc_at": "0", 2024-03-16T19:29:42.454 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_rename_at": "0", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_shutdown_at": "0", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_event_large_threshold": "524288", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_events_per_segment": "1024", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_max_events": "-1", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_max_segments": "128", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_pause": "false", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_segment_size": "0", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_trim_threshold": "128", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-16T19:29:42.455 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log_warn_factor": "2.000000", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_caps_per_client": "1048576", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_completed_flushes": "100000", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_completed_requests": "100000", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_export_size": "20971520", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_file_recover": "32", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_purge_files": "64", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_purge_ops": "8192", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_max_snaps_per_dir": "100", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_metrics_update_interval": "2", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_min_caps_per_client": "100", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_min_caps_working_set": "10000", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-16T19:29:42.456 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_numa_node": "-1", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_op_complaint_time": "30.000000", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_op_history_duration": "600", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_op_history_size": "20", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_op_history_slow_op_size": "20", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_op_log_threshold": "5", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_ping_grace": "15", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_ping_interval": "5", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_recall_max_caps": "30000", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-16T19:29:42.457 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_recall_warning_threshold": "262144", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_replay_interval": "1.000000", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_root_ino_gid": "0", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_root_ino_uid": "0", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-16T19:29:42.458 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_shutdown_check": "0", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_skip_ino": "0", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_snap_max_uid": "4294967294", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_snap_min_uid": "0", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_snap_rstat": "false", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_standby_replay_damaged": "false", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_symlink_recovery": "true", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_thrash_exports": "0", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_thrash_fragments": "0", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_tick_interval": "5.000000", 2024-03-16T19:29:42.459 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_valgrind_exit": "false", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_verify_backtrace": "1", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_verify_scatter": "false", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_wipe_sessions": "false", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mempool_debug": "false", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "memstore_device_bytes": "1073741824", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "memstore_page_set": "false", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "memstore_page_size": "65536", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_client_bytes": "134217728", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_client_messages": "512", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-16T19:29:42.460 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_disabled_modules": "", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_max_pg_num_change": "128", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_mds_bytes": "134217728", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_mds_messages": "128", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_mon_bytes": "134217728", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_mon_messages": "128", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_osd_bytes": "536870912", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_osd_messages": "8192", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_pool": "true", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_standby_modules": "true", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_stats_period": "5", 2024-03-16T19:29:42.461 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_stats_threshold": "5", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_tick_period": "2", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_allow_pool_delete": "true", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_allow_pool_size_one": "true", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_auth_validate_all_caps": "true", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_bytes": "104857600", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_directed_command_retry": "2", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_hunt_parallel": "3", 2024-03-16T19:29:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_log_interval": "1.000000", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_ping_interval": "10.000000", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_client_target_rank": "-1", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_file_level": "debug", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_to_file": "true", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_to_journald": "false", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-16T19:29:42.463 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_compact_on_bootstrap": "false", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_compact_on_start": "false", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_compact_on_trim": "true", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_cpu_threads": "4", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_crush_min_required_version": "hammer", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_daemon_bytes": "419430400", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_data_avail_crit": "5", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_data_avail_warn": "5", 2024-03-16T19:29:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_data_size_warn": "16106127360", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_dump_json": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_dump_transactions": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_extra_checks": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_no_require_reef": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_no_require_squid": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-16T19:29:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_down_mkfs_grace": "120", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_election_default_strategy": "1", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_election_timeout": "5.000000", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_elector_ping_divisor": "2", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_enable_op_tracker": "true", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_fake_pool_delete": "false", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_force_quorum_join": "false", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_globalid_prealloc": "10000", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_health_detail_to_clog": "false", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_health_log_update_period": "5", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_health_max_detail": "50", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_health_to_clog": "true", 2024-03-16T19:29:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_health_to_clog_interval": "600", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_host": "172.21.15.172,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_host_override": "", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_initial_members": "", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_lease": "5.000000", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_log_full_interval": "50", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_log_max": "10000", 2024-03-16T19:29:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_log_max_summary": "50", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_max_log_epochs": "500", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_max_osd": "10000", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_max_pg_per_osd": "10000", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_max_pool_pg_num": "65536", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_mds_force_trim_to": "0", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_mds_skip_sanity": "false", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_memory_autotune": "true", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_memory_target": "2147483648", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_mgr_beacon_grace": "30", 2024-03-16T19:29:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_mgr_digest_period": "5", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_mgr_inactive_grace": "60", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_min_osdmap_epochs": "500", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_op_complaint_time": "120", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_op_history_duration": "600", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_op_history_size": "20", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_op_history_slow_op_size": "20", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_op_log_threshold": "5", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-16T19:29:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_auto_mark_in": "false", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_cache_size": "500", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_down_out_interval": "600", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_force_trim_to": "0", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-16T19:29:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_min_down_reporters": "2", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_report_timeout": "900", 2024-03-16T19:29:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osdmap_full_prune_interval": "10", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osdmap_full_prune_min": "10000", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_pg_stuck_threshold": "60", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-16T19:29:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_probe_timeout": "2.000000", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_reweight_max_change": "0.050000", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_reweight_max_osds": "4", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_scrub_interval": "86400", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_scrub_max_keys": "100", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_scrub_timeout": "300", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_session_timeout": "300", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_smart_report_timeout": "5", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_stat_smooth_intervals": "6", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-16T19:29:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_stretch_pool_min_size": "2", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_stretch_pool_size": "4", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_sync_debug": "false", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_sync_provider_kill_at": "0", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_sync_requester_kill_at": "0", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_sync_timeout": "60.000000", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_target_pg_per_osd": "100", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_tick_interval": "5", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_timecheck_interval": "300.000000", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_use_min_delay_socket": "false", 2024-03-16T19:29:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_older_version_delay": "604800", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_filestore_osds": "false", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_misplaced": "false", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_older_version": "true", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-16T19:29:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "monmap": "", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "motr_tracing_enabled": "false", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_op_threads": "3", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-16T19:29:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_cm": "false", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_device_name": "", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_dscp": "96", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_local_gid": "", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_port_num": "1", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_sl": "3", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_support_srq": "true", 2024-03-16T19:29:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_rdma_type": "ib", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_async_reap_threshold": "5", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_before_connect": "false", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_ipv4": "true", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_ipv6": "false", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_msgr1": "true", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_msgr2": "true", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_port_max": "7568", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_port_min": "6800", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_retry_count": "3", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_bind_retry_delay": "5", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_blackhole_client": "false", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_blackhole_mds": "false", 2024-03-16T19:29:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_blackhole_mgr": "false", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_blackhole_mon": "false", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_blackhole_osd": "false", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_client_mode": "crc secure", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_cluster_mode": "crc secure", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_cluster_type": "", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_compress_secure": "false", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_connection_idle_timeout": "900", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_connection_ready_timeout": "10", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_crc_data": "true", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_crc_header": "true", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_die_on_bad_msg": "false", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_die_on_bug": "true", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_die_on_old_message": "true", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_die_on_skipped_message": "false", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-16T19:29:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_coremask": "0xF", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_enable_tso": "true", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_hugepages": "", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_lro": "true", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_memory_channel": "4", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_pmd": "", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_port_id": "0", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-16T19:29:42.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_dump_on_send": "false", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_initial_backoff": "0.200000", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_inject_delay_max": "1.000000", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_inject_delay_type": "", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_inject_network_congestion": "0", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_inject_socket_failures": "0", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_learn_addr_from_peer": "true", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_max_accept_failures": "4", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_max_backoff": "15.000000", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_mon_client_mode": "secure crc", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_mon_service_mode": "secure crc", 2024-03-16T19:29:42.481 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_osd_compress_min_size": "1024", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_osd_compress_mode": "none", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_pq_min_cost": "65536", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_public_type": "", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_service_mode": "crc secure", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_tcp_listen_backlog": "512", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_tcp_nodelay": "true", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_tcp_rcvbuf": "0", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ms_type": "async+posix", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "no_config_file": "false", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter_completion_locks_per_session": "32", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-16T19:29:42.482 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter_inflight_ops": "1024", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter_tick_interval": "5.000000", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter_timeout": "10.000000", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "objectstore_blackhole": "false", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "openssl_engine_opts": "", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_agent_delay_time": "5.000000", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_agent_hist_halflife": "1000", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_agent_max_low_ops": "2", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_agent_max_ops": "4", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-16T19:29:42.483 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_agent_slop": "0.020000", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_asio_thread_count": "2", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_async_recovery_min_cost": "100", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_backfill_scan_max": "512", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_backfill_scan_min": "64", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_backoff_on_degraded": "false", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_backoff_on_peering": "false", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_backoff_on_unfound": "true", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_beacon_report_interval": "300", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_bench_duration": "30", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_bench_max_block_size": "67108864", 2024-03-16T19:29:42.484 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_blkin_trace_all": "false", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_check_for_log_corruption": "false", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_class_update_on_start": "true", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_client_message_cap": "256", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_client_message_size_cap": "524288000", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_client_op_priority": "63", 2024-03-16T19:29:42.485 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_client_watch_timeout": "30", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_command_max_records": "256", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_compact_on_start": "false", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_crush_chooseleaf_type": "0", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_crush_update_on_start": "true", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_crush_update_weight_set": "true", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_feed_pullee": "-1", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-16T19:29:42.486 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_misdirected_ops": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_no_acting_change": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_no_purge_strays": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_op_order": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_shutdown": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_trim_objects": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-16T19:29:42.487 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_verify_snaps": "false", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_deep_scrub_keys": "1024", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_deep_scrub_stride": "524288", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_default_data_pool_replay_window": "5", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_default_notify_timeout": "30", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_delete_sleep": "0.000000", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-16T19:29:42.488 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_discard_disconnected_ops": "true", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_enable_op_tracker": "true", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_fast_info": "true", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_fast_shutdown": "true", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-16T19:29:42.489 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_function_tracing": "false", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_heartbeat_grace": "20", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_heartbeat_interval": "6", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_heartbeat_min_peers": "10", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_heartbeat_min_size": "2000", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_heartbeat_stale": "600", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_hit_set_max_size": "100000", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_hit_set_min_size": "1000", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-16T19:29:42.490 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_journal_size": "5120", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_kill_backfill_at": "0", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_map_cache_size": "50", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_map_dedup": "true", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_map_message_max": "40", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_map_share_max_epochs": "40", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_attr_name_len": "100", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_attr_size": "0", 2024-03-16T19:29:42.491 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_backfills": "1", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_markdown_count": "5", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_markdown_period": "600", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_object_name_len": "2048", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_object_namespace_len": "256", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_object_size": "134217728", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_pg_blocked_by": "16", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_pg_log_entries": "10000", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_pgls": "1024", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_push_cost": "8388608", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_push_objects": "10", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_scrubs": "3", 2024-03-16T19:29:42.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_trimming_pgs": "2", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_write_op_reply_len": "64", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_max_write_size": "90", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_profile": "balanced", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-16T19:29:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_memory_base": "805306368", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_memory_cache_min": "134217728", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_memory_target": "4294967296", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_memory_target_autotune": "false", 2024-03-16T19:29:42.494 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_min_pg_log_entries": "250", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_min_recovery_priority": "0", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mon_report_interval": "5", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_num_cache_shards": "32", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_num_op_tracker_shard": "32", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_numa_auto_affinity": "true", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_numa_node": "-1", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_numa_prefer_iface": "true", 2024-03-16T19:29:42.495 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_objecter_finishers": "1", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_objectstore": "bluestore", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_objectstore_fuse": "false", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_objectstore_tracing": "false", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_complaint_time": "30.000000", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_history_duration": "600", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_history_size": "20", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_history_slow_op_size": "20", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_log_threshold": "5", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_num_shards": "0", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_num_shards_hdd": "5", 2024-03-16T19:29:42.496 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_num_shards_ssd": "8", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_pq_min_cost": "65536", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_queue_cut_off": "high", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_op_thread_timeout": "15", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_open_classes_on_start": "true", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_os_flags": "0", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_peering_op_priority": "255", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_delete_cost": "1048576", 2024-03-16T19:29:42.497 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_delete_priority": "5", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_log_trim_max": "10000", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_log_trim_min": "100", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-16T19:29:42.498 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_crimson": "false", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_flags": "0", 2024-03-16T19:29:42.499 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_min_size": "0", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_pg_num": "32", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_pgp_num": "0", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_read_ratio": "70", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_size": "2", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_default_type": "replicated", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_push_per_object_cost": "1000", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recover_clone_overlap": "true", 2024-03-16T19:29:42.500 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_cost": "20971520", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_max_active": "0", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_max_single_start": "1", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_op_priority": "3", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_priority": "5", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_sleep": "0.000000", 2024-03-16T19:29:42.501 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_repair_during_recovery": "false", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_requested_scrub_priority": "120", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_auto_repair": "false", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_begin_hour": "0", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_begin_week_day": "0", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_chunk_max": "25", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_chunk_min": "5", 2024-03-16T19:29:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_cost": "52428800", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_during_recovery": "false", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_end_hour": "0", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_end_week_day": "0", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_event_cost": "4096", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_invalid_stats": "true", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_max_preemptions": "5", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-16T19:29:42.503 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_priority": "5", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_sleep": "0.000000", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_skip_data_digest": "false", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_smart_report_timeout": "5", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_snap_trim_cost": "1048576", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_snap_trim_priority": "5", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-16T19:29:42.504 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_target_transaction_size": "30", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-16T19:29:42.505 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_tracing": "false", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_use_stale_snap": "false", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "osdc_blkin_trace_all": "false", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_kill_at": "0", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_max_join_drift": "10", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_min": "500", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_min_wait": "0.050000", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_propose_interval": "1.000000", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_service_trim_max": "500", 2024-03-16T19:29:42.506 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_service_trim_min": "250", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_stash_full_interval": "25", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_trim_max": "500", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "paxos_trim_min": "250", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "perf": "true", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "-", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrv": "", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "public_bind_addr": "-", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "public_network": "", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "public_network_interface": "", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "qat_compressor_enabled": "false", 2024-03-16T19:29:42.507 INFO:teuthology.orchestra.run.smithi172.stdout: "qat_compressor_session_max_number": "256", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rados_mon_op_timeout": "0", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rados_osd_op_timeout": "0", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rados_tracing": "false", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_atime_update_interval": "60", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_balance_parent_reads": "false", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_balance_snap_reads": "false", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_blkin_trace_all": "false", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_cache": "true", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-16T19:29:42.508 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_cache_policy": "writearound", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_cache_size": "33554432", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_clone_copy_on_read": "false", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_compression_hint": "none", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_concurrent_management_ops": "10", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_clone_format": "auto", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_data_pool": "", 2024-03-16T19:29:42.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_features": "61", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_format": "2", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_map_options": "", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_order": "22", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_pool": "rbd", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_stripe_count": "0", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_default_stripe_unit": "0", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_enable_alloc_hint": "true", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_io_scheduler": "simple", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-16T19:29:42.510 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_order": "24", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_pool": "", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_journal_splay_width": "4", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_localize_parent_reads": "false", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_localize_snap_reads": "false", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-16T19:29:42.511 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-16T19:29:42.512 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-16T19:29:42.513 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_mtime_update_interval": "60", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_non_blocking_aio": "true", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_op_thread_timeout": "60", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_op_threads": "1", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_parent_cache_enabled": "false", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_plugins": "", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_bps_burst": "0", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_bps_limit": "0", 2024-03-16T19:29:42.514 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_exclude_ops": "0", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_iops_burst": "0", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_iops_limit": "0", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-16T19:29:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_read_from_replica_policy": "default", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_skip_partial_discard": "true", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_tracing": "false", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_validate_names": "true", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "rbd_validate_pool": "true", 2024-03-16T19:29:42.516 INFO:teuthology.orchestra.run.smithi172.stdout: "restapi_base_url": "", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "restapi_log_level": "", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_acl_grants_max_num": "100", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_admin_entry": "admin", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_backend_store": "rados", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_barbican_url": "", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_beast_enable_async": "true", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_bucket_counters_cache": "false", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-16T19:29:42.517 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_cache_enabled": "true", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_cache_expiry_interval": "900", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_cache_lru_size": "10000", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_config_store": "rados", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_content_length_compat": "false", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_copy_obj_progress": "true", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_cors_rules_max_num": "100", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_cross_domain_policy": "", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_kmip_addr": "", 2024-03-16T19:29:42.518 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_kmip_password": "", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_kmip_username": "", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_require_ssl": "true", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-16T19:29:42.519 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-16T19:29:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-16T19:29:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-16T19:29:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-16T19:29:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-16T19:29:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-16T19:29:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-16T19:29:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-16T19:29:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_addr": "", 2024-03-16T19:29:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_auth": "token", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_namespace": "", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_prefix": "", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_token_file": "", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_curl_buffersize": "524288", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_curl_low_speed_time": "300", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-16T19:29:42.522 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_d4n_port": "6379", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_data_log_changes_size": "1000", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_data_log_num_shards": "128", 2024-03-16T19:29:42.523 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_data_log_window": "30", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-16T19:29:42.524 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dns_name": "", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dns_s3website_name": "", 2024-03-16T19:29:42.525 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_dynamic_resharding": "true", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_enable_gc_threads": "true", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_enable_lc_threads": "true", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_enable_ops_log": "false", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_enable_quota_threads": "true", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_enable_static_website": "false", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_enable_usage_log": "false", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_enforce_swift_acls": "true", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_exit_timeout_secs": "120", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_expose_bucket": "false", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_extended_http_attrs": "", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_filter": "none", 2024-03-16T19:29:42.526 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_frontends": "beast port=7480", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_gc_max_deferred": "50", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_gc_max_objs": "32", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_gc_processor_period": "3600", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-16T19:29:42.527 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_init_timeout": "300", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_kafka_connection_idle": "30", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_admin_domain": "", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_admin_password": "", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_admin_password_path": "", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_admin_project": "", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_admin_tenant": "", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_admin_token": "", 2024-03-16T19:29:42.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_admin_token_path": "", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_admin_user": "", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_api_version": "2", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_barbican_domain": "", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_barbican_password": "", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_barbican_project": "", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_barbican_user": "", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_url": "", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_lc_debug_interval": "-1", 2024-03-16T19:29:42.529 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_lc_lock_max_time": "90", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_lc_max_objs": "32", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_lc_max_rules": "1000", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_lc_max_worker": "3", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_lc_thread_delay": "0", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ldap_dnattr": "uid", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ldap_searchfilter": "", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-16T19:29:42.530 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_log_http_headers": "", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_log_object_name_utc": "false", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_attr_name_len": "0", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_attr_size": "0", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_chunk_size": "4194304", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-16T19:29:42.531 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_listing_results": "1000", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_notify_retries": "10", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_put_param_size": "1048576", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_put_size": "5368709120", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_max_slo_entries": "1000", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_md_log_max_shards": "64", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_mp_lock_max_time": "600", 2024-03-16T19:29:42.532 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_max_gc": "300", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-16T19:29:42.533 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_num_async_rados_threads": "32", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_num_control_oids": "8", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_numa_node": "-1", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_objexp_chunk_size": "100", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_objexp_gc_interval": "600", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_op_thread_timeout": "600", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_op_tracing": "false", 2024-03-16T19:29:42.534 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_opa_token": "", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_opa_url": "", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_opa_verify_ssl": "true", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ops_log_rados": "false", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_ops_log_socket_path": "", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_period_push_interval": "2.000000", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-16T19:29:42.535 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_posix_cache_lanes": "3", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_posix_cache_partitions": "3", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_print_continue": "true", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_rados_tracing": "false", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_realm": "", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_realm_id": "", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_region": "", 2024-03-16T19:29:42.536 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_request_uri": "", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_reshard_batch_size": "64", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_reshard_max_aio": "128", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_reshard_num_logs": "16", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_reshard_thread_interval": "600", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_resolve_cname": "false", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_run_sync_thread": "true", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-16T19:29:42.537 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_scheduler_type": "throttler", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_script_uri": "", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_service_provider_name": "", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sts_client_id": "", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sts_client_secret": "", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sts_entry": "sts", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sts_key": "sts", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-16T19:29:42.538 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sts_min_session_duration": "900", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sts_token_introspection_url": "", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_account_in_url": "false", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_auth_entry": "auth", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_auth_url": "", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_custom_header": "", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_need_stats": "true", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_tenant_name": "", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_token_expiration": "86400", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_url": "", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_url_prefix": "swift", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_lease_period": "120", 2024-03-16T19:29:42.539 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_thread_pool_size": "512", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_torrent_comment": "", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_torrent_createby": "", 2024-03-16T19:29:42.540 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_torrent_encoding": "", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_torrent_flag": "false", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_torrent_origin": "", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_torrent_tracker": "", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_trust_forwarded_https": "false", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_usage_max_shards": "32", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_usage_max_user_shards": "1", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_use_opa_authz": "false", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_counters_cache": "false", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-16T19:29:42.541 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_max_buckets": "1000", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_policies_max_num": "100", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_user_unique_email": "true", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_verify_ssl": "true", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_zone": "", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_zone_id": "", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_zonegroup": "", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_zonegroup_id": "", 2024-03-16T19:29:42.542 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rgwlc_auto_session_clear": "true", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_block_size": "4096", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_cache_size": "536870912", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-16T19:29:42.543 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_index_type": "binary_search", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_partition_filters": "false", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_perf": "false", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "run_dir": "/var/run/ceph", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_block_create": "true", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_cache_lru_size": "67108864", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_default_max_object_size": "16777216", 2024-03-16T19:29:42.544 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_device_size": "53687091200", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_journal_batch_capacity": "16", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_main_device_type": "SSD", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "seastore_segment_size": "67108864", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "set_keepcaps": "false", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "setgroup": "", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "setuser": "", 2024-03-16T19:29:42.545 INFO:teuthology.orchestra.run.smithi172.stdout: "setuser_match_path": "", 2024-03-16T19:29:42.546 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-16T19:29:42.546 INFO:teuthology.orchestra.run.smithi172.stdout: "thp": "false", 2024-03-16T19:29:42.546 INFO:teuthology.orchestra.run.smithi172.stdout: "threadpool_default_timeout": "60", 2024-03-16T19:29:42.546 INFO:teuthology.orchestra.run.smithi172.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-16T19:29:42.546 INFO:teuthology.orchestra.run.smithi172.stdout: "throttler_perf_counter": "true" 2024-03-16T19:29:42.546 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:42.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-16T19:29:42.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.666+0000 7f5912d25640 1 Processor -- start 2024-03-16T19:29:42.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.666+0000 7f5912d25640 1 -- start start 2024-03-16T19:29:42.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5912d25640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f590c154c20 0x7f590c06f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5912d25640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca820 0x7f590c06cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5912d25640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f590c0ca2d0 0x7f590c0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5912d25640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590c06c7d0 con 0x7f590c154c20 2024-03-16T19:29:42.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5912d25640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590c15e910 con 0x7f590c0ca2d0 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5912d25640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590c15ee70 con 0x7f590c0ca820 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5910ac1640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f590c154c20 0x7f590c06f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5910ac1640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f590c154c20 0x7f590c06f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34900/0 (socket says 172.21.15.172:34900) 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5910ac1640 1 -- 172.21.15.172:0/2398607113 learned_addr learned my addr 172.21.15.172:0/2398607113 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f590bfff640 1 --2- 172.21.15.172:0/2398607113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca820 0x7f590c06cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f59112c2640 1 --2- 172.21.15.172:0/2398607113 >> v2:172.21.15.174:3300/0 conn(0x7f590c0ca2d0 0x7f590c0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5910ac1640 1 -- 172.21.15.172:0/2398607113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca820 msgr2=0x7f590c06cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5910ac1640 1 --2- 172.21.15.172:0/2398607113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca820 0x7f590c06cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5910ac1640 1 -- 172.21.15.172:0/2398607113 >> v2:172.21.15.174:3300/0 conn(0x7f590c0ca2d0 msgr2=0x7f590c0ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5910ac1640 1 --2- 172.21.15.172:0/2398607113 >> v2:172.21.15.174:3300/0 conn(0x7f590c0ca2d0 0x7f590c0ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.667+0000 7f5910ac1640 1 -- 172.21.15.172:0/2398607113 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590c06d130 con 0x7f590c154c20 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.668+0000 7f5910ac1640 1 --2- 172.21.15.172:0/2398607113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c154c20 0x7f590c06f420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f590c15bab0 tx=0x7f590002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cbec73b4f1c6c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.668+0000 7f590b7fe640 1 -- 172.21.15.172:0/2398607113 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59000317e0 con 0x7f590c154c20 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.668+0000 7f590b7fe640 1 -- 172.21.15.172:0/2398607113 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5900031940 con 0x7f590c154c20 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.668+0000 7f590b7fe640 1 -- 172.21.15.172:0/2398607113 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900031cd0 con 0x7f590c154c20 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.669+0000 7f5912d25640 1 -- 172.21.15.172:0/2398607113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c154c20 msgr2=0x7f590c06f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.669+0000 7f5912d25640 1 --2- 172.21.15.172:0/2398607113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c154c20 0x7f590c06f420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f590c15bab0 tx=0x7f590002b880 comp rx=0 tx=0).stop 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.669+0000 7f5912d25640 1 -- 172.21.15.172:0/2398607113 shutdown_connections 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.669+0000 7f5912d25640 1 --2- 172.21.15.172:0/2398607113 >> v2:172.21.15.174:3300/0 conn(0x7f590c0ca2d0 0x7f590c0ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.669+0000 7f5912d25640 1 --2- 172.21.15.172:0/2398607113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca820 0x7f590c06cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.669+0000 7f5912d25640 1 --2- 172.21.15.172:0/2398607113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c154c20 0x7f590c06f420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.669+0000 7f5912d25640 1 -- 172.21.15.172:0/2398607113 >> 172.21.15.172:0/2398607113 conn(0x7f590c066c20 msgr2=0x7f590c067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:42.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.669+0000 7f5912d25640 1 -- 172.21.15.172:0/2398607113 shutdown_connections 2024-03-16T19:29:42.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.669+0000 7f5912d25640 1 -- 172.21.15.172:0/2398607113 wait complete. 2024-03-16T19:29:42.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.670+0000 7f5912d25640 1 Processor -- start 2024-03-16T19:29:42.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.670+0000 7f5912d25640 1 -- start start 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f5912d25640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca2d0 0x7f590c0c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f5912d25640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c0ca820 0x7f590c1f08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f5912d25640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c154c20 0x7f590c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f5912d25640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c0b4350 con 0x7f590c0ca820 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f5912d25640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c1f0de0 con 0x7f590c154c20 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f5912d25640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590c0c80d0 con 0x7f590c0ca2d0 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f590bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c0ca820 0x7f590c1f08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f590bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c0ca820 0x7f590c1f08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34914/0 (socket says 172.21.15.172:34914) 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f590bfff640 1 -- 172.21.15.172:0/318742887 learned_addr learned my addr 172.21.15.172:0/318742887 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:42.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f5910ac1640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca2d0 0x7f590c0c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c154c20 0x7f590c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f590bfff640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca2d0 msgr2=0x7f590c0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f590bfff640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca2d0 0x7f590c0c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f590bfff640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c154c20 msgr2=0x7f590c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f590bfff640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c154c20 0x7f590c0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f590bfff640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59000094d0 con 0x7f590c0ca820 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f5910ac1640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ca2d0 0x7f590c0c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f590bfff640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c0ca820 0x7f590c1f08a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f58f800e870 tx=0x7f58f800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f80094f0 con 0x7f590c0ca820 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f800c940 con 0x7f590c0ca820 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.671+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590c0c6fa0 con 0x7f590c0ca820 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.672+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800ccd0 con 0x7f590c0ca820 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.672+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590c0c8320 con 0x7f590c0ca820 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.672+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f590c0681c0 con 0x7f590c0ca820 2024-03-16T19:29:42.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.673+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f58f8013470 con 0x7f590c0ca820 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.673+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f58f006b3b0 0x7f58f006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.673+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58f8089cf0 con 0x7f590c0ca820 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.674+0000 7f5910ac1640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f58f006b3b0 0x7f58f006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.674+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+1289 (secure 0 0 0) 0x7f58f808b5e0 con 0x7f590c0ca820 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.674+0000 7f5912d25640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3410 0x7f590c1f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.674+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c0c3df0 con 0x7f590c1f3410 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.674+0000 7f5910ac1640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f58f006b3b0 0x7f58f006d870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5900036000 tx=0x7f590003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.674+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3410 0x7f590c1f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.674+0000 7f59112c2640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c0ca0d0 con 0x7f590c1f3410 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.674+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3410 0x7f590c1f39d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5904011f20 tx=0x7f590400f500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.674+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904010040 con 0x7f590c1f3410 2024-03-16T19:29:42.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.675+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f5904003990 con 0x7f590c1f3410 2024-03-16T19:29:42.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.675+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3410 msgr2=0x7f590c1f39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.675+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3410 0x7f590c1f39d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5904011f20 tx=0x7f590400f500 comp rx=0 tx=0).stop 2024-03-16T19:29:42.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.675+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3410 msgr2=0x7f590c1f39d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:29:42.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.675+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3410 0x7f590c1f39d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.681+0000 7f5912d25640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3f10 0x7f590c151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.681+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c06c110 con 0x7f590c1f3f10 2024-03-16T19:29:42.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.681+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3f10 0x7f590c151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.681+0000 7f59112c2640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f590c155000 con 0x7f590c1f3f10 2024-03-16T19:29:42.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.681+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3f10 0x7f590c151a10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f590401b000 tx=0x7f5904004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.682+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904023070 con 0x7f590c1f3f10 2024-03-16T19:29:42.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.683+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.0 v2:172.21.15.172:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f5904016800 con 0x7f590c1f3f10 2024-03-16T19:29:42.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.683+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3f10 msgr2=0x7f590c151a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.683+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3f10 0x7f590c151a10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f590401b000 tx=0x7f5904004a40 comp rx=0 tx=0).stop 2024-03-16T19:29:42.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.683+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3f10 msgr2=0x7f590c151a10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:29:42.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.683+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c1f3f10 0x7f590c151a10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.665 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a: {} 2024-03-16T19:29:42.666 INFO:teuthology.orchestra.run.smithi172.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-16T19:29:42.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.683+0000 7f5912d25640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c151f50 0x7f590c0c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.683+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c0cdc10 con 0x7f590c151f50 2024-03-16T19:29:42.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.683+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c151f50 0x7f590c0c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.683+0000 7f59112c2640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c1f42f0 con 0x7f590c151f50 2024-03-16T19:29:42.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.684+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c151f50 0x7f590c0c5660 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5904027000 tx=0x7f590400f500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.684+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904031420 con 0x7f590c151f50 2024-03-16T19:29:42.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.684+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f5904031580 con 0x7f590c151f50 2024-03-16T19:29:42.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.684+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c151f50 msgr2=0x7f590c0c5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.684+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c151f50 0x7f590c0c5660 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5904027000 tx=0x7f590400f500 comp rx=0 tx=0).stop 2024-03-16T19:29:42.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.684+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c151f50 msgr2=0x7f590c0c5660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:29:42.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.684+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c151f50 0x7f590c0c5660 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5904027000 tx=0x7f590400f500 comp rx=0 tx=0).stop 2024-03-16T19:29:42.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.684+0000 7f59112c2640 1 -- 172.21.15.172:0/318742887 reap_dead start 2024-03-16T19:29:42.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.690+0000 7f5912d25640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c152360 0x7f590c0c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.690+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c0009f0 con 0x7f590c152360 2024-03-16T19:29:42.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.690+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c152360 0x7f590c0c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.690+0000 7f59112c2640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f590c0b0dc0 con 0x7f590c152360 2024-03-16T19:29:42.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.691+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c152360 0x7f590c0c7560 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5904004930 tx=0x7f590402b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.691+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904040070 con 0x7f590c152360 2024-03-16T19:29:42.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.692+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f5904031900 con 0x7f590c152360 2024-03-16T19:29:42.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.692+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c152360 msgr2=0x7f590c0c7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.692+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c152360 0x7f590c0c7560 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5904004930 tx=0x7f590402b000 comp rx=0 tx=0).stop 2024-03-16T19:29:42.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.692+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c152360 msgr2=0x7f590c0c7560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:29:42.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.692+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f590c152360 0x7f590c0c7560 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5904004930 tx=0x7f590402b000 comp rx=0 tx=0).stop 2024-03-16T19:29:42.674 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b: {} 2024-03-16T19:29:42.675 INFO:teuthology.orchestra.run.smithi172.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.692+0000 7f5912d25640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ce4a0 0x7f590c0c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.692+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590c0c8700 con 0x7f590c0ce4a0 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.692+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ce4a0 0x7f590c0c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.693+0000 7f59112c2640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c0ce880 con 0x7f590c0ce4a0 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.693+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ce4a0 0x7f590c0c5ba0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5904004980 tx=0x7f5904020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.693+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590400d050 con 0x7f590c0ce4a0 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.693+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.2 v2:172.21.15.174:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f5904031c80 con 0x7f590c0ce4a0 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.693+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ce4a0 msgr2=0x7f590c0c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.693+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ce4a0 0x7f590c0c5ba0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5904004980 tx=0x7f5904020000 comp rx=0 tx=0).stop 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.693+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ce4a0 msgr2=0x7f590c0c5ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:29:42.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.693+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0ce4a0 0x7f590c0c5ba0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.699+0000 7f5912d25640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0c60e0 0x7f590c0ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.699+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590c0c8700 con 0x7f590c0c60e0 2024-03-16T19:29:42.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.699+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0c60e0 0x7f590c0ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.700+0000 7f59112c2640 1 -- 172.21.15.172:0/318742887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f590c0c64c0 con 0x7f590c0c60e0 2024-03-16T19:29:42.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.700+0000 7f59112c2640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0c60e0 0x7f590c0ae8a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5904004930 tx=0x7f5904045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.700+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590403bdd0 con 0x7f590c0c60e0 2024-03-16T19:29:42.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 <== mon.2 v2:172.21.15.174:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f590403bbd0 con 0x7f590c0c60e0 2024-03-16T19:29:42.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0c60e0 msgr2=0x7f590c0ae8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0c60e0 0x7f590c0ae8a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5904004930 tx=0x7f5904045000 comp rx=0 tx=0).stop 2024-03-16T19:29:42.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f59097fa640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0c60e0 msgr2=0x7f590c0ae8a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:29:42.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f59097fa640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f590c0c60e0 0x7f590c0ae8a0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5904004930 tx=0x7f5904045000 comp rx=0 tx=0).stop 2024-03-16T19:29:42.683 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c: {} 2024-03-16T19:29:42.684 INFO:teuthology.orchestra.run.smithi172.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-16T19:29:42.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f58f006b3b0 msgr2=0x7f58f006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f5912d25640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f58f006b3b0 0x7f58f006d870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5900036000 tx=0x7f590003d040 comp rx=0 tx=0).stop 2024-03-16T19:29:42.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c0ca820 msgr2=0x7f590c1f08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f5912d25640 1 --2- 172.21.15.172:0/318742887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f590c0ca820 0x7f590c1f08a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f58f800e870 tx=0x7f58f800ed30 comp rx=0 tx=0).stop 2024-03-16T19:29:42.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.701+0000 7f59112c2640 1 -- 172.21.15.172:0/318742887 reap_dead start 2024-03-16T19:29:42.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.702+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 shutdown_connections 2024-03-16T19:29:42.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.702+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 >> 172.21.15.172:0/318742887 conn(0x7f590c066c20 msgr2=0x7f590c152740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:42.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.702+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 shutdown_connections 2024-03-16T19:29:42.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.702+0000 7f5912d25640 1 -- 172.21.15.172:0/318742887 wait complete. 2024-03-16T19:29:42.694 DEBUG:teuthology.orchestra.run.smithi174:> type ceph-objectstore-tool 2024-03-16T19:29:42.708 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-16T19:29:42.708 DEBUG:teuthology.orchestra.run.smithi172:> type ceph-objectstore-tool 2024-03-16T19:29:42.722 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-16T19:29:42.725 INFO:teuthology.run_tasks:Running task workunit... 2024-03-16T19:29:42.738 INFO:tasks.workunit:Pulling workunits from ref aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:29:42.738 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-16T19:29:42.738 INFO:tasks.workunit:timeout=3h 2024-03-16T19:29:42.738 INFO:tasks.workunit:cleanup=True 2024-03-16T19:29:42.739 DEBUG:teuthology.orchestra.run.smithi172:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-16T19:29:42.739 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-16T19:29:42.739 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:29:42.739 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:29:42.739 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-16T19:29:42.739 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-03-16T19:29:42.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 0 2024-03-16T19:29:42.740 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-16T19:29:42.740 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-16T19:29:42.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:29:42.740 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-16T19:29:42.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:29:42.741 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-16T19:29:42.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:29:42.764 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:42.782+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-16T19:29:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-16T19:29:42.780 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 29h/41d Inode: 1 Links: 2 2024-03-16T19:29:42.780 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T19:29:42.780 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-16T19:29:42.780 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-16T19:29:42.780 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-16 19:29:31.354310095 +0000 2024-03-16T19:29:42.780 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-16 19:29:39.843045373 +0000 2024-03-16T19:29:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-16 19:29:31.354310095 +0000 2024-03-16T19:29:42.781 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-16T19:29:42.781 DEBUG:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-16T19:29:42.841 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:42.861+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:42.864 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T19:29:42.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.888+0000 7f0586478640 1 Processor -- start 2024-03-16T19:29:42.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.888+0000 7f0586478640 1 -- start start 2024-03-16T19:29:42.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.889+0000 7f0586478640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f058006f370 0x7f05800ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.889+0000 7f0586478640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.889+0000 7f0586478640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058015e090 con 0x7f0580154c50 2024-03-16T19:29:42.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.889+0000 7f0586478640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058015eeb0 con 0x7f058006f370 2024-03-16T19:29:42.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.889+0000 7f0586478640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058015e330 con 0x7f05800ce7a0 2024-03-16T19:29:42.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.889+0000 7f057ffff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0580154c50 0x7f058015b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:55148/0 (socket says 172.21.15.172:55148) 2024-03-16T19:29:42.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.889+0000 7f057ffff640 1 -- 172.21.15.172:0/727905760 learned_addr learned my addr 172.21.15.172:0/727905760 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:42.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.889+0000 7f057f7fe640 1 --2- 172.21.15.172:0/727905760 >> v2:172.21.15.172:3300/0 conn(0x7f058006f370 0x7f05800ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.889+0000 7f0584a15640 1 --2- 172.21.15.172:0/727905760 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.890+0000 7f0584a15640 1 -- 172.21.15.172:0/727905760 >> v2:172.21.15.172:3300/0 conn(0x7f058006f370 msgr2=0x7f05800ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:42.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.890+0000 7f0584a15640 1 --2- 172.21.15.172:0/727905760 >> v2:172.21.15.172:3300/0 conn(0x7f058006f370 0x7f05800ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.890+0000 7f0584a15640 1 -- 172.21.15.172:0/727905760 >> v1:172.21.15.172:6789/0 conn(0x7f0580154c50 legacy=0x7f058015b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.890+0000 7f0584a15640 1 -- 172.21.15.172:0/727905760 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058015e610 con 0x7f05800ce7a0 2024-03-16T19:29:42.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.890+0000 7f057effd640 1 -- 172.21.15.172:0/727905760 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985539713 0 0) 0x7f058015e090 con 0x7f0580154c50 2024-03-16T19:29:42.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.891+0000 7f0584a15640 1 --2- 172.21.15.172:0/727905760 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c9ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0578009d90 tx=0x7f057802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ae5dad5856d3184 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.891+0000 7f057effd640 1 -- 172.21.15.172:0/727905760 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057802cce0 con 0x7f05800ce7a0 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.891+0000 7f057effd640 1 -- 172.21.15.172:0/727905760 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057802ce40 con 0x7f05800ce7a0 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.891+0000 7f057effd640 1 -- 172.21.15.172:0/727905760 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578034660 con 0x7f05800ce7a0 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7f0586478640 1 -- 172.21.15.172:0/727905760 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 msgr2=0x7f05800c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7f0586478640 1 --2- 172.21.15.172:0/727905760 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c9ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0578009d90 tx=0x7f057802bf30 comp rx=0 tx=0).stop 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7f0586478640 1 -- 172.21.15.172:0/727905760 shutdown_connections 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7f0586478640 1 --2- 172.21.15.172:0/727905760 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c9ef0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7f0586478640 1 --2- 172.21.15.172:0/727905760 >> v2:172.21.15.172:3300/0 conn(0x7f058006f370 0x7f05800ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7f0586478640 1 -- 172.21.15.172:0/727905760 >> 172.21.15.172:0/727905760 conn(0x7f0580066c20 msgr2=0x7f0580067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7f0586478640 1 -- 172.21.15.172:0/727905760 shutdown_connections 2024-03-16T19:29:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7ff8ecd6c640 1 Processor -- start 2024-03-16T19:29:42.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7ff8ecd6c640 1 -- start start 2024-03-16T19:29:42.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.892+0000 7f0586478640 1 -- 172.21.15.172:0/727905760 wait complete. 2024-03-16T19:29:42.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.893+0000 7ff8ecd6c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff8e8151910 0x7ff8e8072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.893+0000 7ff8ecd6c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff8e80c3ea0 0x7ff8e80c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.893+0000 7ff8ecd6c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3930 0x7ff8e80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.893+0000 7ff8ecd6c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e80c2330 con 0x7ff8e8151910 2024-03-16T19:29:42.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.893+0000 7ff8ecd6c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e8072b10 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.893+0000 7ff8ecd6c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80c7da0 con 0x7ff8e80c3930 2024-03-16T19:29:42.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.893+0000 7ff8e659c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff8e8151910 0x7ff8e8072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.893+0000 7ff8e659c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff8e8151910 0x7ff8e8072fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34938/0 (socket says 172.21.15.172:34938) 2024-03-16T19:29:42.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.893+0000 7ff8e659c640 1 -- 172.21.15.172:0/826867787 learned_addr learned my addr 172.21.15.172:0/826867787 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:42.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.894+0000 7ff8e5d9b640 1 --2- 172.21.15.172:0/826867787 >> v2:172.21.15.174:3300/0 conn(0x7ff8e80c3ea0 0x7ff8e80c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.894+0000 7ff8e6d9d640 1 --2- 172.21.15.172:0/826867787 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3930 0x7ff8e80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.894+0000 7ff8e659c640 1 -- 172.21.15.172:0/826867787 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3930 msgr2=0x7ff8e80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:42.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.894+0000 7ff8e659c640 1 --2- 172.21.15.172:0/826867787 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3930 0x7ff8e80ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.894+0000 7ff8e659c640 1 -- 172.21.15.172:0/826867787 >> v2:172.21.15.174:3300/0 conn(0x7ff8e80c3ea0 msgr2=0x7ff8e80c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:42.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.894+0000 7ff8e659c640 1 --2- 172.21.15.172:0/826867787 >> v2:172.21.15.174:3300/0 conn(0x7ff8e80c3ea0 0x7ff8e80c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.894+0000 7ff8e659c640 1 -- 172.21.15.172:0/826867787 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8071b10 con 0x7ff8e8151910 2024-03-16T19:29:42.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.894+0000 7f0586478640 1 Processor -- start 2024-03-16T19:29:42.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.894+0000 7f0586478640 1 -- start start 2024-03-16T19:29:42.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7ff8e659c640 1 --2- 172.21.15.172:0/826867787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e8151910 0x7ff8e8072fa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80ce020 tx=0x7ff8d802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ab270ad8764f3f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7ff8e559a640 1 -- 172.21.15.172:0/826867787 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d802ca80 con 0x7ff8e8151910 2024-03-16T19:29:42.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7ff8e559a640 1 -- 172.21.15.172:0/826867787 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d802cbe0 con 0x7ff8e8151910 2024-03-16T19:29:42.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7f0586478640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f058006f370 0x7f0580151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7f0586478640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7f0586478640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0580154c50 0x7f05800c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7f0586478640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05800b4480 con 0x7f058006f370 2024-03-16T19:29:42.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7f0586478640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05801f0220 con 0x7f0580154c50 2024-03-16T19:29:42.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7f0586478640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800c68a0 con 0x7f05800ce7a0 2024-03-16T19:29:42.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.895+0000 7ff8e559a640 1 -- 172.21.15.172:0/826867787 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d80343f0 con 0x7ff8e8151910 2024-03-16T19:29:42.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f058006f370 0x7f0580151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f058006f370 0x7f0580151bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34940/0 (socket says 172.21.15.172:34940) 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057ffff640 1 -- 172.21.15.172:0/3819919436 learned_addr learned my addr 172.21.15.172:0/3819919436 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f0584a15640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0580154c50 0x7f05800c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057f7fe640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0580154c50 msgr2=0x7f05800c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057f7fe640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0580154c50 0x7f05800c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057f7fe640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f058006f370 msgr2=0x7f0580151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057f7fe640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f058006f370 0x7f0580151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057f7fe640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0578009a70 con 0x7f05800ce7a0 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057ffff640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f058006f370 0x7f0580151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057f7fe640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c5760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f057400e870 tx=0x7f057400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057400ee60 con 0x7f05800ce7a0 2024-03-16T19:29:42.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/826867787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e8151910 msgr2=0x7ff8e8072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/826867787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e8151910 0x7ff8e8072fa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80ce020 tx=0x7ff8d802bee0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.897+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/826867787 shutdown_connections 2024-03-16T19:29:42.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f0586478640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05801eff50 con 0x7f05800ce7a0 2024-03-16T19:29:42.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.896+0000 7f0586478640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05800c6e40 con 0x7f05800ce7a0 2024-03-16T19:29:42.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.897+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057400c940 con 0x7f05800ce7a0 2024-03-16T19:29:42.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.897+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/826867787 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3930 0x7ff8e80ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.897+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/826867787 >> v2:172.21.15.174:3300/0 conn(0x7ff8e80c3ea0 0x7ff8e80c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.897+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/826867787 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e8151910 0x7ff8e8072fa0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.897+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/826867787 >> 172.21.15.172:0/826867787 conn(0x7ff8e806db70 msgr2=0x7ff8e806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:42.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.897+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/826867787 shutdown_connections 2024-03-16T19:29:42.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.897+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/826867787 wait complete. 2024-03-16T19:29:42.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.897+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057400ccd0 con 0x7f05800ce7a0 2024-03-16T19:29:42.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.898+0000 7f0586478640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0560005270 con 0x7f05800ce7a0 2024-03-16T19:29:42.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.898+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0574013470 con 0x7f05800ce7a0 2024-03-16T19:29:42.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.899+0000 7f057cff9640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f056c06b400 0x7f056c06d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.899+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0574089a10 con 0x7f05800ce7a0 2024-03-16T19:29:42.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.899+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v21) v1 ==== 54+0+16 (secure 0 0 0) 0x7f057408e240 con 0x7f05800ce7a0 2024-03-16T19:29:42.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.899+0000 7f057ffff640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f056c06b400 0x7f056c06d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.899+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f0558001650 0x7f0558003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.899+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0558006bb0 con 0x7f0558001650 2024-03-16T19:29:42.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.899+0000 7ff8ecd6c640 1 Processor -- start 2024-03-16T19:29:42.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.899+0000 7ff8ecd6c640 1 -- start start 2024-03-16T19:29:42.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.899+0000 7f0584a15640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f0558001650 0x7f0558003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.900+0000 7f057ffff640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f056c06b400 0x7f056c06d8c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0570000f60 tx=0x7f0570007e10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.900+0000 7f0584a15640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f0558001650 0x7f0558003b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.900+0000 7ff8ecd6c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e80c3930 0x7ff8e81580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8ecd6c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3ea0 0x7ff8e81f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8ecd6c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e8151910 0x7ff8e81543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e659c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e80c3930 0x7ff8e81580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0558006bb0 con 0x7f0558001650 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e5d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3ea0 0x7ff8e81f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e5d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3ea0 0x7ff8e81f4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48302/0 (socket says 172.21.15.172:48302) 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e5d9b640 1 -- 172.21.15.172:0/2700313797 learned_addr learned my addr 172.21.15.172:0/2700313797 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e659c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e80c3930 0x7ff8e81580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34950/0 (socket says 172.21.15.172:34950) 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e6d9d640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e8151910 0x7ff8e81543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8144c60 con 0x7ff8e80c3930 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e81f47e0 con 0x7ff8e8151910 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e806e180 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e5d9b640 1 -- 172.21.15.172:0/2700313797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e8151910 msgr2=0x7ff8e81543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e5d9b640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e8151910 0x7ff8e81543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e5d9b640 1 -- 172.21.15.172:0/2700313797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e80c3930 msgr2=0x7ff8e81580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e5d9b640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e80c3930 0x7ff8e81580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.901+0000 7ff8e5d9b640 1 -- 172.21.15.172:0/2700313797 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d8009b30 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.902+0000 7f911480f640 1 Processor -- start 2024-03-16T19:29:42.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.902+0000 7ff8e5d9b640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3ea0 0x7ff8e81f4c70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000e8a0 tx=0x7ff8d000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.902+0000 7f911480f640 1 -- start start 2024-03-16T19:29:42.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.902+0000 7ff8deffd640 1 -- 172.21.15.172:0/2700313797 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d00094f0 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.902+0000 7ff8deffd640 1 -- 172.21.15.172:0/2700313797 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d000c940 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.902+0000 7ff8deffd640 1 -- 172.21.15.172:0/2700313797 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d000ccf0 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.902+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e8157580 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.902+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e8157b40 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.903+0000 7f911480f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9110152ae0 0x7f91100c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.903+0000 7f911480f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f91100c84a0 0x7f91100cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.903+0000 7f911480f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f911015bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.903+0000 7f910e59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9110152ae0 0x7f91100c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.903+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8cc005290 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.903+0000 7f910dd9b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f91100c84a0 0x7f91100cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f910dd9b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f91100c84a0 0x7f91100cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34956/0 (socket says 172.21.15.172:34956) 2024-03-16T19:29:42.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f910dd9b640 1 -- 172.21.15.172:0/3462660412 learned_addr learned my addr 172.21.15.172:0/3462660412 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:42.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f910ed9d640 1 --2- 172.21.15.172:0/3462660412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f911015bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f911480f640 1 -- 172.21.15.172:0/3462660412 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91100c33b0 con 0x7f91100c84a0 2024-03-16T19:29:42.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f911480f640 1 -- 172.21.15.172:0/3462660412 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91100c3650 con 0x7f9110152ae0 2024-03-16T19:29:42.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7ff8deffd640 1 -- 172.21.15.172:0/2700313797 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff8d0013470 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f911480f640 1 -- 172.21.15.172:0/3462660412 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91100c3910 con 0x7f91100c4f40 2024-03-16T19:29:42.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7ff8deffd640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff8c806b3e0 0x7ff8c806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7ff8deffd640 1 -- 172.21.15.172:0/2700313797 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8d0089d00 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.905+0000 7ff8e659c640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff8c806b3e0 0x7ff8c806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f910e59c640 1 -- 172.21.15.172:0/3462660412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 msgr2=0x7f911015bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:42.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f910e59c640 1 --2- 172.21.15.172:0/3462660412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f911015bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f910e59c640 1 -- 172.21.15.172:0/3462660412 >> v2:172.21.15.172:3300/0 conn(0x7f91100c84a0 msgr2=0x7f91100cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:42.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f910e59c640 1 --2- 172.21.15.172:0/3462660412 >> v2:172.21.15.172:3300/0 conn(0x7f91100c84a0 0x7f91100cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.904+0000 7f910e59c640 1 -- 172.21.15.172:0/3462660412 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9110152ec0 con 0x7f9110152ae0 2024-03-16T19:29:42.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.905+0000 7ff8e659c640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff8c806b3e0 0x7ff8c806d8a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8035000 tx=0x7ff8d8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.905+0000 7f910e59c640 1 --2- 172.21.15.172:0/3462660412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9110152ae0 0x7f91100c8c10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9110067780 tx=0x7f910002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92a3a06c1e6646e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.906+0000 7f910d59a640 1 -- 172.21.15.172:0/3462660412 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100031830 con 0x7f9110152ae0 2024-03-16T19:29:42.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.906+0000 7f910d59a640 1 -- 172.21.15.172:0/3462660412 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9100031990 con 0x7f9110152ae0 2024-03-16T19:29:42.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.906+0000 7f910d59a640 1 -- 172.21.15.172:0/3462660412 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100031d20 con 0x7f9110152ae0 2024-03-16T19:29:42.897 INFO:tasks.workunit.client.0.smithi172.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-16T19:29:42.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.907+0000 7f911480f640 1 -- 172.21.15.172:0/3462660412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9110152ae0 msgr2=0x7f91100c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.907+0000 7f911480f640 1 --2- 172.21.15.172:0/3462660412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9110152ae0 0x7f91100c8c10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9110067780 tx=0x7f910002b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.907+0000 7f911480f640 1 -- 172.21.15.172:0/3462660412 shutdown_connections 2024-03-16T19:29:42.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.907+0000 7f911480f640 1 --2- 172.21.15.172:0/3462660412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f911015bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.907+0000 7f911480f640 1 --2- 172.21.15.172:0/3462660412 >> v2:172.21.15.172:3300/0 conn(0x7f91100c84a0 0x7f91100cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.907+0000 7f911480f640 1 --2- 172.21.15.172:0/3462660412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9110152ae0 0x7f91100c8c10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.907+0000 7f911480f640 1 -- 172.21.15.172:0/3462660412 >> 172.21.15.172:0/3462660412 conn(0x7f9110066c20 msgr2=0x7f9110067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:42.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.907+0000 7f911480f640 1 -- 172.21.15.172:0/3462660412 shutdown_connections 2024-03-16T19:29:42.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.907+0000 7f911480f640 1 -- 172.21.15.172:0/3462660412 wait complete. 2024-03-16T19:29:42.899 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:42.899 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:42.900 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:29:42.900 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:42.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.909+0000 7ff8deffd640 1 -- 172.21.15.172:0/2700313797 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8d00548e0 con 0x7ff8e80c3ea0 2024-03-16T19:29:42.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.910+0000 7f911480f640 1 Processor -- start 2024-03-16T19:29:42.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.910+0000 7f911480f640 1 -- start start 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.911+0000 7f911480f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f91101559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.911+0000 7f911480f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91100c84a0 0x7f91101efe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.911+0000 7f910e59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f91101559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.911+0000 7f910e59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f91101559f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48322/0 (socket says 172.21.15.172:48322) 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.911+0000 7f910e59c640 1 -- 172.21.15.172:0/2660356887 learned_addr learned my addr 172.21.15.172:0/2660356887 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.912+0000 7f911480f640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9110152ae0 0x7f911014cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.912+0000 7f910dd9b640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91100c84a0 0x7f91101efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.912+0000 7f910ed9d640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9110152ae0 0x7f911014cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.912+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9110155460 con 0x7f9110152ae0 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.912+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911015a960 con 0x7f91100c84a0 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.912+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911015a0c0 con 0x7f91100c4f40 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.913+0000 7f910e59c640 1 -- 172.21.15.172:0/2660356887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91100c84a0 msgr2=0x7f91101efe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.913+0000 7f910e59c640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91100c84a0 0x7f91101efe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.913+0000 7f910e59c640 1 -- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9110152ae0 msgr2=0x7f911014cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.913+0000 7f910e59c640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9110152ae0 0x7f911014cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.913+0000 7f910e59c640 1 -- 172.21.15.172:0/2660356887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9100009400 con 0x7f91100c4f40 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.913+0000 7f910dd9b640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91100c84a0 0x7f91101efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.913+0000 7f910ed9d640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9110152ae0 0x7f911014cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.913+0000 7f910e59c640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f91101559f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9100034040 tx=0x7f91000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.914+0000 7f90feffd640 1 -- 172.21.15.172:0/2660356887 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910003aa60 con 0x7f91100c4f40 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.914+0000 7f90feffd640 1 -- 172.21.15.172:0/2660356887 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910003abe0 con 0x7f91100c4f40 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.914+0000 7f90feffd640 1 -- 172.21.15.172:0/2660356887 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91000443f0 con 0x7f91100c4f40 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.914+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911015a3a0 con 0x7f91100c4f40 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.914+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91101ef680 con 0x7f91100c4f40 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.916+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91100688d0 con 0x7f91100c4f40 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.916+0000 7f90feffd640 1 -- 172.21.15.172:0/2660356887 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f910004d450 con 0x7f91100c4f40 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.917+0000 7f90feffd640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f90f006b400 0x7f90f006d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.917+0000 7f90feffd640 1 -- 172.21.15.172:0/2660356887 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91000abd50 con 0x7f91100c4f40 2024-03-16T19:29:42.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.917+0000 7f910dd9b640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f90f006b400 0x7f90f006d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:29:42.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.918+0000 7f910dd9b640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f90f006b400 0x7f90f006d8c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f911006cbd0 tx=0x7f9104008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.921+0000 7f90feffd640 1 -- 172.21.15.172:0/2660356887 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9100076a10 con 0x7f91100c4f40 2024-03-16T19:29:42.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.927+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0558005c00 con 0x7f0558001650 2024-03-16T19:29:42.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.929+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0558005c00 con 0x7f0558001650 2024-03-16T19:29:42.911 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:29:42.911 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:29:42.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.929+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0558004190 0x7f0558004640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.929+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0558004b80 con 0x7f0558004190 2024-03-16T19:29:42.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.929+0000 7f057ffff640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0558004190 0x7f0558004640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.930+0000 7f057ffff640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0558004190 0x7f0558004640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.931+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0558004b80 con 0x7f0558004190 2024-03-16T19:29:42.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.948+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0558005050 con 0x7f0558004190 2024-03-16T19:29:42.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.950+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0558005050 con 0x7f0558004190 2024-03-16T19:29:42.932 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:29:42.932 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:29:42.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.950+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f05580092a0 0x7f055800b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.950+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f055800e510 con 0x7f05580092a0 2024-03-16T19:29:42.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.950+0000 7f057f7fe640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f05580092a0 0x7f055800b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.951+0000 7f057f7fe640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f05580092a0 0x7f055800b690 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.952+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f055800e510 con 0x7f05580092a0 2024-03-16T19:29:42.942 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:42.960+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:42.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.968+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f055800bf00 con 0x7f05580092a0 2024-03-16T19:29:42.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.970+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f055800bf00 con 0x7f05580092a0 2024-03-16T19:29:42.952 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:29:42.953 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:29:42.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.970+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f055800e130 0x7f055800c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.970+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f055800c920 con 0x7f055800e130 2024-03-16T19:29:42.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.971+0000 7f0584a15640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f055800e130 0x7f055800c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.971+0000 7f0584a15640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f055800e130 0x7f055800c110 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.972+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f055800c920 con 0x7f055800e130 2024-03-16T19:29:42.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.986+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f055800cdf0 con 0x7f055800e130 2024-03-16T19:29:42.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.988+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f055800cdf0 con 0x7f055800e130 2024-03-16T19:29:42.970 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:29:42.970 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:29:42.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.988+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f055800d360 0x7f05580126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.988+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05580156f0 con 0x7f055800d360 2024-03-16T19:29:42.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.988+0000 7f057ffff640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f055800d360 0x7f05580126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.989+0000 7f057ffff640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f055800d360 0x7f05580126a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:42.989+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f05580156f0 con 0x7f055800d360 2024-03-16T19:29:42.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.005+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0558014410 con 0x7f055800d360 2024-03-16T19:29:42.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.006+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0558014410 con 0x7f055800d360 2024-03-16T19:29:42.988 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:29:42.989 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:29:42.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.007+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f0558014af0 0x7f0558014f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:42.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.007+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0558018b00 con 0x7f0558014af0 2024-03-16T19:29:42.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.007+0000 7f057f7fe640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f0558014af0 0x7f0558014f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:42.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.007+0000 7f057f7fe640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f0558014af0 0x7f0558014f50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:42.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.008+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0558018b00 con 0x7f0558014af0 2024-03-16T19:29:43.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.022+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0558017980 con 0x7f0558014af0 2024-03-16T19:29:43.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.024+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0558017980 con 0x7f0558014af0 2024-03-16T19:29:43.006 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:29:43.007 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:29:43.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.024+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0558018060 0x7f0558018480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:43.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.024+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f055801be30 con 0x7f0558018060 2024-03-16T19:29:43.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.024+0000 7f0584a15640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0558018060 0x7f0558018480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:43.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.025+0000 7f0584a15640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0558018060 0x7f0558018480 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:43.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.026+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f055801be30 con 0x7f0558018060 2024-03-16T19:29:43.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.041+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0558012cd0 con 0x7f0558018060 2024-03-16T19:29:43.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.043+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0558012cd0 con 0x7f0558018060 2024-03-16T19:29:43.025 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:29:43.025 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:29:43.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.043+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f05580133b0 0x7f05580137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:43.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.043+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0558014070 con 0x7f05580133b0 2024-03-16T19:29:43.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.043+0000 7f057ffff640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f05580133b0 0x7f05580137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:43.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.044+0000 7f057ffff640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f05580133b0 0x7f05580137f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:43.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.045+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0558014070 con 0x7f05580133b0 2024-03-16T19:29:43.028 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:43.028 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:43.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:29:43.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.060+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0558014240 con 0x7f05580133b0 2024-03-16T19:29:43.043 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:43.062+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:43.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f057cff9640 1 -- 172.21.15.172:0/3819919436 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0558014240 con 0x7f05580133b0 2024-03-16T19:29:43.044 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:29:43.045 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:29:43.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f0558001650 msgr2=0x7f0558003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f0558001650 0x7f0558003b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0558004190 msgr2=0x7f0558004640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0558004190 0x7f0558004640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f05580092a0 msgr2=0x7f055800b690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f05580092a0 0x7f055800b690 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f055800e130 msgr2=0x7f055800c110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f0584a15640 1 -- 172.21.15.172:0/3819919436 reap_dead start 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f055800e130 0x7f055800c110 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f055800d360 msgr2=0x7f05580126a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f055800d360 0x7f05580126a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f0558014af0 msgr2=0x7f0558014f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f0558014af0 0x7f0558014f50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0558018060 msgr2=0x7f0558018480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0558018060 0x7f0558018480 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f05580133b0 msgr2=0x7f05580137f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.062+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f05580133b0 0x7f05580137f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f0584a15640 1 -- 172.21.15.172:0/3819919436 reap_dead start 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f056c06b400 msgr2=0x7f056c06d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f056c06b400 0x7f056c06d8c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0570000f60 tx=0x7f0570007e10 comp rx=0 tx=0).stop 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 msgr2=0x7f05800c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c5760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f057400e870 tx=0x7f057400ed30 comp rx=0 tx=0).stop 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 shutdown_connections 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f056c06b400 0x7f056c06d8c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 --2- 172.21.15.172:0/3819919436 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05800ce7a0 0x7f05800c5760 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 >> 172.21.15.172:0/3819919436 conn(0x7f0580066c20 msgr2=0x7f058006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 shutdown_connections 2024-03-16T19:29:43.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.063+0000 7f05667fc640 1 -- 172.21.15.172:0/3819919436 wait complete. 2024-03-16T19:29:43.091 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.109+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff8cc005ac0 con 0x7ff8e80c3ea0 2024-03-16T19:29:43.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.138+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} v 0) v1 -- 0x7f91100c8900 con 0x7f91100c4f40 2024-03-16T19:29:43.143 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:43.161+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:43.151 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:43.152 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:43.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:29:43.245 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:43.264+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:43.312 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:43.312 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:43.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:29:43.363 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:43.381+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:43.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.404+0000 7ff8deffd640 1 -- 172.21.15.172:0/2700313797 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v22) v1 ==== 87+0+0 (secure 0 0 0) 0x7ff8d0057fa0 con 0x7ff8e80c3ea0 2024-03-16T19:29:43.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.404+0000 7f90feffd640 1 -- 172.21.15.172:0/2660356887 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]=0 set osd.0 primary-affinity to 0 (802) v22) v1 ==== 129+0+0 (secure 0 0 0) 0x7f91000350a0 con 0x7f91100c4f40 2024-03-16T19:29:43.387 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:29:43.387 INFO:teuthology.orchestra.run.smithi172.stderr:set osd.0 primary-affinity to 0 (802) 2024-03-16T19:29:43.390 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.408+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff8c806b3e0 msgr2=0x7ff8c806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.390 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.408+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff8c806b3e0 0x7ff8c806d8a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8035000 tx=0x7ff8d8030000 comp rx=0 tx=0).stop 2024-03-16T19:29:43.390 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.408+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3ea0 msgr2=0x7ff8e81f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.390 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.408+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3ea0 0x7ff8e81f4c70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000e8a0 tx=0x7ff8d000ed60 comp rx=0 tx=0).stop 2024-03-16T19:29:43.390 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.408+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f90f006b400 msgr2=0x7f90f006d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.390 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.408+0000 7f911480f640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f90f006b400 0x7f90f006d8c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f911006cbd0 tx=0x7f9104008040 comp rx=0 tx=0).stop 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.408+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 msgr2=0x7f91101559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.408+0000 7f911480f640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f91101559f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9100034040 tx=0x7f91000046c0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 shutdown_connections 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7f911480f640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9110152ae0 0x7f911014cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7f911480f640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91100c84a0 0x7f91101efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7f911480f640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f90f006b400 0x7f90f006d8c0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f911006cbd0 tx=0x7f9104008040 comp rx=0 tx=0).stop 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7f911480f640 1 --2- 172.21.15.172:0/2660356887 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91100c4f40 0x7f91101559f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 >> 172.21.15.172:0/2660356887 conn(0x7f9110066c20 msgr2=0x7f911006c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 shutdown_connections 2024-03-16T19:29:43.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e8151910 0x7ff8e81543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff8c806b3e0 0x7ff8c806d8a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e80c3ea0 0x7ff8e81f4c70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7ff8ecd6c640 1 --2- 172.21.15.172:0/2700313797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff8e80c3930 0x7ff8e81580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:43.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 >> 172.21.15.172:0/2700313797 conn(0x7ff8e806db70 msgr2=0x7ff8e806bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:43.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 shutdown_connections 2024-03-16T19:29:43.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.409+0000 7f911480f640 1 -- 172.21.15.172:0/2660356887 wait complete. 2024-03-16T19:29:43.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.411+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 shutdown_connections 2024-03-16T19:29:43.393 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:43.411+0000 7ff8ecd6c640 1 -- 172.21.15.172:0/2700313797 wait complete. 2024-03-16T19:29:43.446 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:43.464+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:43.474 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:43.475 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:43.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:29:43.567 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:43.586+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:43.630 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:43.630 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:43.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:29:43.648 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:43.667+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:43.748 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:43.767+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:43.786 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:43.786 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:43.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:29:43.848 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:43.868+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:43.943 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:43.943 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:43.949 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:43.967+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:43.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:29:44.050 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:44.069+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:44.098 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:44.098 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:44.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:29:44.151 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:44.170+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:44.213 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:44.213 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:44.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:29:44.251 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:44.271+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:44.327 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:44.328 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:44.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:29:44.352 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:44.371+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:44.448 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:44.448 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:44.452 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:44.471+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:44.457 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:29:44.554 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:44.572+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:44.565 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:44.565 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:44.573 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:29:44.654 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:44.674+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:44.680 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:44.680 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:44.688 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:29:44.771 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:44.791+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:44.857 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:44.876+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:44.876 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:44.876 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:44.888 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:29:44.958 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:44.976+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:45.034 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:45.035 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:45.043 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:29:45.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:29:45.058 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:45.077+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:45.159 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:45.177+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.177+0000 7f6bfd6ae640 1 Processor -- start 2024-03-16T19:29:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.177+0000 7f6bfd6ae640 1 -- start start 2024-03-16T19:29:45.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.178+0000 7f6bfd6ae640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6bf80c9ef0 0x7f6bf80ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.178+0000 7f6bfd6ae640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf80ca8a0 con 0x7f6bf8154c50 2024-03-16T19:29:45.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.178+0000 7f6bfd6ae640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf815e090 con 0x7f6bf806f370 2024-03-16T19:29:45.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.178+0000 7f6bfd6ae640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf815eeb0 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.178+0000 7f6bf77fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6bf80c9ef0 0x7f6bf80ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.178+0000 7f6bf6ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6bf8154c50 0x7f6bf815b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:55156/0 (socket says 172.21.15.172:55156) 2024-03-16T19:29:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.178+0000 7f6bf6ffd640 1 -- 172.21.15.172:0/3581436968 learned_addr learned my addr 172.21.15.172:0/3581436968 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.178+0000 7f6bf77fe640 1 -- 172.21.15.172:0/3581436968 >> v1:172.21.15.174:6789/0 conn(0x7f6bf806f370 legacy=0x7f6bf80ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.179+0000 7f6bf77fe640 1 -- 172.21.15.172:0/3581436968 >> v1:172.21.15.172:6789/0 conn(0x7f6bf8154c50 legacy=0x7f6bf815b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.179+0000 7f6bf5ffb640 1 -- 172.21.15.172:0/3581436968 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956231867 0 0) 0x7f6bf80ca8a0 con 0x7f6bf8154c50 2024-03-16T19:29:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.179+0000 7f6bf77fe640 1 -- 172.21.15.172:0/3581436968 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf815e330 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.179+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3581436968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 0x7f6bf80ca360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009d90 tx=0x7f6bf002bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b39bfe10be35045 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.179+0000 7f6bf5ffb640 1 -- 172.21.15.172:0/3581436968 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf002cd80 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.179+0000 7f6bf5ffb640 1 -- 172.21.15.172:0/3581436968 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf0034440 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.179+0000 7f6bf5ffb640 1 -- 172.21.15.172:0/3581436968 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf00347d0 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.180+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3581436968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 msgr2=0x7f6bf80ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.180+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3581436968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 0x7f6bf80ca360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009d90 tx=0x7f6bf002bfd0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.180+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3581436968 shutdown_connections 2024-03-16T19:29:45.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.180+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3581436968 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 0x7f6bf80ca360 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.180+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3581436968 >> 172.21.15.172:0/3581436968 conn(0x7f6bf8066c20 msgr2=0x7f6bf8067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:45.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.180+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3581436968 shutdown_connections 2024-03-16T19:29:45.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.180+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3581436968 wait complete. 2024-03-16T19:29:45.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.181+0000 7f6bfd6ae640 1 Processor -- start 2024-03-16T19:29:45.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.181+0000 7f6bfd6ae640 1 -- start start 2024-03-16T19:29:45.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bfd6ae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bf806f370 0x7f6bf8153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bfd6ae640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 0x7f6bf81e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bfd6ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bf8154c50 0x7f6bf814dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bfd6ae640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf80b0410 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bfd6ae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81e58d0 con 0x7f6bf8154c50 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bfd6ae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81e52c0 con 0x7f6bf806f370 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf67fc640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 0x7f6bf81e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf67fc640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 0x7f6bf81e5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34988/0 (socket says 172.21.15.172:34988) 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf67fc640 1 -- 172.21.15.172:0/3994377447 learned_addr learned my addr 172.21.15.172:0/3994377447 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bf806f370 0x7f6bf8153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bf8154c50 0x7f6bf814dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf67fc640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bf806f370 msgr2=0x7f6bf8153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf67fc640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bf806f370 0x7f6bf8153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf67fc640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bf8154c50 msgr2=0x7f6bf814dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf67fc640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bf8154c50 0x7f6bf814dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.182+0000 7f6bf67fc640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf0009a70 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.183+0000 7f6bf67fc640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 0x7f6bf81e5d60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00c400 tx=0x7f6bec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.183+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0052d0 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.183+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bec005430 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.183+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf81f0f50 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.183+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec003830 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.183+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf81f1230 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.184+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6bf80cbbd0 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.184+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6bec003990 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bdf7fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6be406b3b0 0x7f6be406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f6bec089070 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6be406b3b0 0x7f6be406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v23) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6bec090070 con 0x7f6bf80c9ef0 2024-03-16T19:29:45.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 0x7f6bf81e42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bf8155030 con 0x7f6bf806c9f0 2024-03-16T19:29:45.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6be406b3b0 0x7f6be406d870 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6be8000f60 tx=0x7f6be8007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 0x7f6bf81e42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bf77fe640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 msgr2=0x7f6bf81e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-03-16T19:29:45.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bf77fe640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 msgr2=0x7f6bf81e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:29:45.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 0x7f6bf81e42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:29:45.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.185+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 0x7f6bf81e42a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:29:45.200 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:45.200 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:45.211 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:29:45.259 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:45.278+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:45.318 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:45.318 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:45.327 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:29:45.360 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:45.379+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:45.367 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.386+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 0x7f6bf81e42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.368 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.386+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 0x7f6bf81e42a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.368 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.387+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6bf00345b0 con 0x7f6bf806c9f0 2024-03-16T19:29:45.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.401+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6bf815ebd0 con 0x7f6bf806c9f0 2024-03-16T19:29:45.405 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:29:45.434 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:45.434 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:45.443 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:29:45.461 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:45.479+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:45.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.526+0000 7fd911b43640 1 Processor -- start 2024-03-16T19:29:45.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.526+0000 7fd911b43640 1 -- start start 2024-03-16T19:29:45.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd911b43640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd90c154bb0 0x7fd90c15afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd911b43640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 0x7fd90c0c9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd911b43640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c15e240 con 0x7fd90c06eb40 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd911b43640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c0c7e00 con 0x7fd90c154bb0 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd911b43640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c15a890 con 0x7fd90c15b500 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd90b7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd90c154bb0 0x7fd90c15afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd90affd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd90c06eb40 0x7fd90c0c9180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:55166/0 (socket says 172.21.15.172:55166) 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd90affd640 1 -- 172.21.15.172:0/2853045848 learned_addr learned my addr 172.21.15.172:0/2853045848 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd90bfff640 1 --2- 172.21.15.172:0/2853045848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 0x7fd90c0c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd90b7fe640 1 -- 172.21.15.172:0/2853045848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 msgr2=0x7fd90c0c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd90b7fe640 1 --2- 172.21.15.172:0/2853045848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 0x7fd90c0c9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd90b7fe640 1 -- 172.21.15.172:0/2853045848 >> v1:172.21.15.172:6789/0 conn(0x7fd90c06eb40 legacy=0x7fd90c0c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.527+0000 7fd90a7fc640 1 -- 172.21.15.172:0/2853045848 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196501319 0 0) 0x7fd90c15e240 con 0x7fd90c06eb40 2024-03-16T19:29:45.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.528+0000 7fd90b7fe640 1 -- 172.21.15.172:0/2853045848 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c15e520 con 0x7fd90c154bb0 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.528+0000 7fd90b7fe640 1 --2- 172.21.15.172:0/2853045848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 0x7fd90c15afc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0097a0 tx=0x7fd8fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c62d71628a9a94 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.528+0000 7fd90a7fc640 1 -- 172.21.15.172:0/2853045848 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc031830 con 0x7fd90c154bb0 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.528+0000 7fd90a7fc640 1 -- 172.21.15.172:0/2853045848 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc031990 con 0x7fd90c154bb0 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.528+0000 7fd90a7fc640 1 -- 172.21.15.172:0/2853045848 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc031d20 con 0x7fd90c154bb0 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.529+0000 7fd911b43640 1 -- 172.21.15.172:0/2853045848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 msgr2=0x7fd90c15afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.529+0000 7fd911b43640 1 --2- 172.21.15.172:0/2853045848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 0x7fd90c15afc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0097a0 tx=0x7fd8fc02b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.529+0000 7fd911b43640 1 -- 172.21.15.172:0/2853045848 shutdown_connections 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.529+0000 7fd911b43640 1 --2- 172.21.15.172:0/2853045848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 0x7fd90c0c9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.529+0000 7fd911b43640 1 --2- 172.21.15.172:0/2853045848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 0x7fd90c15afc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.529+0000 7fd911b43640 1 -- 172.21.15.172:0/2853045848 >> 172.21.15.172:0/2853045848 conn(0x7fd90c0cc720 msgr2=0x7fd90c0ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.529+0000 7fd911b43640 1 -- 172.21.15.172:0/2853045848 shutdown_connections 2024-03-16T19:29:45.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.529+0000 7fd911b43640 1 -- 172.21.15.172:0/2853045848 wait complete. 2024-03-16T19:29:45.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.530+0000 7fd911b43640 1 Processor -- start 2024-03-16T19:29:45.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd911b43640 1 -- start start 2024-03-16T19:29:45.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd911b43640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c06eb40 0x7fd90c1f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd911b43640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 0x7fd90c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd911b43640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 0x7fd90c0c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd911b43640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0cc570 con 0x7fd90c154bb0 2024-03-16T19:29:45.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd911b43640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0c46f0 con 0x7fd90c06eb40 2024-03-16T19:29:45.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd911b43640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c15ba60 con 0x7fd90c15b500 2024-03-16T19:29:45.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd90affd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 0x7fd90c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd90affd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 0x7fd90c0c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:35006/0 (socket says 172.21.15.172:35006) 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd90affd640 1 -- 172.21.15.172:0/971985969 learned_addr learned my addr 172.21.15.172:0/971985969 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd90bfff640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 0x7fd90c0c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.531+0000 7fd90b7fe640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c06eb40 0x7fd90c1f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd90affd640 1 -- 172.21.15.172:0/971985969 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 msgr2=0x7fd90c0c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd90affd640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 0x7fd90c0c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd90affd640 1 -- 172.21.15.172:0/971985969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c06eb40 msgr2=0x7fd90c1f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd90affd640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c06eb40 0x7fd90c1f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd90affd640 1 -- 172.21.15.172:0/971985969 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc009400 con 0x7fd90c154bb0 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd90affd640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 0x7fd90c0c4b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd90000c530 tx=0x7fd90000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd8f3fff640 1 -- 172.21.15.172:0/971985969 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9000074e0 con 0x7fd90c154bb0 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd8f3fff640 1 -- 172.21.15.172:0/971985969 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd900007660 con 0x7fd90c154bb0 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd8f3fff640 1 -- 172.21.15.172:0/971985969 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900004a20 con 0x7fd90c154bb0 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c0cc020 con 0x7fd90c154bb0 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.532+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c0cc290 con 0x7fd90c154bb0 2024-03-16T19:29:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.533+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c0c44c0 con 0x7fd90c154bb0 2024-03-16T19:29:45.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.533+0000 7fd8f3fff640 1 -- 172.21.15.172:0/971985969 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd900004b80 con 0x7fd90c154bb0 2024-03-16T19:29:45.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.534+0000 7fd8f3fff640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd8f806b3e0 0x7fd8f806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.534+0000 7fd8f3fff640 1 -- 172.21.15.172:0/971985969 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5741+0+0 (secure 0 0 0) 0x7fd900089040 con 0x7fd90c154bb0 2024-03-16T19:29:45.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.534+0000 7fd90b7fe640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd8f806b3e0 0x7fd8f806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.534+0000 7fd90b7fe640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd8f806b3e0 0x7fd8f806d8a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc036000 tx=0x7fd8fc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.538+0000 7fd8f3fff640 1 -- 172.21.15.172:0/971985969 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd90c0c44c0 con 0x7fd90c154bb0 2024-03-16T19:29:45.549 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:45.549 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:45.558 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:29:45.601 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:45.621+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:45.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.663+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6bf0034720 con 0x7f6bf806c9f0 2024-03-16T19:29:45.645 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:29:45.646 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:29:45.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.663+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6bf806cdd0 0x7f6bf81f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.663+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bf80688d0 con 0x7f6bf806cdd0 2024-03-16T19:29:45.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.664+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6bf806cdd0 0x7f6bf81f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.664+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6bf806cdd0 0x7f6bf81f38a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.665+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6bf80688d0 con 0x7f6bf806cdd0 2024-03-16T19:29:45.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.680+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6bf81585b0 con 0x7f6bf806cdd0 2024-03-16T19:29:45.662 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:45.680+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:45.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.681+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6bf81585b0 con 0x7f6bf806cdd0 2024-03-16T19:29:45.663 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:29:45.664 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:29:45.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.681+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6bf81f42f0 0x7f6bf8061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.681+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bf81f4700 con 0x7f6bf81f42f0 2024-03-16T19:29:45.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.681+0000 7f6bf67fc640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6bf81f42f0 0x7f6bf8061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.682+0000 7f6bf67fc640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6bf81f42f0 0x7f6bf8061930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.682+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6bf81f4700 con 0x7f6bf81f42f0 2024-03-16T19:29:45.665 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:45.665 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:45.672 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:29:45.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.696+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6bf81e4cf0 con 0x7f6bf81f42f0 2024-03-16T19:29:45.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.698+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6bf81e4cf0 con 0x7f6bf81f42f0 2024-03-16T19:29:45.680 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:29:45.681 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:29:45.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.698+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6bf8062380 0x7f6bf8152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.698+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bf81e6410 con 0x7f6bf8062380 2024-03-16T19:29:45.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.698+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6bf8062380 0x7f6bf8152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.698+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6bf8062380 0x7f6bf8152190 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.699+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6bf81e6410 con 0x7f6bf8062380 2024-03-16T19:29:45.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.713+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6bf81526d0 con 0x7f6bf8062380 2024-03-16T19:29:45.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.715+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6bf81526d0 con 0x7f6bf8062380 2024-03-16T19:29:45.696 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:29:45.697 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:29:45.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.715+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6bf8152d50 0x7f6bf802bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.715+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bf8153190 con 0x7f6bf8152d50 2024-03-16T19:29:45.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.715+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6bf8152d50 0x7f6bf802bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.715+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6bf8152d50 0x7f6bf802bde0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.716+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6be80246d0 con 0x7f6bf8152d50 2024-03-16T19:29:45.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.721+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd90c0c44c0 con 0x7fd90c154bb0 2024-03-16T19:29:45.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.730+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6bf81e4e60 con 0x7f6bf8152d50 2024-03-16T19:29:45.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.732+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6be80246d0 con 0x7f6bf8152d50 2024-03-16T19:29:45.714 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:29:45.714 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:29:45.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.732+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6bf802c9a0 0x7f6bf802ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.732+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bf82094c0 con 0x7f6bf802c9a0 2024-03-16T19:29:45.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.732+0000 7f6bf67fc640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6bf802c9a0 0x7f6bf802ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.732+0000 7f6bf67fc640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6bf802c9a0 0x7f6bf802ce00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.733+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6bf82094c0 con 0x7f6bf802c9a0 2024-03-16T19:29:45.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.747+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6bf8206c30 con 0x7f6bf802c9a0 2024-03-16T19:29:45.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.749+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6bf8206c30 con 0x7f6bf802c9a0 2024-03-16T19:29:45.730 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:29:45.731 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:29:45.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.749+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6bf82072b0 0x7f6bf82096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.749+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bf820c5b0 con 0x7f6bf82072b0 2024-03-16T19:29:45.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.749+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6bf82072b0 0x7f6bf82096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.749+0000 7f6bf77fe640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6bf82072b0 0x7f6bf82096a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.750+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6bf820c5b0 con 0x7f6bf82072b0 2024-03-16T19:29:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.764+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6bf8209be0 con 0x7f6bf82072b0 2024-03-16T19:29:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.765+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6bf8209be0 con 0x7f6bf82072b0 2024-03-16T19:29:45.747 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:29:45.748 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:29:45.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.765+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 0x7f6bf820a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:45.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.765+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bf820f9c0 con 0x7f6bf820a2c0 2024-03-16T19:29:45.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.766+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 0x7f6bf820a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.766+0000 7f6bf6ffd640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 msgr2=0x7f6bf820a720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 28 2024-03-16T19:29:45.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.766+0000 7f6bf6ffd640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 msgr2=0x7f6bf820a720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:29:45.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.766+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 0x7f6bf820a720 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:29:45.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.766+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 0x7f6bf820a720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:29:45.762 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:45.782+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:45.820 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:45.820 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:45.828 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:29:45.863 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:45.882+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:45.948 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.967+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 0x7f6bf820a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:45.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.967+0000 7f6bf6ffd640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 0x7f6bf820a720 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:45.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.968+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6bf820f9c0 con 0x7f6bf820a2c0 2024-03-16T19:29:45.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.981+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6bf820e840 con 0x7f6bf820a2c0 2024-03-16T19:29:45.963 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:45.983+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:45.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bdf7fe640 1 -- 172.21.15.172:0/3994377447 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6bf820e840 con 0x7f6bf820a2c0 2024-03-16T19:29:45.965 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:29:45.966 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:29:45.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 msgr2=0x7f6bf81e42a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6bf806c9f0 0x7f6bf81e42a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6bf806cdd0 msgr2=0x7f6bf81f38a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6bf806cdd0 0x7f6bf81f38a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6bf81f42f0 msgr2=0x7f6bf8061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6bf81f42f0 0x7f6bf8061930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6bf8062380 msgr2=0x7f6bf8152190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6bf8062380 0x7f6bf8152190 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6bf8152d50 msgr2=0x7f6bf802bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6bf8152d50 0x7f6bf802bde0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6bf802c9a0 msgr2=0x7f6bf802ce00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6bf802c9a0 0x7f6bf802ce00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6bf82072b0 msgr2=0x7f6bf82096a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6bf82072b0 0x7f6bf82096a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bf77fe640 1 -- 172.21.15.172:0/3994377447 reap_dead start 2024-03-16T19:29:45.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 msgr2=0x7f6bf820a720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 0x7f6bf820a720 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bf77fe640 1 -- 172.21.15.172:0/3994377447 reap_dead start 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bf77fe640 1 -- 172.21.15.172:0/3994377447 reap_dead start 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bf77fe640 1 -- 172.21.15.172:0/3994377447 reap_dead start 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6be406b3b0 msgr2=0x7f6be406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6be406b3b0 0x7f6be406d870 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6be8000f60 tx=0x7f6be8007e40 comp rx=0 tx=0).stop 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 msgr2=0x7f6bf81e5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 0x7f6bf81e5d60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00c400 tx=0x7f6bec00c8c0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 shutdown_connections 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6bf820a2c0 0x7f6bf820a720 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bf80c9ef0 0x7f6bf81e5d60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 --2- 172.21.15.172:0/3994377447 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6be406b3b0 0x7f6be406d870 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 >> 172.21.15.172:0/3994377447 conn(0x7f6bf8066c20 msgr2=0x7f6bf815d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:45.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.983+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 shutdown_connections 2024-03-16T19:29:45.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:45.984+0000 7f6bfd6ae640 1 -- 172.21.15.172:0/3994377447 wait complete. 2024-03-16T19:29:45.979 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:45.979 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:45.979 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:29:45.980 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:29:45.980 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:45.988 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:29:46.073 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:46.092+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:46.139 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:46.139 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:29:46.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:29:46.139 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.927381383, 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:29:46.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:29:46.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:29:46.145 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.550685836, 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:29:46.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:29:46.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.774709391999998, 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:29:46.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:29:46.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.774624978999999, 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:29:46.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:29:46.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:29:46.161 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.774617623000001, 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:29:46.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:29:46.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.167 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.582078430999999, 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:29:46.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:29:46.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:29:46.173 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.582072623999998, 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:29:46.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.581954491000001, 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:29:46.179 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:29:46.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:29:46.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.581859294000001, 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.185 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:29:46.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:29:46.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:29:46.190 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.581855282999999, 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.191 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:29:46.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:29:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:29:46.196 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.162923409000001, 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:29:46.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.162874039999998, 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:29:46.202 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:29:46.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:29:46.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.162795207999999, 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.208 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:29:46.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.162791996999999, 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:29:46.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:29:46.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:29:46.219 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.162789044, 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:29:46.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.162674070000001, 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.225 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:29:46.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.230 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.162670718000001, 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:29:46.236 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.162667601999999, 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:29:46.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.162607133000002, 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.242 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:29:46.247 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 27.553927083000001, 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:29:46.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:29:46.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:46.253 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:46.254 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:46.255 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:29:46.256 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:46.273+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:46.356 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:46.375+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:46.364 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:46.364 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:46.364 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:29:46.365 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:46.373 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:29:46.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.410+0000 7fd8f3fff640 1 -- 172.21.15.172:0/971985969 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v24) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd900053d30 con 0x7fd90c154bb0 2024-03-16T19:29:46.392 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:29:46.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd8f806b3e0 msgr2=0x7fd8f806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:46.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd8f806b3e0 0x7fd8f806d8a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc036000 tx=0x7fd8fc03c040 comp rx=0 tx=0).stop 2024-03-16T19:29:46.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 msgr2=0x7fd90c0c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:46.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 0x7fd90c0c4b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd90000c530 tx=0x7fd90000c9f0 comp rx=0 tx=0).stop 2024-03-16T19:29:46.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 shutdown_connections 2024-03-16T19:29:46.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd8f806b3e0 0x7fd8f806d8a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:46.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c15b500 0x7fd90c0c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:46.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd90c154bb0 0x7fd90c0c4b80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:46.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 --2- 172.21.15.172:0/971985969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c06eb40 0x7fd90c1f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:46.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 >> 172.21.15.172:0/971985969 conn(0x7fd90c0cc720 msgr2=0x7fd90c159060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:46.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 shutdown_connections 2024-03-16T19:29:46.396 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:46.413+0000 7fd911b43640 1 -- 172.21.15.172:0/971985969 wait complete. 2024-03-16T19:29:46.457 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:46.476+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:46.482 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:46.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:46.482 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:29:46.482 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:29:46.482 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:46.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:29:46.575 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:46.594+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:46.641 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:29:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:29:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.066347627999999, 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:29:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:29:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:29:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.276121456999999, 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:29:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:29:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:29:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.275361222000001, 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:29:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:29:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:29:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:29:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.274547266999999, 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:29:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:29:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:29:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.273907342000001, 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.273249244999999, 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.272716670000001, 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.214544767, 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:29:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:29:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.213942573000001, 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:29:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.213299628000001, 2024-03-16T19:29:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:29:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.212795667000002, 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:29:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.054649400999999, 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:29:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.054119265000001, 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:29:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.302150006, 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:29:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:29:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:29:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.30212547, 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.30212094, 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:29:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:29:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:29:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.302111128, 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:29:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:29:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.302107275000001, 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.715 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:29:46.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:29:46.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:29:46.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.302103959, 2024-03-16T19:29:46.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:29:46.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:29:46.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.300908376000001, 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:46.724 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:29:46.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:29:46.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:29:46.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:46.728 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:46.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:29:46.730 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:46.748+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:46.830 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:46.849+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:46.877 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:46.878 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:46.878 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:29:46.878 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:46.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:29:46.970 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:46.989+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:47.032 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:47.050+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:47.032 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:47.032 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:47.032 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:29:47.032 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:29:47.032 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:47.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:29:47.133 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:47.151+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:47.192 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:29:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:29:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:29:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.853005859, 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:29:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:29:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.852988744999999, 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:29:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:29:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.852981464999999, 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:29:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.852967597999999, 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.852294763, 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:29:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:29:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.849996513000001, 2024-03-16T19:29:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.849951411999999, 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:29:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:29:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:29:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.849923357, 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:29:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:29:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.849901943000001, 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.849879592000001, 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:29:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.849859193, 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:29:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.849753201, 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:29:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:29:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.847961489999999, 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:29:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.847913989, 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.847674738, 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:29:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:30 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.847519159000001, 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126666, 2024-03-16T19:29:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362824+0000", 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7999999999999995e-07 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362828+0000", 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0879999999999997e-06 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362837+0000", 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.704e-06 2024-03-16T19:29:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362843+0000", 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9390000000000002e-06 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362877+0000", 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4779e-05 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362883+0000", 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9009999999999999e-06 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362950+0000", 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6674999999999994e-05 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.839176592999999, 2024-03-16T19:29:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.837836717, 2024-03-16T19:29:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:29:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:29:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.272 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:47.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:29:47.274 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:47.293+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:47.374 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:47.393+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:47.422 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:47.422 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:47.423 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:29:47.423 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:47.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:29:47.475 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:47.494+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:47.538 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:29:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:29:47.539 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:47.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:29:47.576 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:47.595+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:47.676 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:47.696+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:47.696 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.357868192000002, 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053568110000000004, 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357146+0000", 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5860000000000001e-06 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357154+0000", 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5439999999999999e-06 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357217+0000", 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2986999999999999e-05 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357229+0000", 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1853e-05 2024-03-16T19:29:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357343+0000", 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113663 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357504+0000", 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016176 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:29:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013908599999999999 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360705+0000", 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030619419999999998 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362409+0000", 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001704021 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362456+0000", 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6140000000000002e-05 2024-03-16T19:29:47.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362496+0000", 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0228999999999997e-05 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.357848165, 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078702890000000008, 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.702 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:29:47.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4299999999999998e-07 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357163+0000", 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2559999999999998e-06 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357254+0000", 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1701999999999997e-05 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357258+0000", 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5719999999999999e-06 2024-03-16T19:29:47.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357931+0000", 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067308000000000005 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357999+0000", 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8323000000000003e-05 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358113+0000", 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011366699999999999 2024-03-16T19:29:47.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362505+0000", 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004391509 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024372970000000002 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364992+0000", 2024-03-16T19:29:47.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9741999999999999e-05 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365029+0000", 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7697999999999997e-05 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:29:47.707 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.357841362999999, 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079944349999999994, 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:29:47.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6e-07 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357172+0000", 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7960000000000001e-06 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357285+0000", 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113183 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357290+0000", 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6190000000000002e-06 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358306+0000", 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010160670000000001 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358400+0000", 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3854e-05 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358492+0000", 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2954000000000005e-05 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362510+0000", 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040175469999999998 2024-03-16T19:29:47.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025680310000000001 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365123+0000", 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4515000000000003e-05 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365160+0000", 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7608999999999999e-05 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.357834467, 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080787759999999993, 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.713 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.36e-07 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1270000000000001e-06 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357297+0000", 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121885 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357300+0000", 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049999999999999e-06 2024-03-16T19:29:47.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358709+0000", 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001409083 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358776+0000", 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6489999999999995e-05 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358885+0000", 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109271 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362514+0000", 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036292199999999998 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365197+0000", 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026832100000000001 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365226+0000", 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9136999999999999e-05 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365251+0000", 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5012e-05 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.357831482999998, 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081787609999999997, 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.718 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357177+0000", 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5090000000000001e-06 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357306+0000", 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012856900000000001 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357308+0000", 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3700000000000002e-06 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359071+0000", 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017631210000000001 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359182+0000", 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110265 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359280+0000", 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8595e-05 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362520+0000", 2024-03-16T19:29:47.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032394120000000001 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365297+0000", 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002777542 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365328+0000", 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0899999999999999e-05 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365354+0000", 2024-03-16T19:29:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6276000000000001e-05 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.357828795, 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083197830000000007, 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357179+0000", 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357187+0000", 2024-03-16T19:29:47.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357314+0000", 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126978 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357316+0000", 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3800000000000001e-06 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359512+0000", 2024-03-16T19:29:47.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021950799999999999 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359585+0000", 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3168000000000001e-05 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359702+0000", 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011741 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362525+0000", 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028225419999999999 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365408+0000", 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028829010000000002 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365460+0000", 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839e-05 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365498+0000", 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7673999999999999e-05 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.357817669999999, 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084493169999999996, 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.728 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357190+0000", 2024-03-16T19:29:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-07 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357191+0000", 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7069999999999999e-06 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357325+0000", 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133585 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357330+0000", 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0799999999999996e-06 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359915+0000", 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025849290000000001 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9741999999999995e-05 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360121+0000", 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362826+0000", 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027046819999999999 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365564+0000", 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027387179999999998 2024-03-16T19:29:47.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365606+0000", 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1715999999999998e-05 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365639+0000", 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660999999999998e-05 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.357815000999999, 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085533069999999996, 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0699999999999999e-07 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3030000000000001e-06 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357339+0000", 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014513199999999999 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000003e-06 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030087619999999999 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360414+0000", 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1297999999999995e-05 2024-03-16T19:29:47.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360498+0000", 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4538999999999998e-05 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362832+0000", 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023338410000000001 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365670+0000", 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028379550000000001 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365724+0000", 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3533999999999999e-05 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365745+0000", 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1725999999999998e-05 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.739 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.357811776999998, 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052850789999999998, 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357196+0000", 2024-03-16T19:29:47.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.173e-06 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357359+0000", 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016222400000000001 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357365+0000", 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7320000000000003e-06 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357451+0000", 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6494000000000002e-05 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357555+0000", 2024-03-16T19:29:47.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010401200000000001 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8224000000000001e-05 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362294+0000", 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046502399999999999 2024-03-16T19:29:47.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362388+0000", 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4448000000000001e-05 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362433+0000", 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4888000000000001e-05 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362480+0000", 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7441000000000002e-05 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.35780974, 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053762239999999998, 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.746 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357198+0000", 2024-03-16T19:29:47.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357199+0000", 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357415+0000", 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021638699999999999 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0069e-05 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358132+0000", 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070650599999999997 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358207+0000", 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5197999999999994e-05 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358342+0000", 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013503199999999999 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362377+0000", 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040354600000000003 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362493+0000", 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115951 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362536+0000", 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3189999999999998e-05 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362574+0000", 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7113999999999997e-05 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:6 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:29:47.751 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.354750839000001, 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045449999999999999, 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360257+0000", 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7400000000000002e-07 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360260+0000", 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3550000000000001e-06 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360270+0000", 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360277+0000", 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5760000000000002e-06 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360697+0000", 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042039200000000001 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360702+0000", 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7140000000000001e-06 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360711+0000", 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1209999999999996e-06 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:22 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.352806663999999, 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100673, 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362201+0000", 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5799999999999998e-07 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362206+0000", 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8579999999999999e-06 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362219+0000", 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.239e-05 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362226+0000", 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2710000000000003e-06 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362281+0000", 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4756000000000002e-05 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362288+0000", 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.075e-06 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362301+0000", 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3365e-05 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.758 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:23 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.352695395000001, 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017205200000000001, 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4499999999999997e-07 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362315+0000", 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1350000000000001e-06 2024-03-16T19:29:47.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362322+0000", 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9469999999999997e-06 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362328+0000", 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362366+0000", 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8160000000000001e-05 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362373+0000", 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5799999999999997e-06 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362484+0000", 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111173 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:7 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.350151523000001, 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018177499999999999, 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364861+0000", 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3699999999999997e-06 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364875+0000", 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4059e-05 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364881+0000", 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5270000000000004e-06 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364929+0000", 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000002e-05 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:29:47.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3699999999999997e-06 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365037+0000", 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100819 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:8 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:29:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.350094466000002, 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025451199999999998, 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364916+0000", 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8109999999999999e-06 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364924+0000", 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1419999999999993e-06 2024-03-16T19:29:47.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364930+0000", 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2260000000000002e-06 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365071+0000", 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014030100000000001 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365075+0000", 2024-03-16T19:29:47.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365167+0000", 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2196999999999998e-05 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:9 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.350068226000001, 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032082900000000001, 2024-03-16T19:29:47.770 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:47.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.22e-07 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3980000000000002e-06 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364947+0000", 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3090000000000003e-06 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364951+0000", 2024-03-16T19:29:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1309999999999999e-06 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365191+0000", 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023992600000000001 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365195+0000", 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.704e-06 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365260+0000", 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5139000000000001e-05 2024-03-16T19:29:47.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:10 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.350039840000001, 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039415499999999998, 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:29:47.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364968+0000", 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364970+0000", 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4650000000000001e-06 2024-03-16T19:29:47.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364977+0000", 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.443e-06 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364981+0000", 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2320000000000003e-06 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365289+0000", 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030856900000000002 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365294+0000", 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6920000000000001e-06 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365361+0000", 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7358000000000001e-05 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:11 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.350019339999999, 2024-03-16T19:29:47.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051803199999999998, 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9299999999999999e-07 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364993+0000", 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9080000000000003e-06 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4240000000000001e-06 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365002+0000", 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7570000000000002e-06 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365395+0000", 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039237199999999999 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365401+0000", 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9380000000000003e-06 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365506+0000", 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010433999999999999 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:12 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.349997336000001, 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063430500000000005, 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9500000000000001e-07 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365012+0000", 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7129999999999999e-06 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365015+0000", 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2629999999999999e-06 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365017+0000", 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1749999999999999e-06 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365544+0000", 2024-03-16T19:29:47.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052690099999999998 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365559+0000", 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4610000000000001e-05 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365644+0000", 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5447999999999999e-05 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:13 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.349986838, 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00073113500000000005, 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1000000000000001e-07 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365021+0000", 2024-03-16T19:29:47.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.23e-07 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365024+0000", 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7369999999999998e-06 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365026+0000", 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1289999999999999e-06 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365663+0000", 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063679600000000002 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365667+0000", 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.233e-06 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365751+0000", 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4307000000000004e-05 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.788 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:47.789 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:47.789 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:47.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:29:47.790 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:47.808+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:47.890 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:47.909+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:47.939 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:47.940 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:47.940 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:29:47.940 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:47.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:29:47.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:29:47.992 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:48.011+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:48.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.108+0000 7ffa9c95e640 1 Processor -- start 2024-03-16T19:29:48.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.108+0000 7ffa9c95e640 1 -- start start 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa9c95e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa98152920 0x7ffa981591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa9c95e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffa9815c390 0x7ffa9815c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa9c95e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa98159700 con 0x7ffa9806d860 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa9c95e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa980713c0 con 0x7ffa9815c390 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa9c95e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980c89b0 con 0x7ffa98152920 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa98152920 0x7ffa981591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa96d9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffa9815c390 0x7ffa9815c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa96d9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffa9815c390 0x7ffa9815c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:36336/0 (socket says 172.21.15.172:36336) 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa96d9d640 1 -- 172.21.15.172:0/1500393433 learned_addr learned my addr 172.21.15.172:0/1500393433 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1500393433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa98152920 0x7ffa981591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48350/0 (socket says 172.21.15.172:48350) 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.109+0000 7ffa95d9b640 1 --1- 172.21.15.172:0/1500393433 >> v1:172.21.15.174:6789/0 conn(0x7ffa9806d860 0x7ffa9806cc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:52084/0 (socket says 172.21.15.172:52084) 2024-03-16T19:29:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.110+0000 7ffa9659c640 1 -- 172.21.15.172:0/1500393433 >> v2:172.21.15.174:3300/0 conn(0x7ffa9815c390 msgr2=0x7ffa9815c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:48.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.110+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1500393433 >> v2:172.21.15.174:3300/0 conn(0x7ffa9815c390 0x7ffa9815c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.110+0000 7ffa9659c640 1 -- 172.21.15.172:0/1500393433 >> v1:172.21.15.174:6789/0 conn(0x7ffa9806d860 legacy=0x7ffa9806cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.110+0000 7ffa9659c640 1 -- 172.21.15.172:0/1500393433 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa98070f10 con 0x7ffa98152920 2024-03-16T19:29:48.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.110+0000 7ffa9559a640 1 -- 172.21.15.172:0/1500393433 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652534138 0 0) 0x7ffa98159700 con 0x7ffa9806d860 2024-03-16T19:29:48.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.110+0000 7ffa96d9d640 1 --2- 172.21.15.172:0/1500393433 >> v2:172.21.15.174:3300/0 conn(0x7ffa9815c390 0x7ffa9815c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:48.094 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:48.113+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.110+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1500393433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa98152920 0x7ffa981591c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffa880097a0 tx=0x7ffa8802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a3675a271f18b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9559a640 1 -- 172.21.15.172:0/1500393433 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88031830 con 0x7ffa98152920 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9559a640 1 -- 172.21.15.172:0/1500393433 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa88031990 con 0x7ffa98152920 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9559a640 1 -- 172.21.15.172:0/1500393433 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88031d20 con 0x7ffa98152920 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1500393433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa98152920 msgr2=0x7ffa981591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1500393433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa98152920 0x7ffa981591c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffa880097a0 tx=0x7ffa8802b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1500393433 shutdown_connections 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1500393433 >> v2:172.21.15.174:3300/0 conn(0x7ffa9815c390 0x7ffa9815c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1500393433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa98152920 0x7ffa981591c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1500393433 >> 172.21.15.172:0/1500393433 conn(0x7ffa980c8b30 msgr2=0x7ffa980c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1500393433 shutdown_connections 2024-03-16T19:29:48.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.111+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1500393433 wait complete. 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.113+0000 7ffa9c95e640 1 Processor -- start 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.113+0000 7ffa9c95e640 1 -- start start 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9c95e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa9806d860 0x7ffa98153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9c95e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa98152920 0x7ffa981543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9c95e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa9815c390 0x7ffa9806b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9c95e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981548e0 con 0x7ffa9806d860 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9c95e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981536c0 con 0x7ffa98152920 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9c95e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981577a0 con 0x7ffa9815c390 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa9806d860 0x7ffa98153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa9806d860 0x7ffa98153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:35018/0 (socket says 172.21.15.172:35018) 2024-03-16T19:29:48.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9659c640 1 -- 172.21.15.172:0/1777354255 learned_addr learned my addr 172.21.15.172:0/1777354255 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa95d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa98152920 0x7ffa981543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa96d9d640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa9815c390 0x7ffa9806b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9659c640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa9815c390 msgr2=0x7ffa9806b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa9815c390 0x7ffa9806b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9659c640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa98152920 msgr2=0x7ffa981543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa98152920 0x7ffa981543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9659c640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa90009b30 con 0x7ffa9806d860 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa9806d860 0x7ffa98153030 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffa88000c00 tx=0x7ffa880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88004230 con 0x7ffa9806d860 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa88004390 con 0x7ffa9806d860 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8803a880 con 0x7ffa9806d860 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.115+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa88009400 con 0x7ffa9806d860 2024-03-16T19:29:48.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.116+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa981ee1c0 con 0x7ffa9806d860 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.116+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ffa98072b50 con 0x7ffa9806d860 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.117+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ffa8803a9e0 con 0x7ffa9806d860 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.117+0000 7ffa86ffd640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffa7806b3b0 0x7ffa7806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.118+0000 7ffa95d9b640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffa7806b3b0 0x7ffa7806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.118+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5741+0+0 (secure 0 0 0) 0x7ffa880abf70 con 0x7ffa9806d860 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.118+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v24) v1 ==== 54+0+16 (secure 0 0 0) 0x7ffa880ae5b0 con 0x7ffa9806d860 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.118+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ffa981ee3b0 0x7ffa981ee790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.118+0000 7ffa95d9b640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffa7806b3b0 0x7ffa7806d870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffa980c9970 tx=0x7ffa8c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.118+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa98152d00 con 0x7ffa981ee3b0 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.118+0000 7ffa96d9d640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ffa981ee3b0 0x7ffa981ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.118+0000 7ffa96d9d640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ffa981ee3b0 0x7ffa981ee790 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.119+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffa9000f040 con 0x7ffa981ee3b0 2024-03-16T19:29:48.102 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:29:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:29:48.102 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:48.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:29:48.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.134+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa980681c0 con 0x7ffa981ee3b0 2024-03-16T19:29:48.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.136+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ffa980681c0 con 0x7ffa981ee3b0 2024-03-16T19:29:48.118 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:29:48.119 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:29:48.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.136+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ffa980c4f40 0x7ffa981511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.136+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa98159230 con 0x7ffa980c4f40 2024-03-16T19:29:48.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.136+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ffa980c4f40 0x7ffa981511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.136+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ffa980c4f40 0x7ffa981511c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.137+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffa98159230 con 0x7ffa980c4f40 2024-03-16T19:29:48.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.152+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa98158f50 con 0x7ffa980c4f40 2024-03-16T19:29:48.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.154+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ffa98158f50 con 0x7ffa980c4f40 2024-03-16T19:29:48.136 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:29:48.137 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:29:48.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.154+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ffa981f1850 0x7ffa981f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.154+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa981f2170 con 0x7ffa981f1850 2024-03-16T19:29:48.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.154+0000 7ffa95d9b640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ffa981f1850 0x7ffa981f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.155+0000 7ffa95d9b640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ffa981f1850 0x7ffa981f1c30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.155+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffa981f2170 con 0x7ffa981f1850 2024-03-16T19:29:48.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.170+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa980c5350 con 0x7ffa981f1850 2024-03-16T19:29:48.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.171+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ffa980c5350 con 0x7ffa981f1850 2024-03-16T19:29:48.153 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:29:48.154 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:29:48.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.171+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ffa98062200 0x7ffa980ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.171+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa98151d10 con 0x7ffa98062200 2024-03-16T19:29:48.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.172+0000 7ffa96d9d640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ffa98062200 0x7ffa980ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.172+0000 7ffa96d9d640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ffa98062200 0x7ffa980ca380 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.172+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffa98151d10 con 0x7ffa98062200 2024-03-16T19:29:48.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.187+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa98062850 con 0x7ffa98062200 2024-03-16T19:29:48.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.188+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ffa98062850 con 0x7ffa98062200 2024-03-16T19:29:48.170 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:29:48.171 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:29:48.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.189+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ffa980cadd0 0x7ffa9802bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.189+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa980cb260 con 0x7ffa980cadd0 2024-03-16T19:29:48.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.189+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ffa980cadd0 0x7ffa9802bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.189+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ffa980cadd0 0x7ffa9802bde0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.190+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffa980cb260 con 0x7ffa980cadd0 2024-03-16T19:29:48.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.204+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa980cb260 con 0x7ffa980cadd0 2024-03-16T19:29:48.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.206+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ffa980cb260 con 0x7ffa980cadd0 2024-03-16T19:29:48.188 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:29:48.188 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:29:48.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.206+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ffa9802c830 0x7ffa9802ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.206+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa9802d270 con 0x7ffa9802c830 2024-03-16T19:29:48.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.206+0000 7ffa95d9b640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ffa9802c830 0x7ffa9802ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.207+0000 7ffa95d9b640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ffa9802c830 0x7ffa9802ccb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.207+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffa9802d270 con 0x7ffa9802c830 2024-03-16T19:29:48.194 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:48.214+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:48.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.222+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa98204750 con 0x7ffa9802c830 2024-03-16T19:29:48.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.224+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ffa98204750 con 0x7ffa9802c830 2024-03-16T19:29:48.206 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:29:48.206 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:29:48.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.224+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ffa98204e30 0x7ffa98205270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.224+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa9820a390 con 0x7ffa98204e30 2024-03-16T19:29:48.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.224+0000 7ffa96d9d640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ffa98204e30 0x7ffa98205270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.225+0000 7ffa96d9d640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ffa98204e30 0x7ffa98205270 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.225+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffa9820a390 con 0x7ffa98204e30 2024-03-16T19:29:48.220 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:48.220 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:48.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.239+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa982058c0 con 0x7ffa98204e30 2024-03-16T19:29:48.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.241+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ffa982058c0 con 0x7ffa98204e30 2024-03-16T19:29:48.223 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:29:48.223 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:29:48.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.241+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ffa98205dd0 0x7ffa98206230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.241+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa98206770 con 0x7ffa98205dd0 2024-03-16T19:29:48.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.241+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ffa98205dd0 0x7ffa98206230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.242+0000 7ffa9659c640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ffa98205dd0 0x7ffa98206230 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.229 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:29:48.295 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:48.314+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:48.337 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:48.337 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:48.346 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:29:48.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.421+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffa98206770 con 0x7ffa98205dd0 2024-03-16T19:29:48.405 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:29:48.405 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:29:48.405 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-16T19:29:48.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-03-16T19:29:48.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:29:48.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.437+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa98206770 con 0x7ffa98205dd0 2024-03-16T19:29:48.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.438+0000 7ffa86ffd640 1 -- 172.21.15.172:0/1777354255 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ffa98206770 con 0x7ffa98205dd0 2024-03-16T19:29:48.420 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:29:48.422 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:29:48.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ffa981ee3b0 msgr2=0x7ffa981ee790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ffa981ee3b0 0x7ffa981ee790 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ffa980c4f40 msgr2=0x7ffa981511c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ffa980c4f40 0x7ffa981511c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ffa981f1850 msgr2=0x7ffa981f1c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7ffa981f1850 0x7ffa981f1c30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ffa98062200 msgr2=0x7ffa980ca380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ffa98062200 0x7ffa980ca380 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ffa980cadd0 msgr2=0x7ffa9802bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ffa980cadd0 0x7ffa9802bde0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ffa9802c830 msgr2=0x7ffa9802ccb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ffa9802c830 0x7ffa9802ccb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ffa98204e30 msgr2=0x7ffa98205270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ffa98204e30 0x7ffa98205270 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ffa98205dd0 msgr2=0x7ffa98206230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ffa98205dd0 0x7ffa98206230 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffa7806b3b0 msgr2=0x7ffa7806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffa7806b3b0 0x7ffa7806d870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffa980c9970 tx=0x7ffa8c007de0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa96d9d640 1 -- 172.21.15.172:0/1777354255 reap_dead start 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa9806d860 msgr2=0x7ffa98153030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 --2- 172.21.15.172:0/1777354255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa9806d860 0x7ffa98153030 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffa88000c00 tx=0x7ffa880046c0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa96d9d640 1 -- 172.21.15.172:0/1777354255 reap_dead start 2024-03-16T19:29:48.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa96d9d640 1 -- 172.21.15.172:0/1777354255 reap_dead start 2024-03-16T19:29:48.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa96d9d640 1 -- 172.21.15.172:0/1777354255 reap_dead start 2024-03-16T19:29:48.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa96d9d640 1 -- 172.21.15.172:0/1777354255 reap_dead start 2024-03-16T19:29:48.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 shutdown_connections 2024-03-16T19:29:48.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 >> 172.21.15.172:0/1777354255 conn(0x7ffa980c8b30 msgr2=0x7ffa980c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:48.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 shutdown_connections 2024-03-16T19:29:48.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.439+0000 7ffa9c95e640 1 -- 172.21.15.172:0/1777354255 wait complete. 2024-03-16T19:29:48.429 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:48.449+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:48.494 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:48.494 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:48.496 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:48.516+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:48.503 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:29:48.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.529+0000 7fd75d26f640 1 Processor -- start 2024-03-16T19:29:48.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.529+0000 7fd75d26f640 1 -- start start 2024-03-16T19:29:48.511 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.529+0000 7f52bc53a640 1 Processor -- start 2024-03-16T19:29:48.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.529+0000 7fd75d26f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7581517e0 0x7fd7580ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.529+0000 7fd75d26f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd758073030 0x7fd75815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.529+0000 7fd75d26f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75815ae30 con 0x7fd75806be10 2024-03-16T19:29:48.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.529+0000 7fd75d26f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75806b560 con 0x7fd758073030 2024-03-16T19:29:48.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.529+0000 7fd75d26f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd758072cf0 con 0x7fd7581517e0 2024-03-16T19:29:48.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.529+0000 7f52bc53a640 1 -- start start 2024-03-16T19:29:48.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd7567fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd75806be10 0x7fd75815dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:49974/0 (socket says 172.21.15.172:49974) 2024-03-16T19:29:48.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd7567fc640 1 -- 172.21.15.172:0/3213236602 learned_addr learned my addr 172.21.15.172:0/3213236602 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:48.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd756ffd640 1 --2- 172.21.15.172:0/3213236602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7581517e0 0x7fd7580ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd7577fe640 1 --2- 172.21.15.172:0/3213236602 >> v2:172.21.15.174:3300/0 conn(0x7fd758073030 0x7fd75815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd755ffb640 1 -- 172.21.15.172:0/3213236602 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156471982 0 0) 0x7fd75815ae30 con 0x7fd75806be10 2024-03-16T19:29:48.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd755ffb640 1 -- 172.21.15.172:0/3213236602 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd740003520 con 0x7fd75806be10 2024-03-16T19:29:48.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd756ffd640 1 -- 172.21.15.172:0/3213236602 >> v2:172.21.15.174:3300/0 conn(0x7fd758073030 msgr2=0x7fd75815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:48.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd756ffd640 1 --2- 172.21.15.172:0/3213236602 >> v2:172.21.15.174:3300/0 conn(0x7fd758073030 0x7fd75815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd756ffd640 1 -- 172.21.15.172:0/3213236602 >> v1:172.21.15.172:6789/0 conn(0x7fd75806be10 legacy=0x7fd75815dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd756ffd640 1 -- 172.21.15.172:0/3213236602 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75806c300 con 0x7fd7581517e0 2024-03-16T19:29:48.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.530+0000 7fd7577fe640 1 --2- 172.21.15.172:0/3213236602 >> v2:172.21.15.174:3300/0 conn(0x7fd758073030 0x7fd75815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52bc53a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f52b4158f60 0x7f52b40ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52bc53a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f52b40720f0 0x7f52b40cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52bc53a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b406cf50 con 0x7f52b40c3f30 2024-03-16T19:29:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52bc53a640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b406b090 con 0x7f52b4158f60 2024-03-16T19:29:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52bc53a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b406c990 con 0x7f52b40720f0 2024-03-16T19:29:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52ba2d6640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f52b4158f60 0x7f52b40ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52b9ad5640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f52b40c3f30 0x7f52b40719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:49976/0 (socket says 172.21.15.172:49976) 2024-03-16T19:29:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52b9ad5640 1 -- 172.21.15.172:0/329884592 learned_addr learned my addr 172.21.15.172:0/329884592 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52baad7640 1 --2- 172.21.15.172:0/329884592 >> v2:172.21.15.174:3300/0 conn(0x7f52b40720f0 0x7f52b40cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7fd756ffd640 1 --2- 172.21.15.172:0/3213236602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7581517e0 0x7fd7580ae8a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0097a0 tx=0x7fd74c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfe4d7cd5298000d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52ba2d6640 1 -- 172.21.15.172:0/329884592 >> v2:172.21.15.174:3300/0 conn(0x7f52b40720f0 msgr2=0x7f52b40cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:48.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52ba2d6640 1 --2- 172.21.15.172:0/329884592 >> v2:172.21.15.174:3300/0 conn(0x7f52b40720f0 0x7f52b40cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52ba2d6640 1 -- 172.21.15.172:0/329884592 >> v1:172.21.15.172:6789/0 conn(0x7f52b40c3f30 legacy=0x7f52b40719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7f52ba2d6640 1 -- 172.21.15.172:0/329884592 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b406cc70 con 0x7f52b4158f60 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7fd755ffb640 1 -- 172.21.15.172:0/3213236602 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c030610 con 0x7fd7581517e0 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7fd755ffb640 1 -- 172.21.15.172:0/3213236602 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c030770 con 0x7fd7581517e0 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.531+0000 7fd755ffb640 1 -- 172.21.15.172:0/3213236602 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c030b00 con 0x7fd7581517e0 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7fd75d26f640 1 -- 172.21.15.172:0/3213236602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7581517e0 msgr2=0x7fd7580ae8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7fd75d26f640 1 --2- 172.21.15.172:0/3213236602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7581517e0 0x7fd7580ae8a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0097a0 tx=0x7fd74c02b960 comp rx=0 tx=0).stop 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7fd75d26f640 1 -- 172.21.15.172:0/3213236602 shutdown_connections 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7fd75d26f640 1 --2- 172.21.15.172:0/3213236602 >> v2:172.21.15.174:3300/0 conn(0x7fd758073030 0x7fd75815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7fd75d26f640 1 --2- 172.21.15.172:0/3213236602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7581517e0 0x7fd7580ae8a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7fd75d26f640 1 -- 172.21.15.172:0/3213236602 >> 172.21.15.172:0/3213236602 conn(0x7fd75806f6f0 msgr2=0x7fd75806fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7fd75d26f640 1 -- 172.21.15.172:0/3213236602 shutdown_connections 2024-03-16T19:29:48.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7fd75d26f640 1 -- 172.21.15.172:0/3213236602 wait complete. 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7f52ba2d6640 1 --2- 172.21.15.172:0/329884592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b4158f60 0x7f52b40ccf60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f52b40c3cd0 tx=0x7f52ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a599e83fa33e39 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7f52b92d4640 1 -- 172.21.15.172:0/329884592 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac02cc80 con 0x7f52b4158f60 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7f52b92d4640 1 -- 172.21.15.172:0/329884592 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac02cde0 con 0x7f52b4158f60 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.532+0000 7f52b92d4640 1 -- 172.21.15.172:0/329884592 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac034660 con 0x7f52b4158f60 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7f52bc53a640 1 -- 172.21.15.172:0/329884592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b4158f60 msgr2=0x7f52b40ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7f52bc53a640 1 --2- 172.21.15.172:0/329884592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b4158f60 0x7f52b40ccf60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f52b40c3cd0 tx=0x7f52ac02bed0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7f52bc53a640 1 -- 172.21.15.172:0/329884592 shutdown_connections 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7f52bc53a640 1 --2- 172.21.15.172:0/329884592 >> v2:172.21.15.174:3300/0 conn(0x7f52b40720f0 0x7f52b40cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7f52bc53a640 1 --2- 172.21.15.172:0/329884592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b4158f60 0x7f52b40ccf60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7f52bc53a640 1 -- 172.21.15.172:0/329884592 >> 172.21.15.172:0/329884592 conn(0x7f52b406db70 msgr2=0x7f52b406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7f52bc53a640 1 -- 172.21.15.172:0/329884592 shutdown_connections 2024-03-16T19:29:48.520 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7f52bc53a640 1 -- 172.21.15.172:0/329884592 wait complete. 2024-03-16T19:29:48.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7fd75d26f640 1 Processor -- start 2024-03-16T19:29:48.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.533+0000 7fd75d26f640 1 -- start start 2024-03-16T19:29:48.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.534+0000 7f52bc53a640 1 Processor -- start 2024-03-16T19:29:48.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.534+0000 7f52bc53a640 1 -- start start 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd75d26f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd75806be10 0x7fd75814bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd75d26f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd758073030 0x7fd7581f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd75d26f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7581517e0 0x7fd758153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd75d26f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75814d6e0 con 0x7fd7581517e0 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd75d26f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75814c090 con 0x7fd75806be10 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd75d26f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75814cdc0 con 0x7fd758073030 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd7577fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7581517e0 0x7fd758153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd756ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd75806be10 0x7fd75814bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd7577fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7581517e0 0x7fd758153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54102/0 (socket says 172.21.15.172:54102) 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd7567fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd758073030 0x7fd7581f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd7577fe640 1 -- 172.21.15.172:0/879291079 learned_addr learned my addr 172.21.15.172:0/879291079 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:48.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd756ffd640 1 -- 172.21.15.172:0/879291079 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd758073030 msgr2=0x7fd7581f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.524 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd756ffd640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd758073030 0x7fd7581f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.524 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd756ffd640 1 -- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7581517e0 msgr2=0x7fd758153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.524 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd756ffd640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7581517e0 0x7fd758153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.524 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd756ffd640 1 -- 172.21.15.172:0/879291079 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd750009a70 con 0x7fd75806be10 2024-03-16T19:29:48.524 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd7577fe640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7581517e0 0x7fd758153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:48.524 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd756ffd640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd75806be10 0x7fd75814bb50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd74c033000 tx=0x7fd74c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.524 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7f52bc53a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b40720f0 0x7f52b40cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7f52bc53a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b40c3f30 0x7f52b406b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7f52bc53a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52b4158f60 0x7f52b4147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7f52bc53a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b414c2d0 con 0x7f52b40720f0 2024-03-16T19:29:48.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7f52bc53a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b41f7b20 con 0x7f52b40c3f30 2024-03-16T19:29:48.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7f52bc53a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b415c660 con 0x7f52b4158f60 2024-03-16T19:29:48.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd73f7fe640 1 -- 172.21.15.172:0/879291079 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c03f070 con 0x7fd75806be10 2024-03-16T19:29:48.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd73f7fe640 1 -- 172.21.15.172:0/879291079 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c03a5a0 con 0x7fd75806be10 2024-03-16T19:29:48.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7f52ba2d6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b40720f0 0x7f52b40cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c009400 con 0x7fd75806be10 2024-03-16T19:29:48.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7fd73f7fe640 1 -- 172.21.15.172:0/879291079 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c03a990 con 0x7fd75806be10 2024-03-16T19:29:48.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7f52ba2d6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b40720f0 0x7f52b40cc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54114/0 (socket says 172.21.15.172:54114) 2024-03-16T19:29:48.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75814d400 con 0x7fd75806be10 2024-03-16T19:29:48.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.535+0000 7f52ba2d6640 1 -- 172.21.15.172:0/3537572303 learned_addr learned my addr 172.21.15.172:0/3537572303 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52baad7640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52b4158f60 0x7f52b4147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52b9ad5640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b40c3f30 0x7f52b406b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52ba2d6640 1 -- 172.21.15.172:0/3537572303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52b4158f60 msgr2=0x7f52b4147fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52ba2d6640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52b4158f60 0x7f52b4147fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52ba2d6640 1 -- 172.21.15.172:0/3537572303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b40c3f30 msgr2=0x7f52b406b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52ba2d6640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b40c3f30 0x7f52b406b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52ba2d6640 1 -- 172.21.15.172:0/3537572303 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac009a70 con 0x7f52b40720f0 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52ba2d6640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b40720f0 0x7f52b40cc320 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f52ac037040 tx=0x7f52ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52aaffd640 1 -- 172.21.15.172:0/3537572303 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac040650 con 0x7f52b40720f0 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b415a820 con 0x7f52b40720f0 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52aaffd640 1 -- 172.21.15.172:0/3537572303 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac0407b0 con 0x7f52b40720f0 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52aaffd640 1 -- 172.21.15.172:0/3537572303 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac040b60 con 0x7f52b40720f0 2024-03-16T19:29:48.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.536+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b41f8050 con 0x7f52b40720f0 2024-03-16T19:29:48.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.537+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7580688d0 con 0x7fd75806be10 2024-03-16T19:29:48.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.537+0000 7fd73f7fe640 1 -- 172.21.15.172:0/879291079 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd74c043070 con 0x7fd75806be10 2024-03-16T19:29:48.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.538+0000 7fd73f7fe640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd74406b3e0 0x7fd74406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.538+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b4075a70 con 0x7f52b40720f0 2024-03-16T19:29:48.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.538+0000 7fd73f7fe640 1 -- 172.21.15.172:0/879291079 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5741+0+0 (secure 0 0 0) 0x7fd74c0ab390 con 0x7fd75806be10 2024-03-16T19:29:48.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.538+0000 7fd7567fc640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd74406b3e0 0x7fd74406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.538+0000 7fd7567fc640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd74406b3e0 0x7fd74406d8a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7581478f0 tx=0x7fd748009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.538+0000 7f52aaffd640 1 -- 172.21.15.172:0/3537572303 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f52ac040cc0 con 0x7f52b40720f0 2024-03-16T19:29:48.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.539+0000 7f52aaffd640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f529806b3b0 0x7f529806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:48.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.539+0000 7f52b9ad5640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f529806b3b0 0x7f529806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:48.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.539+0000 7f52aaffd640 1 -- 172.21.15.172:0/3537572303 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f52ac0ac860 con 0x7f52b40720f0 2024-03-16T19:29:48.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.539+0000 7f52b9ad5640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f529806b3b0 0x7f529806d870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f52b40c3060 tx=0x7f52a4007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:48.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.542+0000 7fd73f7fe640 1 -- 172.21.15.172:0/879291079 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd74c076110 con 0x7fd75806be10 2024-03-16T19:29:48.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.543+0000 7f52aaffd640 1 -- 172.21.15.172:0/3537572303 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52ac077590 con 0x7f52b40720f0 2024-03-16T19:29:48.597 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:48.617+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:48.651 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:48.651 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:48.660 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:29:48.700 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:48.720+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:48.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.721+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd75814c1d0 con 0x7fd75806be10 2024-03-16T19:29:48.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.723+0000 7fd73f7fe640 1 -- 172.21.15.172:0/879291079 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v24) v1 ==== 81+0+34395 (secure 0 0 0) 0x7fd74c0797d0 con 0x7fd75806be10 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 24, 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "fsid": "6c00e600-a879-4eb6-acee-0fdf0f76922a", 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "created": "2024-03-16T19:29:02.010581+0000", 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "modified": "2024-03-16T19:29:46.406158+0000", 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "last_up_change": "2024-03-16T19:29:13.180924+0000", 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "last_in_change": "2024-03-16T19:29:04.178546+0000", 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-16T19:29:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_num": 5806080, 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_set": [ 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "nodeep-scrub", 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "noscrub", 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "pglog_hardlimit", 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "purged_snapdirs", 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_deletes", 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "sortbitwise" 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_version": 4, 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "full_ratio": 0.94999998807907104, 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_snapshot": "", 2024-03-16T19:29:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_max": 3, 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "max_osd": 8, 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "require_min_compat_client": "luminous", 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "min_compat_client": "jewel", 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "require_osd_release": "squid", 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "allow_crimson": false, 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "pools": [ 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "pool": 1, 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_name": ".mgr", 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "create_time": "2024-03-16T19:29:14.242742+0000", 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": 1, 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_names": "hashpspool", 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "type": 1, 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 2, 2024-03-16T19:29:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "min_size": 1, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_rule": 0, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_count": 0, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_target": 0, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_barrier": 0, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "object_hash": 2, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_mode": "off", 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num": 1, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num": 1, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num_target": 1, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_target": 1, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_pending": 1, 2024-03-16T19:29:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "last_pg_merge_meta": { 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "source_pgid": "0.0", 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "ready_epoch": 0, 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_started": 0, 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_clean": 0, 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "source_version": "0'0", 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "target_version": "0'0" 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "last_change": "17", 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend": "0", 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_preluminous": "0", 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "auid": 0, 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_mode": "selfmanaged", 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_seq": 0, 2024-03-16T19:29:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_epoch": 0, 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_snaps": [], 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps": "[]", 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_bytes": 0, 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_objects": 0, 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "tiers": [], 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_of": -1, 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "read_tier": -1, 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "write_tier": -1, 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_mode": "none", 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_bytes": 0, 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_objects": 0, 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-16T19:29:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_flush_age": 0, 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_evict_age": 0, 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profile": "", 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_params": { 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "none" 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_period": 0, 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_count": 0, 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "use_gmt_hitset": true, 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "min_read_recency_for_promote": 0, 2024-03-16T19:29:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "min_write_recency_for_promote": 0, 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_grade_decay_rate": 0, 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_search_last_n": 0, 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "grade_table": [], 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "stripe_width": 0, 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "expected_num_objects": 0, 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "fast_read": false, 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "options": { 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_max": 32, 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_min": 1 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "application_metadata": { 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": {} 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "read_balance": { 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "score_type": "Fair distribution", 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "score_acting": 6.9000000953674316, 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "score_stable": 6.9000000953674316, 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "optimal_score": 0.28999999165534973, 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_acting": 2, 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_stable": 2, 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity_weighted": 0.875, 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity": 1, 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "pool": 2, 2024-03-16T19:29:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_name": "cephfs_metadata", 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "create_time": "2024-03-16T19:29:24.121132+0000", 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": 1, 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_names": "hashpspool", 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "type": 1, 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 2, 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "min_size": 1, 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_rule": 0, 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_count": 0, 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_target": 0, 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_barrier": 0, 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-16T19:29:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "object_hash": 2, 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_mode": "off", 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num": 32, 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num": 32, 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num_target": 32, 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_target": 32, 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_pending": 32, 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "last_pg_merge_meta": { 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "source_pgid": "0.0", 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "ready_epoch": 0, 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_started": 0, 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_clean": 0, 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "source_version": "0'0", 2024-03-16T19:29:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "target_version": "0'0" 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "last_change": "20", 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend": "0", 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_preluminous": "0", 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "auid": 0, 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_mode": "selfmanaged", 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_seq": 0, 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_epoch": 0, 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_snaps": [], 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps": "[]", 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_bytes": 0, 2024-03-16T19:29:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_objects": 0, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "tiers": [], 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_of": -1, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "read_tier": -1, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "write_tier": -1, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_mode": "none", 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_bytes": 0, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_objects": 0, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_flush_age": 0, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_evict_age": 0, 2024-03-16T19:29:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profile": "", 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_params": { 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "none" 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_period": 0, 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_count": 0, 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "use_gmt_hitset": true, 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "min_read_recency_for_promote": 0, 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "min_write_recency_for_promote": 0, 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_grade_decay_rate": 0, 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_search_last_n": 0, 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "grade_table": [], 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "stripe_width": 0, 2024-03-16T19:29:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "expected_num_objects": 0, 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "fast_read": false, 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "options": { 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_bias": 4, 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_min": 16, 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_priority": 5 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "application_metadata": { 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs": { 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "metadata": "cephfs" 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "read_balance": { 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "score_type": "Fair distribution", 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "score_acting": 1.5399999618530273, 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "score_stable": 1.5399999618530273, 2024-03-16T19:29:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "optimal_score": 1.1399999856948853, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_acting": 1.75, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_stable": 1.75, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity_weighted": 0.875, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity": 1, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "pool": 3, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_name": "cephfs_data", 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "create_time": "2024-03-16T19:29:24.651099+0000", 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": 1, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_names": "hashpspool", 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "type": 1, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 2, 2024-03-16T19:29:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "min_size": 1, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_rule": 0, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_count": 0, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_target": 0, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_barrier": 0, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "object_hash": 2, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_mode": "off", 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num": 64, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num": 64, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num_target": 64, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_target": 64, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_pending": 64, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "last_pg_merge_meta": { 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "source_pgid": "0.0", 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "ready_epoch": 0, 2024-03-16T19:29:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_started": 0, 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_clean": 0, 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "source_version": "0'0", 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "target_version": "0'0" 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "last_change": "21", 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend": "0", 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_preluminous": "0", 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "auid": 0, 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_mode": "selfmanaged", 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_seq": 0, 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_epoch": 0, 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_snaps": [], 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps": "[]", 2024-03-16T19:29:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_bytes": 0, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_objects": 0, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "tiers": [], 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_of": -1, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "read_tier": -1, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "write_tier": -1, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_mode": "none", 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_bytes": 0, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_objects": 0, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_flush_age": 0, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_evict_age": 0, 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profile": "", 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_params": { 2024-03-16T19:29:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "none" 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_period": 0, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_count": 0, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "use_gmt_hitset": true, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "min_read_recency_for_promote": 0, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "min_write_recency_for_promote": 0, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_grade_decay_rate": 0, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_search_last_n": 0, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "grade_table": [], 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "stripe_width": 0, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "expected_num_objects": 0, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "fast_read": false, 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "options": { 2024-03-16T19:29:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_min": 64, 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "target_size_ratio": 0.89999997615814209 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "application_metadata": { 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs": { 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "data": "cephfs" 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "read_balance": { 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "score_type": "Fair distribution", 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "score_acting": 1.5399999618530273, 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "score_stable": 1.5399999618530273, 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "optimal_score": 1.1399999856948853, 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_acting": 1.75, 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_stable": 1.75, 2024-03-16T19:29:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity_weighted": 0.875, 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity": 1, 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "osds": [ 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 0, 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "460ed9e4-eacf-4544-9199-23ca7208b663", 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 0, 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 19, 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6800", 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6801", 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6802", 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6803", 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6806", 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6807", 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6804", 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6805", 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6801/491754535", 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6803/491754535", 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6807/491754535", 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6805/491754535", 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 1, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "b84ebb50-079d-460f-bf05-71040f17500a", 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6808", 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6809", 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6810", 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6811", 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6814", 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6815", 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6812", 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6813", 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6809/3098410089", 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6811/3098410089", 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6815/3098410089", 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6813/3098410089", 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 2, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "6774c42c-5373-4543-8f8c-ed7b82c12057", 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6823", 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6825", 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6826", 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6827", 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6830", 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6831", 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6828", 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6829", 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6825/1599824697", 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6827/1599824697", 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6831/1599824697", 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6829/1599824697", 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 3, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "b96838ef-d991-43ab-b2d7-16deab58ca78", 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6816", 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6817", 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6818", 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6819", 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6822", 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6824", 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6820", 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6821", 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6817/1613555619", 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6819/1613555619", 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6824/1613555619", 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6821/1613555619", 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 4, 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "f5c78559-d22e-4847-be3f-2f597d7f6789", 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6808", 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6809", 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6810", 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6811", 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6814", 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6815", 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6812", 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6813", 2024-03-16T19:29:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6809/967866068", 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6811/967866068", 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6815/967866068", 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6813/967866068", 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 5, 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "e1f4797f-31c3-4292-a4a1-88fa0ab64f9a", 2024-03-16T19:29:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6816", 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6817", 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6818", 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6820", 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6827", 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6829", 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6822", 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6825", 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6817/3985312519", 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6820/3985312519", 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6829/3985312519", 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6825/3985312519", 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 6, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "ea3e888e-c4c6-444a-8ebe-e26cddb2084b", 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 19, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6819", 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6821", 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6823", 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6824", 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6830", 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6831", 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6826", 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6828", 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6821/3676139844", 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6824/3676139844", 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6831/3676139844", 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6828/3676139844", 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 7, 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "bb0252a5-3aa1-43f0-be38-37413819720b", 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6800", 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6801", 2024-03-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6802", 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6803", 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6806", 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6807", 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6804", 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6805", 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6801/339941177", 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6803/339941177", 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6807/339941177", 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6805/339941177", 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_xinfo": [ 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 0, 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.304258+0000", 2024-03-16T19:29:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 1, 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.336924+0000", 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 2, 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.359766+0000", 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 3, 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.384687+0000", 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 4, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.291288+0000", 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 5, 2024-03-16T19:29:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.318479+0000", 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 6, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.313564+0000", 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 7, 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.232618+0000", 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:29:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap": [], 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_items": [], 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_primaries": [], 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_temp": [], 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_temp": [], 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "blocklist": {}, 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "range_blocklist": {}, 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profiles": { 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "default": { 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "crush-failure-domain": "osd", 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "k": "2", 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "m": "1", 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "plugin": "jerasure", 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "technique": "reed_sol_van" 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps_queue": [], 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "new_removed_snaps": [], 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "new_purged_snaps": [], 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_node_flags": {}, 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "device_class_flags": {}, 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_mode": { 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_mode_enabled": false, 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_bucket_count": 0, 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "degraded_stretch_mode": 0, 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "recovering_stretch_mode": 0, 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_mode_bucket": 0 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:48.776 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:48.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd74406b3e0 msgr2=0x7fd74406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd74406b3e0 0x7fd74406d8a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7581478f0 tx=0x7fd748009040 comp rx=0 tx=0).stop 2024-03-16T19:29:48.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd75806be10 msgr2=0x7fd75814bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:48.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd75806be10 0x7fd75814bb50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd74c033000 tx=0x7fd74c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 shutdown_connections 2024-03-16T19:29:48.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7581517e0 0x7fd758153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd74406b3e0 0x7fd74406d8a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd758073030 0x7fd7581f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 --2- 172.21.15.172:0/879291079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd75806be10 0x7fd75814bb50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:48.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 >> 172.21.15.172:0/879291079 conn(0x7fd75806f6f0 msgr2=0x7fd7580cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:48.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 shutdown_connections 2024-03-16T19:29:48.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.725+0000 7fd75d26f640 1 -- 172.21.15.172:0/879291079 wait complete. 2024-03-16T19:29:48.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:48.732+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f52b4072560 con 0x7f52b40720f0 2024-03-16T19:29:48.779 INFO:tasks.thrashosds.thrasher:j is {'epoch': 24, 'fsid': '6c00e600-a879-4eb6-acee-0fdf0f76922a', 'created': '2024-03-16T19:29:02.010581+0000', 'modified': '2024-03-16T19:29:46.406158+0000', 'last_up_change': '2024-03-16T19:29:13.180924+0000', 'last_in_change': '2024-03-16T19:29:04.178546+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:29:14.242742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:29:24.121132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:29:24.651099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '460ed9e4-eacf-4544-9199-23ca7208b663', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 491754535}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 491754535}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 491754535}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 491754535}]}, 'public_addr': '172.21.15.172:6801/491754535', 'cluster_addr': '172.21.15.172:6803/491754535', 'heartbeat_back_addr': '172.21.15.172:6807/491754535', 'heartbeat_front_addr': '172.21.15.172:6805/491754535', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b84ebb50-079d-460f-bf05-71040f17500a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6809', 'nonce': 3098410089}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6810', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 3098410089}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6814', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 3098410089}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6813', 'nonce': 3098410089}]}, 'public_addr': '172.21.15.172:6809/3098410089', 'cluster_addr': '172.21.15.172:6811/3098410089', 'heartbeat_back_addr': '172.21.15.172:6815/3098410089', 'heartbeat_front_addr': '172.21.15.172:6813/3098410089', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6774c42c-5373-4543-8f8c-ed7b82c12057', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6823', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 1599824697}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 1599824697}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 1599824697}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 1599824697}]}, 'public_addr': '172.21.15.172:6825/1599824697', 'cluster_addr': '172.21.15.172:6827/1599824697', 'heartbeat_back_addr': '172.21.15.172:6831/1599824697', 'heartbeat_front_addr': '172.21.15.172:6829/1599824697', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'b96838ef-d991-43ab-b2d7-16deab58ca78', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6817', 'nonce': 1613555619}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6818', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6819', 'nonce': 1613555619}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6824', 'nonce': 1613555619}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6820', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 1613555619}]}, 'public_addr': '172.21.15.172:6817/1613555619', 'cluster_addr': '172.21.15.172:6819/1613555619', 'heartbeat_back_addr': '172.21.15.172:6824/1613555619', 'heartbeat_front_addr': '172.21.15.172:6821/1613555619', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f5c78559-d22e-4847-be3f-2f597d7f6789', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6808', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6809', 'nonce': 967866068}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6810', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6811', 'nonce': 967866068}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6814', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6815', 'nonce': 967866068}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6812', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6813', 'nonce': 967866068}]}, 'public_addr': '172.21.15.174:6809/967866068', 'cluster_addr': '172.21.15.174:6811/967866068', 'heartbeat_back_addr': '172.21.15.174:6815/967866068', 'heartbeat_front_addr': '172.21.15.174:6813/967866068', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e1f4797f-31c3-4292-a4a1-88fa0ab64f9a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6816', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6817', 'nonce': 3985312519}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6818', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6820', 'nonce': 3985312519}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6827', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6829', 'nonce': 3985312519}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6822', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6825', 'nonce': 3985312519}]}, 'public_addr': '172.21.15.174:6817/3985312519', 'cluster_addr': '172.21.15.174:6820/3985312519', 'heartbeat_back_addr': '172.21.15.174:6829/3985312519', 'heartbeat_front_addr': '172.21.15.174:6825/3985312519', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ea3e888e-c4c6-444a-8ebe-e26cddb2084b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6819', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6821', 'nonce': 3676139844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6823', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6824', 'nonce': 3676139844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6830', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6831', 'nonce': 3676139844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6826', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6828', 'nonce': 3676139844}]}, 'public_addr': '172.21.15.174:6821/3676139844', 'cluster_addr': '172.21.15.174:6824/3676139844', 'heartbeat_back_addr': '172.21.15.174:6831/3676139844', 'heartbeat_front_addr': '172.21.15.174:6828/3676139844', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bb0252a5-3aa1-43f0-be38-37413819720b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6800', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6801', 'nonce': 339941177}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6802', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6803', 'nonce': 339941177}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6806', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6807', 'nonce': 339941177}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6804', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6805', 'nonce': 339941177}]}, 'public_addr': '172.21.15.174:6801/339941177', 'cluster_addr': '172.21.15.174:6803/339941177', 'heartbeat_back_addr': '172.21.15.174:6807/339941177', 'heartbeat_front_addr': '172.21.15.174:6805/339941177', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.304258+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.336924+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.359766+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.384687+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.291288+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.318479+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.313564+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.232618+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-16T19:29:48.779 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-16T19:29:48.798 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:48.818+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:48.818 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:48.818 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:48.827 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:29:48.911 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:48.931+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:48.992 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:48.992 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:49.000 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:49.020+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:49.001 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:29:49.101 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:49.120+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:49.149 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:49.149 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:49.158 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:29:49.202 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:49.222+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:49.265 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:49.265 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:49.274 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:29:49.304 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:49.322+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:49.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.420+0000 7f52aaffd640 1 -- 172.21.15.172:0/3537572303 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v25) v1 ==== 91+0+0 (secure 0 0 0) 0x7f52ac009450 con 0x7f52b40720f0 2024-03-16T19:29:49.402 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:29:49.404 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:49.424+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:49.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f529806b3b0 msgr2=0x7f529806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:49.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f529806b3b0 0x7f529806d870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f52b40c3060 tx=0x7f52a4007e80 comp rx=0 tx=0).stop 2024-03-16T19:29:49.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b40720f0 msgr2=0x7f52b40cc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:49.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b40720f0 0x7f52b40cc320 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f52ac037040 tx=0x7f52ac004a40 comp rx=0 tx=0).stop 2024-03-16T19:29:49.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 shutdown_connections 2024-03-16T19:29:49.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f529806b3b0 0x7f529806d870 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:49.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52b4158f60 0x7f52b4147fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:49.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b40c3f30 0x7f52b406b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:49.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 --2- 172.21.15.172:0/3537572303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f52b40720f0 0x7f52b40cc320 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:49.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 >> 172.21.15.172:0/3537572303 conn(0x7f52b406db70 msgr2=0x7f52b40cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:49.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 shutdown_connections 2024-03-16T19:29:49.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:49.423+0000 7f52bc53a640 1 -- 172.21.15.172:0/3537572303 wait complete. 2024-03-16T19:29:49.422 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:49.423 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:49.432 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:29:49.505 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:49.524+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:49.582 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:49.583 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:49.592 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:29:49.606 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:49.624+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:49.699 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:49.699 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:49.708 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:29:49.708 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:49.726+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:49.809 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:49.827+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:49.815 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:49.815 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:49.824 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:29:49.909 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:49.929+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:49.932 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:49.932 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:49.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:29:50.010 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:50.030+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:50.058 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:50.058 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:50.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:29:50.111 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:50.131+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:50.173 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:50.173 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:50.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:29:50.225 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:50.244+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:50.288 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:50.288 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:50.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:29:50.312 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:50.332+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:50.413 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:50.431+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:50.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:29:50.445 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:50.445 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:50.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:29:50.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.553+0000 7f77de415640 1 Processor -- start 2024-03-16T19:29:50.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.553+0000 7f77de415640 1 -- start start 2024-03-16T19:29:50.536 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:50.554+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:50.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.554+0000 7f77de415640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d8071e50 0x7f77d815de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.554+0000 7f77de415640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77d806c0d0 0x7f77d80c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.554+0000 7f77de415640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d80c7510 con 0x7f77d8072230 2024-03-16T19:29:50.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.554+0000 7f77de415640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d80c7310 con 0x7f77d806c0d0 2024-03-16T19:29:50.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.554+0000 7f77de415640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d806c560 con 0x7f77d8071e50 2024-03-16T19:29:50.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.554+0000 7f77d77fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f77d8072230 0x7f77d80c7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:49982/0 (socket says 172.21.15.172:49982) 2024-03-16T19:29:50.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.554+0000 7f77dc9b2640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77d806c0d0 0x7f77d80c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.554+0000 7f77d77fe640 1 -- 172.21.15.172:0/1391745851 learned_addr learned my addr 172.21.15.172:0/1391745851 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:50.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.554+0000 7f77d7fff640 1 --2- 172.21.15.172:0/1391745851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d8071e50 0x7f77d815de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.555+0000 7f77dc9b2640 1 -- 172.21.15.172:0/1391745851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d8071e50 msgr2=0x7f77d815de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:50.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.555+0000 7f77dc9b2640 1 --2- 172.21.15.172:0/1391745851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d8071e50 0x7f77d815de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.555+0000 7f77dc9b2640 1 -- 172.21.15.172:0/1391745851 >> v1:172.21.15.172:6789/0 conn(0x7f77d8072230 legacy=0x7f77d80c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.555+0000 7f77dc9b2640 1 -- 172.21.15.172:0/1391745851 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d806b910 con 0x7f77d806c0d0 2024-03-16T19:29:50.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.555+0000 7f77d6ffd640 1 -- 172.21.15.172:0/1391745851 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625585207 0 0) 0x7f77d80c7510 con 0x7f77d8072230 2024-03-16T19:29:50.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.556+0000 7f77dc9b2640 1 --2- 172.21.15.172:0/1391745851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d806c0d0 0x7f77d80c6dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009d90 tx=0x7f77d002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed73014a7d22a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.556+0000 7f77d6ffd640 1 -- 172.21.15.172:0/1391745851 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d002cce0 con 0x7f77d806c0d0 2024-03-16T19:29:50.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.556+0000 7f77d6ffd640 1 -- 172.21.15.172:0/1391745851 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d002ce40 con 0x7f77d806c0d0 2024-03-16T19:29:50.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.556+0000 7f77d6ffd640 1 -- 172.21.15.172:0/1391745851 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0034660 con 0x7f77d806c0d0 2024-03-16T19:29:50.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.557+0000 7f77de415640 1 -- 172.21.15.172:0/1391745851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d806c0d0 msgr2=0x7f77d80c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.557+0000 7f77de415640 1 --2- 172.21.15.172:0/1391745851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d806c0d0 0x7f77d80c6dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009d90 tx=0x7f77d002bf30 comp rx=0 tx=0).stop 2024-03-16T19:29:50.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.557+0000 7f77de415640 1 -- 172.21.15.172:0/1391745851 shutdown_connections 2024-03-16T19:29:50.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.557+0000 7f77de415640 1 --2- 172.21.15.172:0/1391745851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d806c0d0 0x7f77d80c6dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.557+0000 7f77de415640 1 --2- 172.21.15.172:0/1391745851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d8071e50 0x7f77d815de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.557+0000 7f77de415640 1 -- 172.21.15.172:0/1391745851 >> 172.21.15.172:0/1391745851 conn(0x7f77d8072b40 msgr2=0x7f77d8072f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:50.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.557+0000 7f77de415640 1 -- 172.21.15.172:0/1391745851 shutdown_connections 2024-03-16T19:29:50.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.557+0000 7f77de415640 1 -- 172.21.15.172:0/1391745851 wait complete. 2024-03-16T19:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.558+0000 7f77de415640 1 Processor -- start 2024-03-16T19:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.558+0000 7f77de415640 1 -- start start 2024-03-16T19:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77de415640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77d806c0d0 0x7f77d815c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77de415640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d8071e50 0x7f77d80c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77de415640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d8072230 0x7f77d8148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77de415640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80cdbe0 con 0x7f77d8071e50 2024-03-16T19:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77de415640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80cc1f0 con 0x7f77d806c0d0 2024-03-16T19:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77de415640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80cb300 con 0x7f77d8072230 2024-03-16T19:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d8071e50 0x7f77d80c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d8071e50 0x7f77d80c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54144/0 (socket says 172.21.15.172:54144) 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d77fe640 1 -- 172.21.15.172:0/2964448911 learned_addr learned my addr 172.21.15.172:0/2964448911 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77dc9b2640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d8072230 0x7f77d8148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d7fff640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77d806c0d0 0x7f77d815c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d77fe640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d8072230 msgr2=0x7f77d8148900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d77fe640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d8072230 0x7f77d8148900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d77fe640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77d806c0d0 msgr2=0x7f77d815c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d77fe640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77d806c0d0 0x7f77d815c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d77fe640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d0009a70 con 0x7f77d8071e50 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d77fe640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d8071e50 0x7f77d80c59f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f77c800c430 tx=0x7f77c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c80074e0 con 0x7f77d8071e50 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c8007660 con 0x7f77d8071e50 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.559+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8004a20 con 0x7f77d8071e50 2024-03-16T19:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.560+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d80cb5e0 con 0x7f77d8071e50 2024-03-16T19:29:50.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.560+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d80cbba0 con 0x7f77d8071e50 2024-03-16T19:29:50.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.560+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f77d806f430 con 0x7f77d8071e50 2024-03-16T19:29:50.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.561+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f77c8004b80 con 0x7f77d8071e50 2024-03-16T19:29:50.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.561+0000 7f77d4ff9640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77c406b3e0 0x7f77c406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.561+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f77c8089820 con 0x7f77d8071e50 2024-03-16T19:29:50.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.562+0000 7f77d7fff640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77c406b3e0 0x7f77c406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.562+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v25) v1 ==== 54+0+16 (secure 0 0 0) 0x7f77c8089bc0 con 0x7f77d8071e50 2024-03-16T19:29:50.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.562+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f77d80cbdc0 0x7f77d8148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.562+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77d806b5a0 con 0x7f77d80cbdc0 2024-03-16T19:29:50.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.562+0000 7f77d7fff640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77c406b3e0 0x7f77c406d8a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f77cc009b40 tx=0x7f77cc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.562+0000 7f77dc9b2640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f77d80cbdc0 0x7f77d8148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.562+0000 7f77dc9b2640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f77d80cbdc0 0x7f77d8148e40 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.563+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77d003e440 con 0x7f77d80cbdc0 2024-03-16T19:29:50.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.578+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77d81e5140 con 0x7f77d80cbdc0 2024-03-16T19:29:50.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.579+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77d81e5140 con 0x7f77d80cbdc0 2024-03-16T19:29:50.561 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:29:50.562 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:29:50.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.579+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f77d80cae10 0x7f77d8074470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.579+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77d80cc330 con 0x7f77d80cae10 2024-03-16T19:29:50.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.580+0000 7f77d7fff640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f77d80cae10 0x7f77d8074470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.580+0000 7f77d7fff640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f77d80cae10 0x7f77d8074470 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.580+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77d80cc330 con 0x7f77d80cae10 2024-03-16T19:29:50.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.596+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77d815cb50 con 0x7f77d80cae10 2024-03-16T19:29:50.579 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.597+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77d815cb50 con 0x7f77d80cae10 2024-03-16T19:29:50.579 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:29:50.580 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:29:50.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.597+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f77d80cc680 0x7f77d8061e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.597+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77d80c6f70 con 0x7f77d80cc680 2024-03-16T19:29:50.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.598+0000 7f77d77fe640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f77d80cc680 0x7f77d8061e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.598+0000 7f77d77fe640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f77d80cc680 0x7f77d8061e40 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.599+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77d80c6f70 con 0x7f77d80cc680 2024-03-16T19:29:50.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.613+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77d8062650 con 0x7f77d80cc680 2024-03-16T19:29:50.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.614+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77d8062650 con 0x7f77d80cc680 2024-03-16T19:29:50.596 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:29:50.597 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:29:50.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.615+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f77d802c460 0x7f77d802c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.615+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77d802cdc0 con 0x7f77d802c460 2024-03-16T19:29:50.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.615+0000 7f77dc9b2640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f77d802c460 0x7f77d802c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.615+0000 7f77dc9b2640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f77d802c460 0x7f77d802c880 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.616+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77d802cdc0 con 0x7f77d802c460 2024-03-16T19:29:50.606 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:29:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:29:50.606 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:50.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.630+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77d802d260 con 0x7f77d802c460 2024-03-16T19:29:50.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.631+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77d802d260 con 0x7f77d802c460 2024-03-16T19:29:50.613 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:29:50.614 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:29:50.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.632+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f77d82010f0 0x7f77d82034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.632+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77d82062b0 con 0x7f77d82010f0 2024-03-16T19:29:50.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.632+0000 7f77d7fff640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f77d82010f0 0x7f77d82034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.632+0000 7f77d7fff640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f77d82010f0 0x7f77d82034e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.633+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77d82062b0 con 0x7f77d82010f0 2024-03-16T19:29:50.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:29:50.616 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:50.635+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:50.629 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.647+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77d8203a20 con 0x7f77d82010f0 2024-03-16T19:29:50.631 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:29:50.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.649+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77d8203a20 con 0x7f77d82010f0 2024-03-16T19:29:50.632 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:29:50.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.649+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f77d82040a0 0x7f77d8206490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.649+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77d8209260 con 0x7f77d82040a0 2024-03-16T19:29:50.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.649+0000 7f77d77fe640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f77d82040a0 0x7f77d8206490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.649+0000 7f77d77fe640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f77d82040a0 0x7f77d8206490 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.650+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77d8209260 con 0x7f77d82040a0 2024-03-16T19:29:50.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.666+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77d82069d0 con 0x7f77d82040a0 2024-03-16T19:29:50.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.667+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77d82069d0 con 0x7f77d82040a0 2024-03-16T19:29:50.649 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:29:50.650 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:29:50.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.668+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f77d8207050 0x7f77d8209440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.668+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77d820c210 con 0x7f77d8207050 2024-03-16T19:29:50.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.668+0000 7f77dc9b2640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f77d8207050 0x7f77d8209440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.668+0000 7f77dc9b2640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f77d8207050 0x7f77d8209440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.669+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77d820c210 con 0x7f77d8207050 2024-03-16T19:29:50.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.684+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77d8209a00 con 0x7f77d8207050 2024-03-16T19:29:50.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.686+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77d8209a00 con 0x7f77d8207050 2024-03-16T19:29:50.668 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:29:50.668 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:29:50.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.686+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f77d820a0e0 0x7f77d820a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:50.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.686+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77d820f5f0 con 0x7f77d820a0e0 2024-03-16T19:29:50.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.686+0000 7f77d7fff640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f77d820a0e0 0x7f77d820a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:50.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.687+0000 7f77d7fff640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f77d820a0e0 0x7f77d820a520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:50.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.687+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77d820f5f0 con 0x7f77d820a0e0 2024-03-16T19:29:50.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.701+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77d820e470 con 0x7f77d820a0e0 2024-03-16T19:29:50.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77d4ff9640 1 -- 172.21.15.172:0/2964448911 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77d820e470 con 0x7f77d820a0e0 2024-03-16T19:29:50.685 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:29:50.686 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:29:50.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f77d80cbdc0 msgr2=0x7f77d8148e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f77d80cbdc0 0x7f77d8148e40 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f77d80cae10 msgr2=0x7f77d8074470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f77d80cae10 0x7f77d8074470 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f77d80cc680 msgr2=0x7f77d8061e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f77d80cc680 0x7f77d8061e40 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f77d802c460 msgr2=0x7f77d802c880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f77d802c460 0x7f77d802c880 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77dc9b2640 1 -- 172.21.15.172:0/2964448911 reap_dead start 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77dc9b2640 1 -- 172.21.15.172:0/2964448911 reap_dead start 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f77d82010f0 msgr2=0x7f77d82034e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f77d82010f0 0x7f77d82034e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f77d82040a0 msgr2=0x7f77d8206490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.703+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f77d82040a0 0x7f77d8206490 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f77d8207050 msgr2=0x7f77d8209440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f77d8207050 0x7f77d8209440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f77d820a0e0 msgr2=0x7f77d820a520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f77d820a0e0 0x7f77d820a520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77c406b3e0 msgr2=0x7f77c406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77c406b3e0 0x7f77c406d8a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f77cc009b40 tx=0x7f77cc009190 comp rx=0 tx=0).stop 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d8071e50 msgr2=0x7f77d80c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77dc9b2640 1 -- 172.21.15.172:0/2964448911 reap_dead start 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d8071e50 0x7f77d80c59f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f77c800c430 tx=0x7f77c800c8f0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 shutdown_connections 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 --2- 172.21.15.172:0/2964448911 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77d8071e50 0x7f77d80c59f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 >> 172.21.15.172:0/2964448911 conn(0x7f77d8072b40 msgr2=0x7f77d8072f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 shutdown_connections 2024-03-16T19:29:50.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:50.704+0000 7f77de415640 1 -- 172.21.15.172:0/2964448911 wait complete. 2024-03-16T19:29:50.738 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:50.757+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.402406959, 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:29:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:50.746 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.402389844999998, 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:29:50.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:29:50.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:29:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.402382565, 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:29:50.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:50.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.402368698, 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:29:50.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:29:50.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.761 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.401695863, 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:29:50.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:29:50.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:29:50.765 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399397613000001, 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:29:50.769 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399352512, 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:29:50.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:29:50.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.773 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399324456999999, 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:29:50.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:29:50.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:29:50.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399303043, 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:29:50.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:29:50.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:29:50.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399280692000001, 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:29:50.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399260293000001, 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399154300999999, 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.790 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:29:50.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.39736259, 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.794 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:29:50.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:29:50.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.797 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.397315088999999, 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:29:50.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:29:50.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:29:50.801 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.397075837999999, 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:30 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.396920259000002, 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126666, 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362824+0000", 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7999999999999995e-07 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362828+0000", 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0879999999999997e-06 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362837+0000", 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.704e-06 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362843+0000", 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9390000000000002e-06 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362877+0000", 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4779e-05 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362883+0000", 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9009999999999999e-06 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362950+0000", 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6674999999999994e-05 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.388577693, 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:29:50.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:29:50.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:29:50.813 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.387237817000001, 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:29:50.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:29:50.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:29:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:50.817 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:50.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:29:50.819 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:50.838+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:50.919 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:50.938+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:50.967 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:50.967 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:50.967 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:29:50.967 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:50.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:29:51.020 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:51.040+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:51.121 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:51.139+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:51.121 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:29:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:29:51.122 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:51.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:29:51.221 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:51.240+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939948105999999, 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053568110000000004, 2024-03-16T19:29:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357146+0000", 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5860000000000001e-06 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357154+0000", 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5439999999999999e-06 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357217+0000", 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2986999999999999e-05 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357229+0000", 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1853e-05 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357343+0000", 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113663 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357504+0000", 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016176 2024-03-16T19:29:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013908599999999999 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360705+0000", 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030619419999999998 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362409+0000", 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001704021 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362456+0000", 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6140000000000002e-05 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362496+0000", 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0228999999999997e-05 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939928079000001, 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078702890000000008, 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4299999999999998e-07 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357163+0000", 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2559999999999998e-06 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357254+0000", 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1701999999999997e-05 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357258+0000", 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5719999999999999e-06 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357931+0000", 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067308000000000005 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357999+0000", 2024-03-16T19:29:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8323000000000003e-05 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358113+0000", 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011366699999999999 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362505+0000", 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004391509 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024372970000000002 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364992+0000", 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9741999999999999e-05 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365029+0000", 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7697999999999997e-05 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:29:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939921277, 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079944349999999994, 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6e-07 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357172+0000", 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7960000000000001e-06 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357285+0000", 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113183 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357290+0000", 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6190000000000002e-06 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358306+0000", 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010160670000000001 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358400+0000", 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3854e-05 2024-03-16T19:29:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358492+0000", 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2954000000000005e-05 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362510+0000", 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040175469999999998 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025680310000000001 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365123+0000", 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4515000000000003e-05 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365160+0000", 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7608999999999999e-05 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939914381000001, 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080787759999999993, 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.36e-07 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1270000000000001e-06 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357297+0000", 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121885 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357300+0000", 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049999999999999e-06 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358709+0000", 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001409083 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358776+0000", 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6489999999999995e-05 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358885+0000", 2024-03-16T19:29:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109271 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362514+0000", 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036292199999999998 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365197+0000", 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026832100000000001 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365226+0000", 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9136999999999999e-05 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365251+0000", 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5012e-05 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939911396999999, 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081787609999999997, 2024-03-16T19:29:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:29:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357177+0000", 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5090000000000001e-06 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357306+0000", 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012856900000000001 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357308+0000", 2024-03-16T19:29:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3700000000000002e-06 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359071+0000", 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017631210000000001 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359182+0000", 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110265 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359280+0000", 2024-03-16T19:29:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8595e-05 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362520+0000", 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032394120000000001 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365297+0000", 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002777542 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365328+0000", 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0899999999999999e-05 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365354+0000", 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6276000000000001e-05 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939908709000001, 2024-03-16T19:29:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083197830000000007, 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357179+0000", 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357187+0000", 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357314+0000", 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126978 2024-03-16T19:29:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357316+0000", 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3800000000000001e-06 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359512+0000", 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021950799999999999 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359585+0000", 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3168000000000001e-05 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359702+0000", 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011741 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362525+0000", 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028225419999999999 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365408+0000", 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028829010000000002 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365460+0000", 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839e-05 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365498+0000", 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7673999999999999e-05 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:29:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939897584000001, 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084493169999999996, 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:29:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357190+0000", 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-07 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357191+0000", 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7069999999999999e-06 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357325+0000", 2024-03-16T19:29:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133585 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357330+0000", 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0799999999999996e-06 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359915+0000", 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025849290000000001 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9741999999999995e-05 2024-03-16T19:29:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360121+0000", 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362826+0000", 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027046819999999999 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365564+0000", 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027387179999999998 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365606+0000", 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1715999999999998e-05 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365639+0000", 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660999999999998e-05 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:29:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939894915, 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085533069999999996, 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0699999999999999e-07 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3030000000000001e-06 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357339+0000", 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014513199999999999 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000003e-06 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030087619999999999 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360414+0000", 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1297999999999995e-05 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:29:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360498+0000", 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4538999999999998e-05 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362832+0000", 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023338410000000001 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365670+0000", 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028379550000000001 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365724+0000", 2024-03-16T19:29:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3533999999999999e-05 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365745+0000", 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1725999999999998e-05 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939891691, 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052850789999999998, 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357196+0000", 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.173e-06 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357359+0000", 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016222400000000001 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357365+0000", 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7320000000000003e-06 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357451+0000", 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6494000000000002e-05 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357555+0000", 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010401200000000001 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8224000000000001e-05 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362294+0000", 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046502399999999999 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362388+0000", 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4448000000000001e-05 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362433+0000", 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4888000000000001e-05 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362480+0000", 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7441000000000002e-05 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.939889654000002, 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053762239999999998, 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357198+0000", 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357199+0000", 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357415+0000", 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021638699999999999 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0069e-05 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358132+0000", 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070650599999999997 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358207+0000", 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5197999999999994e-05 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358342+0000", 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013503199999999999 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362377+0000", 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040354600000000003 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362493+0000", 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115951 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362536+0000", 2024-03-16T19:29:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3189999999999998e-05 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362574+0000", 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7113999999999997e-05 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:6 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.936830752999999, 2024-03-16T19:29:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045449999999999999, 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360257+0000", 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7400000000000002e-07 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360260+0000", 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3550000000000001e-06 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360270+0000", 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-03-16T19:29:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360277+0000", 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5760000000000002e-06 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360697+0000", 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042039200000000001 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360702+0000", 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7140000000000001e-06 2024-03-16T19:29:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360711+0000", 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1209999999999996e-06 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:22 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.934886578, 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100673, 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362201+0000", 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5799999999999998e-07 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362206+0000", 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8579999999999999e-06 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362219+0000", 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.239e-05 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362226+0000", 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2710000000000003e-06 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362281+0000", 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4756000000000002e-05 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362288+0000", 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.075e-06 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362301+0000", 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3365e-05 2024-03-16T19:29:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:23 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.934775308999999, 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017205200000000001, 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:29:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4499999999999997e-07 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362315+0000", 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1350000000000001e-06 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362322+0000", 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9469999999999997e-06 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362328+0000", 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-03-16T19:29:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362366+0000", 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8160000000000001e-05 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362373+0000", 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5799999999999997e-06 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362484+0000", 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111173 2024-03-16T19:29:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:7 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.932231436999999, 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018177499999999999, 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364861+0000", 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3699999999999997e-06 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364875+0000", 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4059e-05 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364881+0000", 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5270000000000004e-06 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364929+0000", 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000002e-05 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3699999999999997e-06 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365037+0000", 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100819 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:8 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.932174379999999, 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025451199999999998, 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:29:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364916+0000", 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8109999999999999e-06 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364924+0000", 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1419999999999993e-06 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364930+0000", 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2260000000000002e-06 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365071+0000", 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014030100000000001 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365075+0000", 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365167+0000", 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2196999999999998e-05 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:9 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.932148139999999, 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032082900000000001, 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.22e-07 2024-03-16T19:29:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3980000000000002e-06 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364947+0000", 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3090000000000003e-06 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364951+0000", 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1309999999999999e-06 2024-03-16T19:29:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365191+0000", 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023992600000000001 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365195+0000", 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.704e-06 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365260+0000", 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5139000000000001e-05 2024-03-16T19:29:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:10 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.932119753999999, 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039415499999999998, 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364968+0000", 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364970+0000", 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4650000000000001e-06 2024-03-16T19:29:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364977+0000", 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.443e-06 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364981+0000", 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2320000000000003e-06 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365289+0000", 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030856900000000002 2024-03-16T19:29:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365294+0000", 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6920000000000001e-06 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365361+0000", 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7358000000000001e-05 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:11 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.932099254000001, 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051803199999999998, 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9299999999999999e-07 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364993+0000", 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9080000000000003e-06 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4240000000000001e-06 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365002+0000", 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7570000000000002e-06 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365395+0000", 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039237199999999999 2024-03-16T19:29:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365401+0000", 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9380000000000003e-06 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365506+0000", 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010433999999999999 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:12 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.932077249999999, 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063430500000000005, 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9500000000000001e-07 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365012+0000", 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7129999999999999e-06 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365015+0000", 2024-03-16T19:29:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2629999999999999e-06 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365017+0000", 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1749999999999999e-06 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365544+0000", 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052690099999999998 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365559+0000", 2024-03-16T19:29:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4610000000000001e-05 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365644+0000", 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5447999999999999e-05 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:13 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.932066752000001, 2024-03-16T19:29:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00073113500000000005, 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1000000000000001e-07 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365021+0000", 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.23e-07 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365024+0000", 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7369999999999998e-06 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365026+0000", 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1289999999999999e-06 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365663+0000", 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063679600000000002 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365667+0000", 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.233e-06 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365751+0000", 2024-03-16T19:29:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4307000000000004e-05 2024-03-16T19:29:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.367 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:51.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:29:51.369 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:51.388+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:51.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:29:51.498 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:51.516+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:51.519 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:51.519 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:51.519 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:29:51.519 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:51.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:29:51.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.575+0000 7fb3ad90d640 1 Processor -- start 2024-03-16T19:29:51.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.575+0000 7fb3ad90d640 1 -- start start 2024-03-16T19:29:51.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.576+0000 7fb3ad90d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.576+0000 7fb3ad90d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb3a80c3930 0x7fb3a80c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.576+0000 7fb3ad90d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a8072b10 con 0x7fb3a8151910 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.576+0000 7fb3ad90d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a80c2330 con 0x7fb3a80c3930 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.576+0000 7fb3ad90d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80c7da0 con 0x7fb3a80c3ea0 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.576+0000 7fb3a6ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb3a8151910 0x7fb3a80ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:49986/0 (socket says 172.21.15.172:49986) 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.576+0000 7fb3a6ffd640 1 -- 172.21.15.172:0/1882284802 learned_addr learned my addr 172.21.15.172:0/1882284802 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.576+0000 7fb3a77fe640 1 --2- 172.21.15.172:0/1882284802 >> v2:172.21.15.174:3300/0 conn(0x7fb3a80c3930 0x7fb3a80c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.576+0000 7fb3a67fc640 1 --2- 172.21.15.172:0/1882284802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.577+0000 7fb3a5ffb640 1 -- 172.21.15.172:0/1882284802 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194244972 0 0) 0x7fb3a8072b10 con 0x7fb3a8151910 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.577+0000 7fb3a5ffb640 1 -- 172.21.15.172:0/1882284802 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb390003520 con 0x7fb3a8151910 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.577+0000 7fb3a67fc640 1 -- 172.21.15.172:0/1882284802 >> v2:172.21.15.174:3300/0 conn(0x7fb3a80c3930 msgr2=0x7fb3a80c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.577+0000 7fb3a67fc640 1 --2- 172.21.15.172:0/1882284802 >> v2:172.21.15.174:3300/0 conn(0x7fb3a80c3930 0x7fb3a80c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:51.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.577+0000 7fb3a67fc640 1 -- 172.21.15.172:0/1882284802 >> v1:172.21.15.172:6789/0 conn(0x7fb3a8151910 legacy=0x7fb3a80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:51.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.577+0000 7fb3a67fc640 1 -- 172.21.15.172:0/1882284802 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8071b10 con 0x7fb3a80c3ea0 2024-03-16T19:29:51.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.577+0000 7fb3a77fe640 1 --2- 172.21.15.172:0/1882284802 >> v2:172.21.15.174:3300/0 conn(0x7fb3a80c3930 0x7fb3a80c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:51.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3a67fc640 1 --2- 172.21.15.172:0/1882284802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8072fa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0097a0 tx=0x7fb39c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=997ffa0c5bf5161c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:51.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3a5ffb640 1 -- 172.21.15.172:0/1882284802 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c031830 con 0x7fb3a80c3ea0 2024-03-16T19:29:51.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3a5ffb640 1 -- 172.21.15.172:0/1882284802 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c031990 con 0x7fb3a80c3ea0 2024-03-16T19:29:51.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3a5ffb640 1 -- 172.21.15.172:0/1882284802 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c031d20 con 0x7fb3a80c3ea0 2024-03-16T19:29:51.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3ad90d640 1 -- 172.21.15.172:0/1882284802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 msgr2=0x7fb3a8072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:51.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3ad90d640 1 --2- 172.21.15.172:0/1882284802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8072fa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0097a0 tx=0x7fb39c02b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:51.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3ad90d640 1 -- 172.21.15.172:0/1882284802 shutdown_connections 2024-03-16T19:29:51.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3ad90d640 1 --2- 172.21.15.172:0/1882284802 >> v2:172.21.15.174:3300/0 conn(0x7fb3a80c3930 0x7fb3a80c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:51.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3ad90d640 1 --2- 172.21.15.172:0/1882284802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8072fa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3ad90d640 1 -- 172.21.15.172:0/1882284802 >> 172.21.15.172:0/1882284802 conn(0x7fb3a806db70 msgr2=0x7fb3a806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3ad90d640 1 -- 172.21.15.172:0/1882284802 shutdown_connections 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.578+0000 7fb3ad90d640 1 -- 172.21.15.172:0/1882284802 wait complete. 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.579+0000 7fb3ad90d640 1 Processor -- start 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3ad90d640 1 -- start start 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3ad90d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a80c3930 0x7fb3a81575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3ad90d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3ad90d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a8151910 0x7fb3a8154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3ad90d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8157120 con 0x7fb3a8151910 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3ad90d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a81ede00 con 0x7fb3a80c3930 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3ad90d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8156200 con 0x7fb3a80c3ea0 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3a77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a8151910 0x7fb3a8154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3a67fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:51.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3a6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a80c3930 0x7fb3a81575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3a67fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8157af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:55042/0 (socket says 172.21.15.172:55042) 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3a67fc640 1 -- 172.21.15.172:0/24069765 learned_addr learned my addr 172.21.15.172:0/24069765 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.580+0000 7fb3a77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a8151910 0x7fb3a8154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54154/0 (socket says 172.21.15.172:54154) 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3a77fe640 1 -- 172.21.15.172:0/24069765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 msgr2=0x7fb3a8157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3a77fe640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3a77fe640 1 -- 172.21.15.172:0/24069765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a80c3930 msgr2=0x7fb3a81575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3a77fe640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a80c3930 0x7fb3a81575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3a77fe640 1 -- 172.21.15.172:0/24069765 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a0009400 con 0x7fb3a8151910 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3a67fc640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3a6ffd640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a80c3930 0x7fb3a81575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3a77fe640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a8151910 0x7fb3a8154530 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000e7d0 tx=0x7fb3a000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb38ffff640 1 -- 172.21.15.172:0/24069765 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a000eee0 con 0x7fb3a8151910 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb38ffff640 1 -- 172.21.15.172:0/24069765 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a0004020 con 0x7fb3a8151910 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c009400 con 0x7fb3a8151910 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb38ffff640 1 -- 172.21.15.172:0/24069765 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00043b0 con 0x7fb3a8151910 2024-03-16T19:29:51.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.581+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a8156780 con 0x7fb3a8151910 2024-03-16T19:29:51.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.582+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a80c4970 con 0x7fb3a8151910 2024-03-16T19:29:51.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.583+0000 7fb38ffff640 1 -- 172.21.15.172:0/24069765 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb3a000c940 con 0x7fb3a8151910 2024-03-16T19:29:51.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.583+0000 7fb38ffff640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb39406b3b0 0x7fb39406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:51.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.583+0000 7fb38ffff640 1 -- 172.21.15.172:0/24069765 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5741+0+0 (secure 0 0 0) 0x7fb3a0089d40 con 0x7fb3a8151910 2024-03-16T19:29:51.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.583+0000 7fb3a6ffd640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb39406b3b0 0x7fb39406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:51.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.584+0000 7fb3a6ffd640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb39406b3b0 0x7fb39406d870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb398005c00 tx=0x7fb398009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:51.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.587+0000 7fb38ffff640 1 -- 172.21.15.172:0/24069765 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3a80c4970 con 0x7fb3a8151910 2024-03-16T19:29:51.570 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:51.588+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:51.666 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:29:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:29:51.666 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:51.671 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:51.690+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:51.675 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:29:51.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:51.778+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fb3a80c4970 con 0x7fb3a8151910 2024-03-16T19:29:51.772 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:51.790+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:51.823 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:29:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.484915244, 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.484900771, 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:29:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.484396281999999, 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:29:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.484380886, 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:29:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.483922202999999, 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:29:51.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.483376032999999, 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:29:51.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.482818942000002, 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.849 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:29:51.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.481617194000002, 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:29:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.481038484999999, 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:29:51.858 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:29:51.861 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.480988958000001, 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:29:51.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.864 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.480028314999998, 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:29:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.476314117000001, 2024-03-16T19:29:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:29:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.035913205, 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:29:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:29:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:29:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.470360202, 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:29:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.9627614770000008, 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:29:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:29:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:51.885 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:51.887 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:51.906+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:51.887 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:29:51.988 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:52.007+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:52.010 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:52.010 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:52.010 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:29:52.010 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:52.022 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:29:52.089 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:52.108+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:52.170 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:52.170 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:52.170 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:29:52.170 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:29:52.170 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:52.179 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:29:52.190 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:52.208+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.948717164000001, 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:29:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:29:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:29:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:29:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.947489741999998, 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:29:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:29:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.947087035999999, 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.946478285000001, 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:29:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:29:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.946034416, 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.945891639999999, 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.945877099, 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:29:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:29:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.942999403999998, 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:29:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:29:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.319 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:52.321 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:29:52.321 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:52.340+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:52.407 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.425+0000 7fb38ffff640 1 -- 172.21.15.172:0/24069765 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v26) v1 ==== 101+0+0 (secure 0 0 0) 0x7fb3a80c4970 con 0x7fb3a8151910 2024-03-16T19:29:52.407 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:29:52.410 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb39406b3b0 msgr2=0x7fb39406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb39406b3b0 0x7fb39406d870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb398005c00 tx=0x7fb398009040 comp rx=0 tx=0).stop 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a8151910 msgr2=0x7fb3a8154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a8151910 0x7fb3a8154530 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000e7d0 tx=0x7fb3a000ec90 comp rx=0 tx=0).stop 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 shutdown_connections 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3a8151910 0x7fb3a8154530 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb39406b3b0 0x7fb39406d870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a80c3ea0 0x7fb3a8157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 --2- 172.21.15.172:0/24069765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a80c3930 0x7fb3a81575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 >> 172.21.15.172:0/24069765 conn(0x7fb3a806db70 msgr2=0x7fb3a814efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 shutdown_connections 2024-03-16T19:29:52.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.429+0000 7fb3ad90d640 1 -- 172.21.15.172:0/24069765 wait complete. 2024-03-16T19:29:52.421 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:52.439+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:52.474 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:52.474 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:52.474 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:29:52.474 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:52.482 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:29:52.521 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:52.541+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:52.589 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:52.590 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:52.590 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:29:52.590 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:29:52.590 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:52.598 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:29:52.642 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:52.662+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:52.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:29:52.706 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:52.706 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:29:52.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:29:52.706 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:29:52.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.706 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:52.706 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:29:52.706 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.367208573999999, 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:29:52.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:29:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:29:52.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:29:52.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:29:52.711 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.711 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.711 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.367129456000001, 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:29:52.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:29:52.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:29:52.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:29:52.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:29:52.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:29:52.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:29:52.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:29:52.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.366515331999999, 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.717 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:29:52.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:29:52.721 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.366075262999999, 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:29:52.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.726 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.727 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.727 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:29:52.727 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:29:52.727 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.365833673000001, 2024-03-16T19:29:52.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:29:52.727 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:52.727 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:29:52.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:52.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:29:52.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:29:52.732 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.732 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.732 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.732 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:52.732 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:52.732 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:52.733 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:29:52.734 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:52.753+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:52.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.825+0000 7f517a5e2640 1 Processor -- start 2024-03-16T19:29:52.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.825+0000 7f517a5e2640 1 -- start start 2024-03-16T19:29:52.808 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f517a5e2640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5174154c20 0x7f517406f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.808 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f517a5e2640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517415e910 con 0x7f51740ca820 2024-03-16T19:29:52.808 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f517a5e2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517415ee70 con 0x7f51740ca2d0 2024-03-16T19:29:52.808 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f517a5e2640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517406c7d0 con 0x7f5174154c20 2024-03-16T19:29:52.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f5173fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5174154c20 0x7f517406f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f51737fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f51740ca820 0x7f517406cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:49996/0 (socket says 172.21.15.172:49996) 2024-03-16T19:29:52.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f51737fe640 1 -- 172.21.15.172:0/1717607373 learned_addr learned my addr 172.21.15.172:0/1717607373 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:52.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f5173fff640 1 -- 172.21.15.172:0/1717607373 >> v1:172.21.15.174:6789/0 conn(0x7f51740ca2d0 legacy=0x7f51740ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f5173fff640 1 -- 172.21.15.172:0/1717607373 >> v1:172.21.15.172:6789/0 conn(0x7f51740ca820 legacy=0x7f517406cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f5173fff640 1 -- 172.21.15.172:0/1717607373 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517415eb90 con 0x7f5174154c20 2024-03-16T19:29:52.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.826+0000 7f5172ffd640 1 -- 172.21.15.172:0/1717607373 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29001636 0 0) 0x7f517415e910 con 0x7f51740ca820 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.827+0000 7f5173fff640 1 --2- 172.21.15.172:0/1717607373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 0x7f517406f420 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f51640097d0 tx=0x7f516402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e91b7cf072fc87f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.827+0000 7f5172ffd640 1 -- 172.21.15.172:0/1717607373 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5164031830 con 0x7f5174154c20 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.827+0000 7f5172ffd640 1 -- 172.21.15.172:0/1717607373 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5164031990 con 0x7f5174154c20 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.827+0000 7f5172ffd640 1 -- 172.21.15.172:0/1717607373 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5164031d20 con 0x7f5174154c20 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.828+0000 7f517a5e2640 1 -- 172.21.15.172:0/1717607373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 msgr2=0x7f517406f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.828+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1717607373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 0x7f517406f420 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f51640097d0 tx=0x7f516402b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.828+0000 7f517a5e2640 1 -- 172.21.15.172:0/1717607373 shutdown_connections 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.828+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1717607373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 0x7f517406f420 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.828+0000 7f517a5e2640 1 -- 172.21.15.172:0/1717607373 >> 172.21.15.172:0/1717607373 conn(0x7f5174066c20 msgr2=0x7f5174067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.828+0000 7f517a5e2640 1 -- 172.21.15.172:0/1717607373 shutdown_connections 2024-03-16T19:29:52.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.828+0000 7f517a5e2640 1 -- 172.21.15.172:0/1717607373 wait complete. 2024-03-16T19:29:52.812 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.829+0000 7f517a5e2640 1 Processor -- start 2024-03-16T19:29:52.812 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.829+0000 7f517a5e2640 1 -- start start 2024-03-16T19:29:52.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f517a5e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51740ca2d0 0x7f5174151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f517a5e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51740ca820 0x7f51740b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f517a5e2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 0x7f51740b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f517a5e2640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51740b27e0 con 0x7f5174154c20 2024-03-16T19:29:52.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f517a5e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5174158810 con 0x7f51740ca820 2024-03-16T19:29:52.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f517a5e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51741f0370 con 0x7f51740ca2d0 2024-03-16T19:29:52.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5178b7f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 0x7f51740b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5178b7f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 0x7f51740b1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54172/0 (socket says 172.21.15.172:54172) 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5178b7f640 1 -- 172.21.15.172:0/1411670777 learned_addr learned my addr 172.21.15.172:0/1411670777 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5173fff640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51740ca2d0 0x7f5174151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f51737fe640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51740ca820 0x7f51740b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5178b7f640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51740ca2d0 msgr2=0x7f5174151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5178b7f640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51740ca2d0 0x7f5174151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5178b7f640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51740ca820 msgr2=0x7f51740b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5178b7f640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51740ca820 0x7f51740b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5178b7f640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5164009400 con 0x7f5174154c20 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.830+0000 7f5178b7f640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 0x7f51740b1630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c680 tx=0x7f516c00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.831+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c003b70 con 0x7f5174154c20 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.831+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c003cd0 con 0x7f5174154c20 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.831+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5174155180 con 0x7f5174154c20 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.831+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c005310 con 0x7f5174154c20 2024-03-16T19:29:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.831+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51740af8f0 con 0x7f5174154c20 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.831+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f51740b03d0 con 0x7f5174154c20 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.832+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f516c005470 con 0x7f5174154c20 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.832+0000 7f5170ff9640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516006b3b0 0x7f516006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.833+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f516c089820 con 0x7f5174154c20 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.833+0000 7f5173fff640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516006b3b0 0x7f516006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.833+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v26) v1 ==== 54+0+16 (secure 0 0 0) 0x7f51740b03d0 con 0x7f5174154c20 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.833+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51740b0bf0 0x7f51740ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.833+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51740b03d0 con 0x7f51740b0bf0 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.833+0000 7f51737fe640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51740b0bf0 0x7f51740ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.833+0000 7f5173fff640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516006b3b0 0x7f516006d870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5164036000 tx=0x7f516403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.833+0000 7f51737fe640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51740b0bf0 0x7f51740ce1b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.834+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51740b03d0 con 0x7f51740b0bf0 2024-03-16T19:29:52.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.849+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51740681c0 con 0x7f51740b0bf0 2024-03-16T19:29:52.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.850+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51740681c0 con 0x7f51740b0bf0 2024-03-16T19:29:52.832 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:29:52.833 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:29:52.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.850+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51740b2080 0x7f51741f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.850+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51740c9150 con 0x7f51740b2080 2024-03-16T19:29:52.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.850+0000 7f5173fff640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51740b2080 0x7f51741f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.851+0000 7f5173fff640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51740b2080 0x7f51741f38e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.851+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51740c9150 con 0x7f51740b2080 2024-03-16T19:29:52.834 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:52.853+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:52.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.866+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51740670a0 con 0x7f51740b2080 2024-03-16T19:29:52.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.868+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51740670a0 con 0x7f51740b2080 2024-03-16T19:29:52.850 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:29:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:29:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.868+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f517414f0a0 0x7f517414f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.868+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51741f46f0 con 0x7f517414f0a0 2024-03-16T19:29:52.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.868+0000 7f51737fe640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f517414f0a0 0x7f517414f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.868+0000 7f51737fe640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f517414f0a0 0x7f517414f480 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.869+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51741f46f0 con 0x7f517414f0a0 2024-03-16T19:29:52.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.883+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f517414fc10 con 0x7f517414f0a0 2024-03-16T19:29:52.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.885+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f517414fc10 con 0x7f517414f0a0 2024-03-16T19:29:52.866 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:29:52.867 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:29:52.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.885+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f5174061e40 0x7f51740622a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.885+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51740627e0 con 0x7f5174061e40 2024-03-16T19:29:52.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.885+0000 7f5178b7f640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f5174061e40 0x7f51740622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.885+0000 7f5178b7f640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f5174061e40 0x7f51740622a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.886+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51740627e0 con 0x7f5174061e40 2024-03-16T19:29:52.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.900+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5174150dd0 con 0x7f5174061e40 2024-03-16T19:29:52.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.902+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5174150dd0 con 0x7f5174061e40 2024-03-16T19:29:52.884 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:29:52.884 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:29:52.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.902+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51741514b0 0x7f517402bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.902+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51741518d0 con 0x7f51741514b0 2024-03-16T19:29:52.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.902+0000 7f5173fff640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51741514b0 0x7f517402bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.902+0000 7f5173fff640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51741514b0 0x7f517402bde0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.903+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51741518d0 con 0x7f51741514b0 2024-03-16T19:29:52.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.917+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51741518d0 con 0x7f51741514b0 2024-03-16T19:29:52.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.919+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51741518d0 con 0x7f51741514b0 2024-03-16T19:29:52.900 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:29:52.901 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:29:52.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.919+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f517402c830 0x7f517402ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.919+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5174151a70 con 0x7f517402c830 2024-03-16T19:29:52.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.919+0000 7f51737fe640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f517402c830 0x7f517402ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.919+0000 7f51737fe640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f517402c830 0x7f517402ccb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.920+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5174151a70 con 0x7f517402c830 2024-03-16T19:29:52.903 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:52.903 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:52.903 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:29:52.903 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:52.912 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:29:52.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.934+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5174151a40 con 0x7f517402c830 2024-03-16T19:29:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.936+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5174151a40 con 0x7f517402c830 2024-03-16T19:29:52.918 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:29:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:29:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.936+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f5174207390 0x7f5174209780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.936+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f517402d1f0 con 0x7f5174207390 2024-03-16T19:29:52.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.936+0000 7f5178b7f640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f5174207390 0x7f5174209780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.936+0000 7f5178b7f640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f5174207390 0x7f5174209780 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.938+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f517402d1f0 con 0x7f5174207390 2024-03-16T19:29:52.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.951+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f517402d1f0 con 0x7f5174207390 2024-03-16T19:29:52.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.953+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f517402d1f0 con 0x7f5174207390 2024-03-16T19:29:52.936 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:52.954+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:52.936 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:29:52.936 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:29:52.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.954+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f517420a1d0 0x7f517420c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:52.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.954+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f517420f460 con 0x7f517420a1d0 2024-03-16T19:29:52.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.954+0000 7f5173fff640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f517420a1d0 0x7f517420c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:52.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.954+0000 7f5173fff640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f517420a1d0 0x7f517420c5c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:52.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.955+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f517420f460 con 0x7f517420a1d0 2024-03-16T19:29:52.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.969+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f517420cb00 con 0x7f517420a1d0 2024-03-16T19:29:52.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f5170ff9640 1 -- 172.21.15.172:0/1411670777 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f517420cb00 con 0x7f517420a1d0 2024-03-16T19:29:52.953 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:29:52.954 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:29:52.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51740b0bf0 msgr2=0x7f51740ce1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51740b0bf0 0x7f51740ce1b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51740b2080 msgr2=0x7f51741f38e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51740b2080 0x7f51741f38e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f517414f0a0 msgr2=0x7f517414f480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f517414f0a0 0x7f517414f480 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f5174061e40 msgr2=0x7f51740622a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f5174061e40 0x7f51740622a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f5178b7f640 1 -- 172.21.15.172:0/1411670777 reap_dead start 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51741514b0 msgr2=0x7f517402bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51741514b0 0x7f517402bde0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f5178b7f640 1 -- 172.21.15.172:0/1411670777 reap_dead start 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f517402c830 msgr2=0x7f517402ccb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f517402c830 0x7f517402ccb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f5174207390 msgr2=0x7f5174209780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f5174207390 0x7f5174209780 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f517420a1d0 msgr2=0x7f517420c5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f517420a1d0 0x7f517420c5c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516006b3b0 msgr2=0x7f516006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516006b3b0 0x7f516006d870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5164036000 tx=0x7f516403c040 comp rx=0 tx=0).stop 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 msgr2=0x7f51740b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 --2- 172.21.15.172:0/1411670777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5174154c20 0x7f51740b1630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c680 tx=0x7f516c00cb40 comp rx=0 tx=0).stop 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f5178b7f640 1 -- 172.21.15.172:0/1411670777 reap_dead start 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 shutdown_connections 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 >> 172.21.15.172:0/1411670777 conn(0x7f5174066c20 msgr2=0x7f5174153550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 shutdown_connections 2024-03-16T19:29:52.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:52.971+0000 7f517a5e2640 1 -- 172.21.15.172:0/1411670777 wait complete. 2024-03-16T19:29:53.019 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:53.019 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:53.028 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:29:53.071 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:53.091+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:53.137 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:53.156+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:53.183 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:53.183 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:53.195 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:29:53.238 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:53.256+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:53.338 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:53.356+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:53.341 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:53.341 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:53.350 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:29:53.438 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:53.457+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:53.456 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:53.456 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:53.465 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:29:53.539 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:53.558+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:53.572 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:53.572 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:53.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:29:53.640 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:53.660+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:53.689 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:53.690 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:53.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:29:53.742 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:53.761+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:53.781 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:29:53.781 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:29:53.781 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-16T19:29:53.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-03-16T19:29:53.805 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:53.806 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:53.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:29:53.841 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:53.861+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:53.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.902+0000 7fa79bbf4640 1 Processor -- start 2024-03-16T19:29:53.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.902+0000 7fa79bbf4640 1 -- start start 2024-03-16T19:29:53.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79bbf4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa794154b60 0x7fa79415aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:53.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79bbf4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7940c7f80 0x7fa794071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:53.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79bbf4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940c7750 con 0x7fa794154f40 2024-03-16T19:29:53.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79bbf4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7940c6e10 con 0x7fa794154b60 2024-03-16T19:29:53.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79bbf4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940c6f80 con 0x7fa7940c7f80 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79918f640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa794154f40 0x7fa79406c1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:50006/0 (socket says 172.21.15.172:50006) 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79918f640 1 -- 172.21.15.172:0/2139332251 learned_addr learned my addr 172.21.15.172:0/2139332251 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79a191640 1 --2- 172.21.15.172:0/2139332251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7940c7f80 0x7fa794071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa799990640 1 --2- 172.21.15.172:0/2139332251 >> v2:172.21.15.174:3300/0 conn(0x7fa794154b60 0x7fa79415aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79898e640 1 -- 172.21.15.172:0/2139332251 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785884341 0 0) 0x7fa7940c7750 con 0x7fa794154f40 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79898e640 1 -- 172.21.15.172:0/2139332251 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa780003520 con 0x7fa794154f40 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa799990640 1 -- 172.21.15.172:0/2139332251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7940c7f80 msgr2=0x7fa794071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa799990640 1 --2- 172.21.15.172:0/2139332251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7940c7f80 0x7fa794071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa799990640 1 -- 172.21.15.172:0/2139332251 >> v1:172.21.15.172:6789/0 conn(0x7fa794154f40 legacy=0x7fa79406c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa799990640 1 -- 172.21.15.172:0/2139332251 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7940c7190 con 0x7fa794154b60 2024-03-16T19:29:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.903+0000 7fa79a191640 1 --2- 172.21.15.172:0/2139332251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7940c7f80 0x7fa794071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:53.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.904+0000 7fa799990640 1 --2- 172.21.15.172:0/2139332251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794154b60 0x7fa79415aa60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7840097a0 tx=0x7fa78402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d51694362661e4ac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.904+0000 7fa79898e640 1 -- 172.21.15.172:0/2139332251 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784031830 con 0x7fa794154b60 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.904+0000 7fa79898e640 1 -- 172.21.15.172:0/2139332251 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa784031990 con 0x7fa794154b60 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.904+0000 7fa79898e640 1 -- 172.21.15.172:0/2139332251 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784031d20 con 0x7fa794154b60 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.905+0000 7fa79bbf4640 1 -- 172.21.15.172:0/2139332251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794154b60 msgr2=0x7fa79415aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.905+0000 7fa79bbf4640 1 --2- 172.21.15.172:0/2139332251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794154b60 0x7fa79415aa60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7840097a0 tx=0x7fa78402b8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.905+0000 7fa79bbf4640 1 -- 172.21.15.172:0/2139332251 shutdown_connections 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.905+0000 7fa79bbf4640 1 --2- 172.21.15.172:0/2139332251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7940c7f80 0x7fa794071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.905+0000 7fa79bbf4640 1 --2- 172.21.15.172:0/2139332251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794154b60 0x7fa79415aa60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.905+0000 7fa79bbf4640 1 -- 172.21.15.172:0/2139332251 >> 172.21.15.172:0/2139332251 conn(0x7fa79406e8d0 msgr2=0x7fa79406ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.905+0000 7fa79bbf4640 1 -- 172.21.15.172:0/2139332251 shutdown_connections 2024-03-16T19:29:53.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.905+0000 7fa79bbf4640 1 -- 172.21.15.172:0/2139332251 wait complete. 2024-03-16T19:29:53.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.906+0000 7fa79bbf4640 1 Processor -- start 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.906+0000 7fa79bbf4640 1 -- start start 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa79bbf4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7940c7f80 0x7fa7941e3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa79bbf4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794154b60 0x7fa794071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa79bbf4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa794154f40 0x7fa7940ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa79bbf4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940cc930 con 0x7fa7940c7f80 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa79bbf4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794151260 con 0x7fa794154b60 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa79bbf4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940cc2b0 con 0x7fa794154f40 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa799990640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7940c7f80 0x7fa7941e3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa799990640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7940c7f80 0x7fa7941e3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54184/0 (socket says 172.21.15.172:54184) 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa799990640 1 -- 172.21.15.172:0/998581762 learned_addr learned my addr 172.21.15.172:0/998581762 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa79a191640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa794154f40 0x7fa7940ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa79918f640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794154b60 0x7fa794071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa799990640 1 -- 172.21.15.172:0/998581762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa794154f40 msgr2=0x7fa7940ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:53.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa799990640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa794154f40 0x7fa7940ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa799990640 1 -- 172.21.15.172:0/998581762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794154b60 msgr2=0x7fa794071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa799990640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794154b60 0x7fa794071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa799990640 1 -- 172.21.15.172:0/998581762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c009a70 con 0x7fa7940c7f80 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.907+0000 7fa79a191640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa794154f40 0x7fa7940ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.908+0000 7fa799990640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7940c7f80 0x7fa7941e3b90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa78402bee0 tx=0x7fa7840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.908+0000 7fa78a7fc640 1 -- 172.21.15.172:0/998581762 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78403a630 con 0x7fa7940c7f80 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.908+0000 7fa78a7fc640 1 -- 172.21.15.172:0/998581762 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78403a7b0 con 0x7fa7940c7f80 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.908+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa784009400 con 0x7fa7940c7f80 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.908+0000 7fa78a7fc640 1 -- 172.21.15.172:0/998581762 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78403ab60 con 0x7fa7940c7f80 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.908+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7941517c0 con 0x7fa7940c7f80 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.909+0000 7fa78a7fc640 1 -- 172.21.15.172:0/998581762 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa78403acc0 con 0x7fa7940c7f80 2024-03-16T19:29:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.909+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7940688d0 con 0x7fa7940c7f80 2024-03-16T19:29:53.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.909+0000 7fa78a7fc640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa77806b3b0 0x7fa77806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:53.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.910+0000 7fa78a7fc640 1 -- 172.21.15.172:0/998581762 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5741+0+0 (secure 0 0 0) 0x7fa7840ab5d0 con 0x7fa7940c7f80 2024-03-16T19:29:53.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.910+0000 7fa79918f640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa77806b3b0 0x7fa77806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:53.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.910+0000 7fa79918f640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa77806b3b0 0x7fa77806d870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa79415a210 tx=0x7fa790009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:53.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:53.914+0000 7fa78a7fc640 1 -- 172.21.15.172:0/998581762 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa784076350 con 0x7fa7940c7f80 2024-03-16T19:29:53.942 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:53.961+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:53.982 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:53.982 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:53.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:29:54.076 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:54.094+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:54.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.114+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa7941e40d0 con 0x7fa7940c7f80 2024-03-16T19:29:54.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.115+0000 7fa78a7fc640 1 -- 172.21.15.172:0/998581762 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v26) v1 ==== 81+0+34331 (secure 0 0 0) 0x7fa784079a10 con 0x7fa7940c7f80 2024-03-16T19:29:54.097 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:54.097 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:54.097 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 26, 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "fsid": "6c00e600-a879-4eb6-acee-0fdf0f76922a", 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "created": "2024-03-16T19:29:02.010581+0000", 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "modified": "2024-03-16T19:29:52.423682+0000", 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "last_up_change": "2024-03-16T19:29:13.180924+0000", 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "last_in_change": "2024-03-16T19:29:04.178546+0000", 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_num": 5799936, 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_set": [ 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "pglog_hardlimit", 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "purged_snapdirs", 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_deletes", 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "sortbitwise" 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:29:54.098 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_version": 4, 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "full_ratio": 0.94999998807907104, 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_snapshot": "", 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_max": 3, 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "max_osd": 8, 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "require_min_compat_client": "luminous", 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "min_compat_client": "jewel", 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "require_osd_release": "squid", 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "allow_crimson": false, 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "pools": [ 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "pool": 1, 2024-03-16T19:29:54.099 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_name": ".mgr", 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "create_time": "2024-03-16T19:29:14.242742+0000", 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": 1, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_names": "hashpspool", 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "type": 1, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 2, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "min_size": 1, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_rule": 0, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_count": 0, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_target": 0, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_barrier": 0, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "object_hash": 2, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_mode": "off", 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num": 1, 2024-03-16T19:29:54.100 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num": 1, 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num_target": 1, 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_target": 1, 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_pending": 1, 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "last_pg_merge_meta": { 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "source_pgid": "0.0", 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "ready_epoch": 0, 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_started": 0, 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_clean": 0, 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "source_version": "0'0", 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "target_version": "0'0" 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "last_change": "17", 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend": "0", 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-16T19:29:54.101 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_preluminous": "0", 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "auid": 0, 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_mode": "selfmanaged", 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_seq": 0, 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_epoch": 0, 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_snaps": [], 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps": "[]", 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_bytes": 0, 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_objects": 0, 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "tiers": [], 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_of": -1, 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "read_tier": -1, 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "write_tier": -1, 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_mode": "none", 2024-03-16T19:29:54.102 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_bytes": 0, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_objects": 0, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_flush_age": 0, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_evict_age": 0, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profile": "", 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_params": { 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "none" 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_period": 0, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_count": 0, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "use_gmt_hitset": true, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "min_read_recency_for_promote": 0, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "min_write_recency_for_promote": 0, 2024-03-16T19:29:54.103 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_grade_decay_rate": 0, 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_search_last_n": 0, 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "grade_table": [], 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "stripe_width": 0, 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "expected_num_objects": 0, 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "fast_read": false, 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "options": { 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_max": 32, 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_min": 1 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "application_metadata": { 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": {} 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "read_balance": { 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "score_type": "Fair distribution", 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "score_acting": 6.9000000953674316, 2024-03-16T19:29:54.104 INFO:teuthology.orchestra.run.smithi172.stdout: "score_stable": 6.9000000953674316, 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "optimal_score": 0.28999999165534973, 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_acting": 2, 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_stable": 2, 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity_weighted": 0.875, 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity": 1, 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "pool": 2, 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_name": "cephfs_metadata", 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "create_time": "2024-03-16T19:29:24.121132+0000", 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": 1, 2024-03-16T19:29:54.105 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_names": "hashpspool", 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "type": 1, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 2, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "min_size": 1, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_rule": 0, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_count": 0, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_target": 0, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_barrier": 0, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "object_hash": 2, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_mode": "off", 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num": 32, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num": 32, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num_target": 32, 2024-03-16T19:29:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_target": 32, 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_pending": 32, 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "last_pg_merge_meta": { 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "source_pgid": "0.0", 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "ready_epoch": 0, 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_started": 0, 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_clean": 0, 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "source_version": "0'0", 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "target_version": "0'0" 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "last_change": "20", 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend": "0", 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_preluminous": "0", 2024-03-16T19:29:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "auid": 0, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_mode": "selfmanaged", 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_seq": 0, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_epoch": 0, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_snaps": [], 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps": "[]", 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_bytes": 0, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_objects": 0, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "tiers": [], 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_of": -1, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "read_tier": -1, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "write_tier": -1, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_mode": "none", 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_bytes": 0, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_objects": 0, 2024-03-16T19:29:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_flush_age": 0, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_evict_age": 0, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profile": "", 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_params": { 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "none" 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_period": 0, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_count": 0, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "use_gmt_hitset": true, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "min_read_recency_for_promote": 0, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "min_write_recency_for_promote": 0, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_grade_decay_rate": 0, 2024-03-16T19:29:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_search_last_n": 0, 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "grade_table": [], 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "stripe_width": 0, 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "expected_num_objects": 0, 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "fast_read": false, 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "options": { 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_bias": 4, 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_min": 16, 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_priority": 5 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "application_metadata": { 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs": { 2024-03-16T19:29:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "metadata": "cephfs" 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "read_balance": { 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "score_type": "Fair distribution", 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "score_acting": 1.5399999618530273, 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "score_stable": 1.5399999618530273, 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "optimal_score": 1.1399999856948853, 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_acting": 1.75, 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_stable": 1.75, 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity_weighted": 0.875, 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity": 1, 2024-03-16T19:29:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "pool": 3, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_name": "cephfs_data", 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "create_time": "2024-03-16T19:29:24.651099+0000", 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": 1, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_names": "hashpspool", 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "type": 1, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 2, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "min_size": 1, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_rule": 0, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_count": 0, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_target": 0, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_barrier": 0, 2024-03-16T19:29:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "object_hash": 2, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_mode": "off", 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num": 64, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num": 64, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num_target": 64, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_target": 64, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_pending": 64, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "last_pg_merge_meta": { 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "source_pgid": "0.0", 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "ready_epoch": 0, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_started": 0, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_clean": 0, 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "source_version": "0'0", 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "target_version": "0'0" 2024-03-16T19:29:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "last_change": "21", 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend": "0", 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_preluminous": "0", 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "auid": 0, 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_mode": "selfmanaged", 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_seq": 0, 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_epoch": 0, 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_snaps": [], 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps": "[]", 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_bytes": 0, 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_objects": 0, 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "tiers": [], 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_of": -1, 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "read_tier": -1, 2024-03-16T19:29:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "write_tier": -1, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_mode": "none", 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_bytes": 0, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_objects": 0, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_flush_age": 0, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_evict_age": 0, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profile": "", 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_params": { 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "none" 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_period": 0, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_count": 0, 2024-03-16T19:29:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "use_gmt_hitset": true, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "min_read_recency_for_promote": 0, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "min_write_recency_for_promote": 0, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_grade_decay_rate": 0, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_search_last_n": 0, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "grade_table": [], 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "stripe_width": 0, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "expected_num_objects": 0, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "fast_read": false, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "options": { 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_min": 64, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "target_size_ratio": 0.89999997615814209 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "application_metadata": { 2024-03-16T19:29:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs": { 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "data": "cephfs" 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "read_balance": { 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "score_type": "Fair distribution", 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "score_acting": 1.5399999618530273, 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "score_stable": 1.5399999618530273, 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "optimal_score": 1.1399999856948853, 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_acting": 1.75, 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_stable": 1.75, 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity_weighted": 0.875, 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity": 1, 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "osds": [ 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 0, 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "460ed9e4-eacf-4544-9199-23ca7208b663", 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 0, 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 19, 2024-03-16T19:29:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6800", 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6801", 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6802", 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6803", 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6806", 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6807", 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6804", 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6805", 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6801/491754535", 2024-03-16T19:29:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6803/491754535", 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6807/491754535", 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6805/491754535", 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 1, 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "b84ebb50-079d-460f-bf05-71040f17500a", 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6808", 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6809", 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6810", 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6811", 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6814", 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6815", 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6812", 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6813", 2024-03-16T19:29:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6809/3098410089", 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6811/3098410089", 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6815/3098410089", 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6813/3098410089", 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 2, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "6774c42c-5373-4543-8f8c-ed7b82c12057", 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6823", 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6825", 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6826", 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6827", 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6830", 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6831", 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6828", 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6829", 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6825/1599824697", 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6827/1599824697", 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6831/1599824697", 2024-03-16T19:29:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6829/1599824697", 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 3, 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "b96838ef-d991-43ab-b2d7-16deab58ca78", 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6816", 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6817", 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6818", 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6819", 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6822", 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6824", 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6820", 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6821", 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6817/1613555619", 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6819/1613555619", 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6824/1613555619", 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6821/1613555619", 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 4, 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "f5c78559-d22e-4847-be3f-2f597d7f6789", 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:54.139 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6808", 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.140 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6809", 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6810", 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6811", 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6814", 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6815", 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6812", 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.143 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6813", 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6809/967866068", 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6811/967866068", 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6815/967866068", 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6813/967866068", 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 5, 2024-03-16T19:29:54.144 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "e1f4797f-31c3-4292-a4a1-88fa0ab64f9a", 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.145 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6816", 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6817", 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.146 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6818", 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6820", 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.147 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6827", 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6829", 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6822", 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:54.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6825", 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6817/3985312519", 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6820/3985312519", 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6829/3985312519", 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6825/3985312519", 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:54.149 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 6, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "ea3e888e-c4c6-444a-8ebe-e26cddb2084b", 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 19, 2024-03-16T19:29:54.150 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6819", 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6821", 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6823", 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6824", 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.152 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6830", 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6831", 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:54.153 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6826", 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6828", 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6821/3676139844", 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6824/3676139844", 2024-03-16T19:29:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6831/3676139844", 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6828/3676139844", 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 7, 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "bb0252a5-3aa1-43f0-be38-37413819720b", 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:29:54.155 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6800", 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6801", 2024-03-16T19:29:54.156 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6802", 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6803", 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6806", 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6807", 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6804", 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6805", 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.159 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6801/339941177", 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6803/339941177", 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6807/339941177", 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6805/339941177", 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_xinfo": [ 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 0, 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:54.160 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.304258+0000", 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 1, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:54.161 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.336924+0000", 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 2, 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.359766+0000", 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 3, 2024-03-16T19:29:54.162 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.384687+0000", 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 4, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.291288+0000", 2024-03-16T19:29:54.163 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 5, 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.318479+0000", 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 6, 2024-03-16T19:29:54.164 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.313564+0000", 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 7, 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:29:54.165 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.232618+0000", 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap": [], 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_items": [], 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_primaries": [], 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_temp": [], 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_temp": [], 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "blocklist": {}, 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "range_blocklist": {}, 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profiles": { 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "default": { 2024-03-16T19:29:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: "crush-failure-domain": "osd", 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "k": "2", 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "m": "1", 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "plugin": "jerasure", 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "technique": "reed_sol_van" 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps_queue": [], 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "new_removed_snaps": [], 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "new_purged_snaps": [], 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_node_flags": {}, 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "device_class_flags": {}, 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_mode": { 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_mode_enabled": false, 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_bucket_count": 0, 2024-03-16T19:29:54.167 INFO:teuthology.orchestra.run.smithi172.stdout: "degraded_stretch_mode": 0, 2024-03-16T19:29:54.168 INFO:teuthology.orchestra.run.smithi172.stdout: "recovering_stretch_mode": 0, 2024-03-16T19:29:54.168 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_mode_bucket": 0 2024-03-16T19:29:54.168 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:54.168 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa77806b3b0 msgr2=0x7fa77806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa77806b3b0 0x7fa77806d870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa79415a210 tx=0x7fa790009040 comp rx=0 tx=0).stop 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7940c7f80 msgr2=0x7fa7941e3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7940c7f80 0x7fa7941e3b90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa78402bee0 tx=0x7fa7840046c0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 shutdown_connections 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa77806b3b0 0x7fa77806d870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa794154f40 0x7fa7940ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794154b60 0x7fa794071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 --2- 172.21.15.172:0/998581762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa7940c7f80 0x7fa7941e3b90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 >> 172.21.15.172:0/998581762 conn(0x7fa79406e8d0 msgr2=0x7fa794158c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 shutdown_connections 2024-03-16T19:29:54.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.117+0000 7fa79bbf4640 1 -- 172.21.15.172:0/998581762 wait complete. 2024-03-16T19:29:54.171 INFO:tasks.thrashosds.thrasher:j is {'epoch': 26, 'fsid': '6c00e600-a879-4eb6-acee-0fdf0f76922a', 'created': '2024-03-16T19:29:02.010581+0000', 'modified': '2024-03-16T19:29:52.423682+0000', 'last_up_change': '2024-03-16T19:29:13.180924+0000', 'last_in_change': '2024-03-16T19:29:04.178546+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:29:14.242742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:29:24.121132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:29:24.651099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '460ed9e4-eacf-4544-9199-23ca7208b663', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 491754535}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 491754535}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 491754535}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 491754535}]}, 'public_addr': '172.21.15.172:6801/491754535', 'cluster_addr': '172.21.15.172:6803/491754535', 'heartbeat_back_addr': '172.21.15.172:6807/491754535', 'heartbeat_front_addr': '172.21.15.172:6805/491754535', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b84ebb50-079d-460f-bf05-71040f17500a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6809', 'nonce': 3098410089}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6810', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 3098410089}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6814', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 3098410089}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6813', 'nonce': 3098410089}]}, 'public_addr': '172.21.15.172:6809/3098410089', 'cluster_addr': '172.21.15.172:6811/3098410089', 'heartbeat_back_addr': '172.21.15.172:6815/3098410089', 'heartbeat_front_addr': '172.21.15.172:6813/3098410089', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6774c42c-5373-4543-8f8c-ed7b82c12057', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6823', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 1599824697}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 1599824697}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 1599824697}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 1599824697}]}, 'public_addr': '172.21.15.172:6825/1599824697', 'cluster_addr': '172.21.15.172:6827/1599824697', 'heartbeat_back_addr': '172.21.15.172:6831/1599824697', 'heartbeat_front_addr': '172.21.15.172:6829/1599824697', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'b96838ef-d991-43ab-b2d7-16deab58ca78', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6817', 'nonce': 1613555619}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6818', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6819', 'nonce': 1613555619}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6824', 'nonce': 1613555619}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6820', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 1613555619}]}, 'public_addr': '172.21.15.172:6817/1613555619', 'cluster_addr': '172.21.15.172:6819/1613555619', 'heartbeat_back_addr': '172.21.15.172:6824/1613555619', 'heartbeat_front_addr': '172.21.15.172:6821/1613555619', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f5c78559-d22e-4847-be3f-2f597d7f6789', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6808', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6809', 'nonce': 967866068}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6810', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6811', 'nonce': 967866068}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6814', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6815', 'nonce': 967866068}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6812', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6813', 'nonce': 967866068}]}, 'public_addr': '172.21.15.174:6809/967866068', 'cluster_addr': '172.21.15.174:6811/967866068', 'heartbeat_back_addr': '172.21.15.174:6815/967866068', 'heartbeat_front_addr': '172.21.15.174:6813/967866068', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e1f4797f-31c3-4292-a4a1-88fa0ab64f9a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6816', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6817', 'nonce': 3985312519}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6818', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6820', 'nonce': 3985312519}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6827', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6829', 'nonce': 3985312519}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6822', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6825', 'nonce': 3985312519}]}, 'public_addr': '172.21.15.174:6817/3985312519', 'cluster_addr': '172.21.15.174:6820/3985312519', 'heartbeat_back_addr': '172.21.15.174:6829/3985312519', 'heartbeat_front_addr': '172.21.15.174:6825/3985312519', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ea3e888e-c4c6-444a-8ebe-e26cddb2084b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6819', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6821', 'nonce': 3676139844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6823', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6824', 'nonce': 3676139844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6830', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6831', 'nonce': 3676139844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6826', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6828', 'nonce': 3676139844}]}, 'public_addr': '172.21.15.174:6821/3676139844', 'cluster_addr': '172.21.15.174:6824/3676139844', 'heartbeat_back_addr': '172.21.15.174:6831/3676139844', 'heartbeat_front_addr': '172.21.15.174:6828/3676139844', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bb0252a5-3aa1-43f0-be38-37413819720b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6800', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6801', 'nonce': 339941177}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6802', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6803', 'nonce': 339941177}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6806', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6807', 'nonce': 339941177}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6804', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6805', 'nonce': 339941177}]}, 'public_addr': '172.21.15.174:6801/339941177', 'cluster_addr': '172.21.15.174:6803/339941177', 'heartbeat_back_addr': '172.21.15.174:6807/339941177', 'heartbeat_front_addr': '172.21.15.174:6805/339941177', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.304258+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.336924+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.359766+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.384687+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.291288+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.318479+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.313564+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.232618+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-16T19:29:54.172 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:29:54.172 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:54.172 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:54.173 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:29:54.173 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:54.187+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:54.270 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:54.288+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:54.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.332+0000 7fd3b05ac640 1 Processor -- start 2024-03-16T19:29:54.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.332+0000 7fd3b05ac640 1 -- start start 2024-03-16T19:29:54.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.333+0000 7fd3b05ac640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3a80cb2b0 0x7fd3a8163fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.333+0000 7fd3b05ac640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a81590a0 con 0x7fd3a80c3610 2024-03-16T19:29:54.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.333+0000 7fd3b05ac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a81595d0 con 0x7fd3a8152840 2024-03-16T19:29:54.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.333+0000 7fd3b05ac640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80c4260 con 0x7fd3a80cb2b0 2024-03-16T19:29:54.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.333+0000 7fd3aeb49640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd3a80cb2b0 0x7fd3a8163fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.333+0000 7fd3adb47640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd3a80c3610 0x7fd3a80caba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:50014/0 (socket says 172.21.15.172:50014) 2024-03-16T19:29:54.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.333+0000 7fd3adb47640 1 -- 172.21.15.172:0/3502185786 learned_addr learned my addr 172.21.15.172:0/3502185786 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:54.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.334+0000 7fd3ad346640 1 -- 172.21.15.172:0/3502185786 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859919115 0 0) 0x7fd3a81595d0 con 0x7fd3a8152840 2024-03-16T19:29:54.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.334+0000 7fd3aeb49640 1 -- 172.21.15.172:0/3502185786 >> v1:172.21.15.174:6789/0 conn(0x7fd3a8152840 legacy=0x7fd3a806e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.334+0000 7fd3aeb49640 1 -- 172.21.15.172:0/3502185786 >> v1:172.21.15.172:6789/0 conn(0x7fd3a80c3610 legacy=0x7fd3a80caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.334+0000 7fd3aeb49640 1 -- 172.21.15.172:0/3502185786 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80cbb90 con 0x7fd3a80cb2b0 2024-03-16T19:29:54.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.334+0000 7fd3aeb49640 1 --2- 172.21.15.172:0/3502185786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80cb2b0 0x7fd3a8163fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009d90 tx=0x7fd3a002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d08696f928f34f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:54.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3ad346640 1 -- 172.21.15.172:0/3502185786 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a002cab0 con 0x7fd3a80cb2b0 2024-03-16T19:29:54.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3ad346640 1 -- 172.21.15.172:0/3502185786 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a002cc10 con 0x7fd3a80cb2b0 2024-03-16T19:29:54.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3ad346640 1 -- 172.21.15.172:0/3502185786 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0034440 con 0x7fd3a80cb2b0 2024-03-16T19:29:54.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3b05ac640 1 -- 172.21.15.172:0/3502185786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80cb2b0 msgr2=0x7fd3a8163fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3b05ac640 1 --2- 172.21.15.172:0/3502185786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80cb2b0 0x7fd3a8163fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009d90 tx=0x7fd3a002bd80 comp rx=0 tx=0).stop 2024-03-16T19:29:54.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3b05ac640 1 -- 172.21.15.172:0/3502185786 shutdown_connections 2024-03-16T19:29:54.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3b05ac640 1 --2- 172.21.15.172:0/3502185786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80cb2b0 0x7fd3a8163fb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3b05ac640 1 -- 172.21.15.172:0/3502185786 >> 172.21.15.172:0/3502185786 conn(0x7fd3a8070860 msgr2=0x7fd3a8070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:54.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3b05ac640 1 -- 172.21.15.172:0/3502185786 shutdown_connections 2024-03-16T19:29:54.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.335+0000 7fd3b05ac640 1 -- 172.21.15.172:0/3502185786 wait complete. 2024-03-16T19:29:54.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.336+0000 7fd3b05ac640 1 Processor -- start 2024-03-16T19:29:54.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.336+0000 7fd3b05ac640 1 -- start start 2024-03-16T19:29:54.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3b05ac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80c3610 0x7fd3a8157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3b05ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a80cb2b0 0x7fd3a8150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3b05ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a8152840 0x7fd3a814e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3b05ac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a81511a0 con 0x7fd3a80c3610 2024-03-16T19:29:54.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3b05ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a81ee2c0 con 0x7fd3a80cb2b0 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3b05ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8150c00 con 0x7fd3a8152840 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3ae348640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80c3610 0x7fd3a8157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3ae348640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80c3610 0x7fd3a8157230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54206/0 (socket says 172.21.15.172:54206) 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3ae348640 1 -- 172.21.15.172:0/4188838061 learned_addr learned my addr 172.21.15.172:0/4188838061 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3aeb49640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a8152840 0x7fd3a814e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.337+0000 7fd3adb47640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a80cb2b0 0x7fd3a8150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd3ae348640 1 -- 172.21.15.172:0/4188838061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a8152840 msgr2=0x7fd3a814e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd3ae348640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a8152840 0x7fd3a814e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd3ae348640 1 -- 172.21.15.172:0/4188838061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a80cb2b0 msgr2=0x7fd3a8150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd3ae348640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a80cb2b0 0x7fd3a8150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd3ae348640 1 -- 172.21.15.172:0/4188838061 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a0009a70 con 0x7fd3a80c3610 2024-03-16T19:29:54.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd3adb47640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a80cb2b0 0x7fd3a8150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:54.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd3ae348640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80c3610 0x7fd3a8157230 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd39800c410 tx=0x7fd39800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:54.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd39effd640 1 -- 172.21.15.172:0/4188838061 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3980035f0 con 0x7fd3a80c3610 2024-03-16T19:29:54.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd39effd640 1 -- 172.21.15.172:0/4188838061 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398003770 con 0x7fd3a80c3610 2024-03-16T19:29:54.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd3b05ac640 1 -- 172.21.15.172:0/4188838061 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a8150ec0 con 0x7fd3a80c3610 2024-03-16T19:29:54.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd39effd640 1 -- 172.21.15.172:0/4188838061 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398003b20 con 0x7fd3a80c3610 2024-03-16T19:29:54.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.338+0000 7fd3b05ac640 1 -- 172.21.15.172:0/4188838061 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a81ee7b0 con 0x7fd3a80c3610 2024-03-16T19:29:54.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.339+0000 7fd39cff9640 1 -- 172.21.15.172:0/4188838061 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd384005290 con 0x7fd3a80c3610 2024-03-16T19:29:54.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.340+0000 7fd39effd640 1 -- 172.21.15.172:0/4188838061 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd398003c80 con 0x7fd3a80c3610 2024-03-16T19:29:54.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.340+0000 7fd39effd640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd38c06b3b0 0x7fd38c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.340+0000 7fd39effd640 1 -- 172.21.15.172:0/4188838061 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5741+0+0 (secure 0 0 0) 0x7fd3980891a0 con 0x7fd3a80c3610 2024-03-16T19:29:54.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.340+0000 7fd3adb47640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd38c06b3b0 0x7fd38c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.324 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:54.324 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:54.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.341+0000 7fd3adb47640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd38c06b3b0 0x7fd38c06d870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8150260 tx=0x7fd3a4007e20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:54.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.345+0000 7fd39effd640 1 -- 172.21.15.172:0/4188838061 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd398053e90 con 0x7fd3a80c3610 2024-03-16T19:29:54.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:29:54.371 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:54.390+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:54.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:29:54.472 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:54.491+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:54.482 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:54.482 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:54.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:29:54.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.522+0000 7fd39cff9640 1 -- 172.21.15.172:0/4188838061 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd384000bc0 con 0x7fd38c06b3b0 2024-03-16T19:29:54.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.529+0000 7fd39effd640 1 -- 172.21.15.172:0/4188838061 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251868 (secure 0 0 0) 0x7fd384000bc0 con 0x7fd38c06b3b0 2024-03-16T19:29:54.513 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:29:54.515 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:29:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 -- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd38c06b3b0 msgr2=0x7fd38c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd38c06b3b0 0x7fd38c06d870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8150260 tx=0x7fd3a4007e20 comp rx=0 tx=0).stop 2024-03-16T19:29:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 -- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80c3610 msgr2=0x7fd3a8157230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80c3610 0x7fd3a8157230 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd39800c410 tx=0x7fd39800c8d0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 -- 172.21.15.172:0/4188838061 shutdown_connections 2024-03-16T19:29:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd38c06b3b0 0x7fd38c06d870 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a8152840 0x7fd3a814e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a80cb2b0 0x7fd3a8150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 --2- 172.21.15.172:0/4188838061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3a80c3610 0x7fd3a8157230 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 -- 172.21.15.172:0/4188838061 >> 172.21.15.172:0/4188838061 conn(0x7fd3a8070860 msgr2=0x7fd3a80c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:54.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 -- 172.21.15.172:0/4188838061 shutdown_connections 2024-03-16T19:29:54.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.533+0000 7fd39cff9640 1 -- 172.21.15.172:0/4188838061 wait complete. 2024-03-16T19:29:54.527 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-16T19:29:54.205361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596098,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":209,"num_write_kb":2436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219532,"kb_used_data":3372,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755188,"statfs":{"total":773094113280,"available":772869312512,"internally_reserved":0,"allocated":3452928,"data_stored":2017301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.786561"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582217+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:29:52.582217+0000","last_peered":"2024-03-16T19:29:52.582217+0000","last_clean":"2024-03-16T19:29:52.582217+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:29:52.582217+0000","last_undegraded":"2024-03-16T19:29:52.582217+0000","last_fullsized":"2024-03-16T19:29:52.582217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582567+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:29:52.582567+0000","last_peered":"2024-03-16T19:29:52.582567+0000","last_clean":"2024-03-16T19:29:52.582567+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:29:52.582567+0000","last_undegraded":"2024-03-16T19:29:52.582567+0000","last_fullsized":"2024-03-16T19:29:52.582567+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582446+0000","last_change":"2024-03-16T19:29:44.412537+0000","last_active":"2024-03-16T19:29:52.582446+0000","last_peered":"2024-03-16T19:29:52.582446+0000","last_clean":"2024-03-16T19:29:52.582446+0000","last_became_active":"2024-03-16T19:29:44.412236+0000","last_became_peered":"2024-03-16T19:29:44.412236+0000","last_unstale":"2024-03-16T19:29:52.582446+0000","last_undegraded":"2024-03-16T19:29:52.582446+0000","last_fullsized":"2024-03-16T19:29:52.582446+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:50:57.180910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582135+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:29:52.582135+0000","last_peered":"2024-03-16T19:29:52.582135+0000","last_clean":"2024-03-16T19:29:52.582135+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:29:52.582135+0000","last_undegraded":"2024-03-16T19:29:52.582135+0000","last_fullsized":"2024-03-16T19:29:52.582135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582276+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:29:52.582276+0000","last_peered":"2024-03-16T19:29:52.582276+0000","last_clean":"2024-03-16T19:29:52.582276+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:29:52.582276+0000","last_undegraded":"2024-03-16T19:29:52.582276+0000","last_fullsized":"2024-03-16T19:29:52.582276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582190+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:29:52.582190+0000","last_peered":"2024-03-16T19:29:52.582190+0000","last_clean":"2024-03-16T19:29:52.582190+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:29:52.582190+0000","last_undegraded":"2024-03-16T19:29:52.582190+0000","last_fullsized":"2024-03-16T19:29:52.582190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582546+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:29:52.582546+0000","last_peered":"2024-03-16T19:29:52.582546+0000","last_clean":"2024-03-16T19:29:52.582546+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:29:52.582546+0000","last_undegraded":"2024-03-16T19:29:52.582546+0000","last_fullsized":"2024-03-16T19:29:52.582546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429418+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:29:52.429418+0000","last_peered":"2024-03-16T19:29:52.429418+0000","last_clean":"2024-03-16T19:29:52.429418+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:29:52.429418+0000","last_undegraded":"2024-03-16T19:29:52.429418+0000","last_fullsized":"2024-03-16T19:29:52.429418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582300+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:29:52.582300+0000","last_peered":"2024-03-16T19:29:52.582300+0000","last_clean":"2024-03-16T19:29:52.582300+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:29:52.582300+0000","last_undegraded":"2024-03-16T19:29:52.582300+0000","last_fullsized":"2024-03-16T19:29:52.582300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429565+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:29:52.429565+0000","last_peered":"2024-03-16T19:29:52.429565+0000","last_clean":"2024-03-16T19:29:52.429565+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:29:52.429565+0000","last_undegraded":"2024-03-16T19:29:52.429565+0000","last_fullsized":"2024-03-16T19:29:52.429565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429973+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:29:52.429973+0000","last_peered":"2024-03-16T19:29:52.429973+0000","last_clean":"2024-03-16T19:29:52.429973+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:29:52.429973+0000","last_undegraded":"2024-03-16T19:29:52.429973+0000","last_fullsized":"2024-03-16T19:29:52.429973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429726+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:29:52.429726+0000","last_peered":"2024-03-16T19:29:52.429726+0000","last_clean":"2024-03-16T19:29:52.429726+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:29:52.429726+0000","last_undegraded":"2024-03-16T19:29:52.429726+0000","last_fullsized":"2024-03-16T19:29:52.429726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429831+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:29:52.429831+0000","last_peered":"2024-03-16T19:29:52.429831+0000","last_clean":"2024-03-16T19:29:52.429831+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:29:52.429831+0000","last_undegraded":"2024-03-16T19:29:52.429831+0000","last_fullsized":"2024-03-16T19:29:52.429831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582256+0000","last_change":"2024-03-16T19:29:44.413594+0000","last_active":"2024-03-16T19:29:52.582256+0000","last_peered":"2024-03-16T19:29:52.582256+0000","last_clean":"2024-03-16T19:29:52.582256+0000","last_became_active":"2024-03-16T19:29:44.413235+0000","last_became_peered":"2024-03-16T19:29:44.413235+0000","last_unstale":"2024-03-16T19:29:52.582256+0000","last_undegraded":"2024-03-16T19:29:52.582256+0000","last_fullsized":"2024-03-16T19:29:52.582256+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:35.778808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582219+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:29:52.582219+0000","last_peered":"2024-03-16T19:29:52.582219+0000","last_clean":"2024-03-16T19:29:52.582219+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:29:52.582219+0000","last_undegraded":"2024-03-16T19:29:52.582219+0000","last_fullsized":"2024-03-16T19:29:52.582219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.581805+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:29:52.581805+0000","last_peered":"2024-03-16T19:29:52.581805+0000","last_clean":"2024-03-16T19:29:52.581805+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:29:52.581805+0000","last_undegraded":"2024-03-16T19:29:52.581805+0000","last_fullsized":"2024-03-16T19:29:52.581805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582898+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:29:52.582898+0000","last_peered":"2024-03-16T19:29:52.582898+0000","last_clean":"2024-03-16T19:29:52.582898+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:29:52.582898+0000","last_undegraded":"2024-03-16T19:29:52.582898+0000","last_fullsized":"2024-03-16T19:29:52.582898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582462+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:29:52.582462+0000","last_peered":"2024-03-16T19:29:52.582462+0000","last_clean":"2024-03-16T19:29:52.582462+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:29:52.582462+0000","last_undegraded":"2024-03-16T19:29:52.582462+0000","last_fullsized":"2024-03-16T19:29:52.582462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582532+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:29:52.582532+0000","last_peered":"2024-03-16T19:29:52.582532+0000","last_clean":"2024-03-16T19:29:52.582532+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:29:52.582532+0000","last_undegraded":"2024-03-16T19:29:52.582532+0000","last_fullsized":"2024-03-16T19:29:52.582532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429332+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:29:52.429332+0000","last_peered":"2024-03-16T19:29:52.429332+0000","last_clean":"2024-03-16T19:29:52.429332+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:29:52.429332+0000","last_undegraded":"2024-03-16T19:29:52.429332+0000","last_fullsized":"2024-03-16T19:29:52.429332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582203+0000","last_change":"2024-03-16T19:29:44.414141+0000","last_active":"2024-03-16T19:29:52.582203+0000","last_peered":"2024-03-16T19:29:52.582203+0000","last_clean":"2024-03-16T19:29:52.582203+0000","last_became_active":"2024-03-16T19:29:44.413873+0000","last_became_peered":"2024-03-16T19:29:44.413873+0000","last_unstale":"2024-03-16T19:29:52.582203+0000","last_undegraded":"2024-03-16T19:29:52.582203+0000","last_fullsized":"2024-03-16T19:29:52.582203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:03:03.291206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582265+0000","last_change":"2024-03-16T19:29:26.358307+0000","last_active":"2024-03-16T19:29:52.582265+0000","last_peered":"2024-03-16T19:29:52.582265+0000","last_clean":"2024-03-16T19:29:52.582265+0000","last_became_active":"2024-03-16T19:29:26.357970+0000","last_became_peered":"2024-03-16T19:29:26.357970+0000","last_unstale":"2024-03-16T19:29:52.582265+0000","last_undegraded":"2024-03-16T19:29:52.582265+0000","last_fullsized":"2024-03-16T19:29:52.582265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:39:33.976868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429530+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:29:52.429530+0000","last_peered":"2024-03-16T19:29:52.429530+0000","last_clean":"2024-03-16T19:29:52.429530+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:29:52.429530+0000","last_undegraded":"2024-03-16T19:29:52.429530+0000","last_fullsized":"2024-03-16T19:29:52.429530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.581909+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:29:52.581909+0000","last_peered":"2024-03-16T19:29:52.581909+0000","last_clean":"2024-03-16T19:29:52.581909+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:29:52.581909+0000","last_undegraded":"2024-03-16T19:29:52.581909+0000","last_fullsized":"2024-03-16T19:29:52.581909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429671+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:29:52.429671+0000","last_peered":"2024-03-16T19:29:52.429671+0000","last_clean":"2024-03-16T19:29:52.429671+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:29:52.429671+0000","last_undegraded":"2024-03-16T19:29:52.429671+0000","last_fullsized":"2024-03-16T19:29:52.429671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429642+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:29:52.429642+0000","last_peered":"2024-03-16T19:29:52.429642+0000","last_clean":"2024-03-16T19:29:52.429642+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:29:52.429642+0000","last_undegraded":"2024-03-16T19:29:52.429642+0000","last_fullsized":"2024-03-16T19:29:52.429642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582534+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:29:52.582534+0000","last_peered":"2024-03-16T19:29:52.582534+0000","last_clean":"2024-03-16T19:29:52.582534+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:29:52.582534+0000","last_undegraded":"2024-03-16T19:29:52.582534+0000","last_fullsized":"2024-03-16T19:29:52.582534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429717+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:29:52.429717+0000","last_peered":"2024-03-16T19:29:52.429717+0000","last_clean":"2024-03-16T19:29:52.429717+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:29:52.429717+0000","last_undegraded":"2024-03-16T19:29:52.429717+0000","last_fullsized":"2024-03-16T19:29:52.429717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429728+0000","last_change":"2024-03-16T19:29:44.412475+0000","last_active":"2024-03-16T19:29:52.429728+0000","last_peered":"2024-03-16T19:29:52.429728+0000","last_clean":"2024-03-16T19:29:52.429728+0000","last_became_active":"2024-03-16T19:29:44.412176+0000","last_became_peered":"2024-03-16T19:29:44.412176+0000","last_unstale":"2024-03-16T19:29:52.429728+0000","last_undegraded":"2024-03-16T19:29:52.429728+0000","last_fullsized":"2024-03-16T19:29:52.429728+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:29:34.284744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582342+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:29:52.582342+0000","last_peered":"2024-03-16T19:29:52.582342+0000","last_clean":"2024-03-16T19:29:52.582342+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:29:52.582342+0000","last_undegraded":"2024-03-16T19:29:52.582342+0000","last_fullsized":"2024-03-16T19:29:52.582342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582145+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:29:52.582145+0000","last_peered":"2024-03-16T19:29:52.582145+0000","last_clean":"2024-03-16T19:29:52.582145+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:29:52.582145+0000","last_undegraded":"2024-03-16T19:29:52.582145+0000","last_fullsized":"2024-03-16T19:29:52.582145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429117+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:29:52.429117+0000","last_peered":"2024-03-16T19:29:52.429117+0000","last_clean":"2024-03-16T19:29:52.429117+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:29:52.429117+0000","last_undegraded":"2024-03-16T19:29:52.429117+0000","last_fullsized":"2024-03-16T19:29:52.429117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429474+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:29:52.429474+0000","last_peered":"2024-03-16T19:29:52.429474+0000","last_clean":"2024-03-16T19:29:52.429474+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:29:52.429474+0000","last_undegraded":"2024-03-16T19:29:52.429474+0000","last_fullsized":"2024-03-16T19:29:52.429474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.430004+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:29:52.430004+0000","last_peered":"2024-03-16T19:29:52.430004+0000","last_clean":"2024-03-16T19:29:52.430004+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:29:52.430004+0000","last_undegraded":"2024-03-16T19:29:52.430004+0000","last_fullsized":"2024-03-16T19:29:52.430004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.430285+0000","last_change":"2024-03-16T19:29:44.412541+0000","last_active":"2024-03-16T19:29:52.430285+0000","last_peered":"2024-03-16T19:29:52.430285+0000","last_clean":"2024-03-16T19:29:52.430285+0000","last_became_active":"2024-03-16T19:29:44.412194+0000","last_became_peered":"2024-03-16T19:29:44.412194+0000","last_unstale":"2024-03-16T19:29:52.430285+0000","last_undegraded":"2024-03-16T19:29:52.430285+0000","last_fullsized":"2024-03-16T19:29:52.430285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:51:50.130592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429234+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:29:52.429234+0000","last_peered":"2024-03-16T19:29:52.429234+0000","last_clean":"2024-03-16T19:29:52.429234+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:29:52.429234+0000","last_undegraded":"2024-03-16T19:29:52.429234+0000","last_fullsized":"2024-03-16T19:29:52.429234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429739+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:29:52.429739+0000","last_peered":"2024-03-16T19:29:52.429739+0000","last_clean":"2024-03-16T19:29:52.429739+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:29:52.429739+0000","last_undegraded":"2024-03-16T19:29:52.429739+0000","last_fullsized":"2024-03-16T19:29:52.429739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429461+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:29:52.429461+0000","last_peered":"2024-03-16T19:29:52.429461+0000","last_clean":"2024-03-16T19:29:52.429461+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:29:52.429461+0000","last_undegraded":"2024-03-16T19:29:52.429461+0000","last_fullsized":"2024-03-16T19:29:52.429461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.581986+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:29:52.581986+0000","last_peered":"2024-03-16T19:29:52.581986+0000","last_clean":"2024-03-16T19:29:52.581986+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:29:52.581986+0000","last_undegraded":"2024-03-16T19:29:52.581986+0000","last_fullsized":"2024-03-16T19:29:52.581986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582106+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:29:52.582106+0000","last_peered":"2024-03-16T19:29:52.582106+0000","last_clean":"2024-03-16T19:29:52.582106+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:29:52.582106+0000","last_undegraded":"2024-03-16T19:29:52.582106+0000","last_fullsized":"2024-03-16T19:29:52.582106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429312+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:29:52.429312+0000","last_peered":"2024-03-16T19:29:52.429312+0000","last_clean":"2024-03-16T19:29:52.429312+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:29:52.429312+0000","last_undegraded":"2024-03-16T19:29:52.429312+0000","last_fullsized":"2024-03-16T19:29:52.429312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582680+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:29:52.582680+0000","last_peered":"2024-03-16T19:29:52.582680+0000","last_clean":"2024-03-16T19:29:52.582680+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:29:52.582680+0000","last_undegraded":"2024-03-16T19:29:52.582680+0000","last_fullsized":"2024-03-16T19:29:52.582680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582757+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:29:52.582757+0000","last_peered":"2024-03-16T19:29:52.582757+0000","last_clean":"2024-03-16T19:29:52.582757+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:29:52.582757+0000","last_undegraded":"2024-03-16T19:29:52.582757+0000","last_fullsized":"2024-03-16T19:29:52.582757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582869+0000","last_change":"2024-03-16T19:29:44.413558+0000","last_active":"2024-03-16T19:29:52.582869+0000","last_peered":"2024-03-16T19:29:52.582869+0000","last_clean":"2024-03-16T19:29:52.582869+0000","last_became_active":"2024-03-16T19:29:44.413310+0000","last_became_peered":"2024-03-16T19:29:44.413310+0000","last_unstale":"2024-03-16T19:29:52.582869+0000","last_undegraded":"2024-03-16T19:29:52.582869+0000","last_fullsized":"2024-03-16T19:29:52.582869+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:25:53.846575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582570+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:29:52.582570+0000","last_peered":"2024-03-16T19:29:52.582570+0000","last_clean":"2024-03-16T19:29:52.582570+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:29:52.582570+0000","last_undegraded":"2024-03-16T19:29:52.582570+0000","last_fullsized":"2024-03-16T19:29:52.582570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429809+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:29:52.429809+0000","last_peered":"2024-03-16T19:29:52.429809+0000","last_clean":"2024-03-16T19:29:52.429809+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:29:52.429809+0000","last_undegraded":"2024-03-16T19:29:52.429809+0000","last_fullsized":"2024-03-16T19:29:52.429809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'9","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429413+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:29:52.429413+0000","last_peered":"2024-03-16T19:29:52.429413+0000","last_clean":"2024-03-16T19:29:52.429413+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:29:52.429413+0000","last_undegraded":"2024-03-16T19:29:52.429413+0000","last_fullsized":"2024-03-16T19:29:52.429413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582283+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:29:52.582283+0000","last_peered":"2024-03-16T19:29:52.582283+0000","last_clean":"2024-03-16T19:29:52.582283+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:29:52.582283+0000","last_undegraded":"2024-03-16T19:29:52.582283+0000","last_fullsized":"2024-03-16T19:29:52.582283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.430314+0000","last_change":"2024-03-16T19:29:44.412731+0000","last_active":"2024-03-16T19:29:52.430314+0000","last_peered":"2024-03-16T19:29:52.430314+0000","last_clean":"2024-03-16T19:29:52.430314+0000","last_became_active":"2024-03-16T19:29:44.412174+0000","last_became_peered":"2024-03-16T19:29:44.412174+0000","last_unstale":"2024-03-16T19:29:52.430314+0000","last_undegraded":"2024-03-16T19:29:52.430314+0000","last_fullsized":"2024-03-16T19:29:52.430314+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:57.962644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582051+0000","last_change":"2024-03-16T19:29:44.412524+0000","last_active":"2024-03-16T19:29:52.582051+0000","last_peered":"2024-03-16T19:29:52.582051+0000","last_clean":"2024-03-16T19:29:52.582051+0000","last_became_active":"2024-03-16T19:29:44.412200+0000","last_became_peered":"2024-03-16T19:29:44.412200+0000","last_unstale":"2024-03-16T19:29:52.582051+0000","last_undegraded":"2024-03-16T19:29:52.582051+0000","last_fullsized":"2024-03-16T19:29:52.582051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:52.636929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582413+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:29:52.582413+0000","last_peered":"2024-03-16T19:29:52.582413+0000","last_clean":"2024-03-16T19:29:52.582413+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:29:52.582413+0000","last_undegraded":"2024-03-16T19:29:52.582413+0000","last_fullsized":"2024-03-16T19:29:52.582413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582355+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:29:52.582355+0000","last_peered":"2024-03-16T19:29:52.582355+0000","last_clean":"2024-03-16T19:29:52.582355+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:29:52.582355+0000","last_undegraded":"2024-03-16T19:29:52.582355+0000","last_fullsized":"2024-03-16T19:29:52.582355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429882+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:29:52.429882+0000","last_peered":"2024-03-16T19:29:52.429882+0000","last_clean":"2024-03-16T19:29:52.429882+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:29:52.429882+0000","last_undegraded":"2024-03-16T19:29:52.429882+0000","last_fullsized":"2024-03-16T19:29:52.429882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429117+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:29:52.429117+0000","last_peered":"2024-03-16T19:29:52.429117+0000","last_clean":"2024-03-16T19:29:52.429117+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:29:52.429117+0000","last_undegraded":"2024-03-16T19:29:52.429117+0000","last_fullsized":"2024-03-16T19:29:52.429117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582036+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:29:52.582036+0000","last_peered":"2024-03-16T19:29:52.582036+0000","last_clean":"2024-03-16T19:29:52.582036+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:29:52.582036+0000","last_undegraded":"2024-03-16T19:29:52.582036+0000","last_fullsized":"2024-03-16T19:29:52.582036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582622+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:29:52.582622+0000","last_peered":"2024-03-16T19:29:52.582622+0000","last_clean":"2024-03-16T19:29:52.582622+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:29:52.582622+0000","last_undegraded":"2024-03-16T19:29:52.582622+0000","last_fullsized":"2024-03-16T19:29:52.582622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.430163+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:29:52.430163+0000","last_peered":"2024-03-16T19:29:52.430163+0000","last_clean":"2024-03-16T19:29:52.430163+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:29:52.430163+0000","last_undegraded":"2024-03-16T19:29:52.430163+0000","last_fullsized":"2024-03-16T19:29:52.430163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582189+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:29:52.582189+0000","last_peered":"2024-03-16T19:29:52.582189+0000","last_clean":"2024-03-16T19:29:52.582189+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:29:52.582189+0000","last_undegraded":"2024-03-16T19:29:52.582189+0000","last_fullsized":"2024-03-16T19:29:52.582189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582463+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:29:52.582463+0000","last_peered":"2024-03-16T19:29:52.582463+0000","last_clean":"2024-03-16T19:29:52.582463+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:29:52.582463+0000","last_undegraded":"2024-03-16T19:29:52.582463+0000","last_fullsized":"2024-03-16T19:29:52.582463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582398+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:29:52.582398+0000","last_peered":"2024-03-16T19:29:52.582398+0000","last_clean":"2024-03-16T19:29:52.582398+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:29:52.582398+0000","last_undegraded":"2024-03-16T19:29:52.582398+0000","last_fullsized":"2024-03-16T19:29:52.582398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582647+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:29:52.582647+0000","last_peered":"2024-03-16T19:29:52.582647+0000","last_clean":"2024-03-16T19:29:52.582647+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:29:52.582647+0000","last_undegraded":"2024-03-16T19:29:52.582647+0000","last_fullsized":"2024-03-16T19:29:52.582647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582284+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:29:52.582284+0000","last_peered":"2024-03-16T19:29:52.582284+0000","last_clean":"2024-03-16T19:29:52.582284+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:29:52.582284+0000","last_undegraded":"2024-03-16T19:29:52.582284+0000","last_fullsized":"2024-03-16T19:29:52.582284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582383+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:29:52.582383+0000","last_peered":"2024-03-16T19:29:52.582383+0000","last_clean":"2024-03-16T19:29:52.582383+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:29:52.582383+0000","last_undegraded":"2024-03-16T19:29:52.582383+0000","last_fullsized":"2024-03-16T19:29:52.582383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582357+0000","last_change":"2024-03-16T19:29:26.364742+0000","last_active":"2024-03-16T19:29:52.582357+0000","last_peered":"2024-03-16T19:29:52.582357+0000","last_clean":"2024-03-16T19:29:52.582357+0000","last_became_active":"2024-03-16T19:29:26.364328+0000","last_became_peered":"2024-03-16T19:29:26.364328+0000","last_unstale":"2024-03-16T19:29:52.582357+0000","last_undegraded":"2024-03-16T19:29:52.582357+0000","last_fullsized":"2024-03-16T19:29:52.582357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:18.877774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":180,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.581894+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:29:52.581894+0000","last_peered":"2024-03-16T19:29:52.581894+0000","last_clean":"2024-03-16T19:29:52.581894+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:29:52.581894+0000","last_undegraded":"2024-03-16T19:29:52.581894+0000","last_fullsized":"2024-03-16T19:29:52.581894+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.581986+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:29:52.581986+0000","last_peered":"2024-03-16T19:29:52.581986+0000","last_clean":"2024-03-16T19:29:52.581986+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:29:52.581986+0000","last_undegraded":"2024-03-16T19:29:52.581986+0000","last_fullsized":"2024-03-16T19:29:52.581986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429813+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:29:52.429813+0000","last_peered":"2024-03-16T19:29:52.429813+0000","last_clean":"2024-03-16T19:29:52.429813+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:29:52.429813+0000","last_undegraded":"2024-03-16T19:29:52.429813+0000","last_fullsized":"2024-03-16T19:29:52.429813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.581775+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:29:52.581775+0000","last_peered":"2024-03-16T19:29:52.581775+0000","last_clean":"2024-03-16T19:29:52.581775+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:29:52.581775+0000","last_undegraded":"2024-03-16T19:29:52.581775+0000","last_fullsized":"2024-03-16T19:29:52.581775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.430052+0000","last_change":"2024-03-16T19:29:44.412505+0000","last_active":"2024-03-16T19:29:52.430052+0000","last_peered":"2024-03-16T19:29:52.430052+0000","last_clean":"2024-03-16T19:29:52.430052+0000","last_became_active":"2024-03-16T19:29:44.412221+0000","last_became_peered":"2024-03-16T19:29:44.412221+0000","last_unstale":"2024-03-16T19:29:52.430052+0000","last_undegraded":"2024-03-16T19:29:52.430052+0000","last_fullsized":"2024-03-16T19:29:52.430052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:14.739638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582301+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:29:52.582301+0000","last_peered":"2024-03-16T19:29:52.582301+0000","last_clean":"2024-03-16T19:29:52.582301+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:29:52.582301+0000","last_undegraded":"2024-03-16T19:29:52.582301+0000","last_fullsized":"2024-03-16T19:29:52.582301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582191+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:29:52.582191+0000","last_peered":"2024-03-16T19:29:52.582191+0000","last_clean":"2024-03-16T19:29:52.582191+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:29:52.582191+0000","last_undegraded":"2024-03-16T19:29:52.582191+0000","last_fullsized":"2024-03-16T19:29:52.582191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582497+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:29:52.582497+0000","last_peered":"2024-03-16T19:29:52.582497+0000","last_clean":"2024-03-16T19:29:52.582497+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:29:52.582497+0000","last_undegraded":"2024-03-16T19:29:52.582497+0000","last_fullsized":"2024-03-16T19:29:52.582497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.430380+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:29:52.430380+0000","last_peered":"2024-03-16T19:29:52.430380+0000","last_clean":"2024-03-16T19:29:52.430380+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:29:52.430380+0000","last_undegraded":"2024-03-16T19:29:52.430380+0000","last_fullsized":"2024-03-16T19:29:52.430380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582435+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:29:52.582435+0000","last_peered":"2024-03-16T19:29:52.582435+0000","last_clean":"2024-03-16T19:29:52.582435+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:29:52.582435+0000","last_undegraded":"2024-03-16T19:29:52.582435+0000","last_fullsized":"2024-03-16T19:29:52.582435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582593+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:29:52.582593+0000","last_peered":"2024-03-16T19:29:52.582593+0000","last_clean":"2024-03-16T19:29:52.582593+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:29:52.582593+0000","last_undegraded":"2024-03-16T19:29:52.582593+0000","last_fullsized":"2024-03-16T19:29:52.582593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582613+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:29:52.582613+0000","last_peered":"2024-03-16T19:29:52.582613+0000","last_clean":"2024-03-16T19:29:52.582613+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:29:52.582613+0000","last_undegraded":"2024-03-16T19:29:52.582613+0000","last_fullsized":"2024-03-16T19:29:52.582613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582383+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:29:52.582383+0000","last_peered":"2024-03-16T19:29:52.582383+0000","last_clean":"2024-03-16T19:29:52.582383+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:29:52.582383+0000","last_undegraded":"2024-03-16T19:29:52.582383+0000","last_fullsized":"2024-03-16T19:29:52.582383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582653+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:29:52.582653+0000","last_peered":"2024-03-16T19:29:52.582653+0000","last_clean":"2024-03-16T19:29:52.582653+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:29:52.582653+0000","last_undegraded":"2024-03-16T19:29:52.582653+0000","last_fullsized":"2024-03-16T19:29:52.582653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429780+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:29:52.429780+0000","last_peered":"2024-03-16T19:29:52.429780+0000","last_clean":"2024-03-16T19:29:52.429780+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:29:52.429780+0000","last_undegraded":"2024-03-16T19:29:52.429780+0000","last_fullsized":"2024-03-16T19:29:52.429780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429334+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:29:52.429334+0000","last_peered":"2024-03-16T19:29:52.429334+0000","last_clean":"2024-03-16T19:29:52.429334+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:29:52.429334+0000","last_undegraded":"2024-03-16T19:29:52.429334+0000","last_fullsized":"2024-03-16T19:29:52.429334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582542+0000","last_change":"2024-03-16T19:29:44.413626+0000","last_active":"2024-03-16T19:29:52.582542+0000","last_peered":"2024-03-16T19:29:52.582542+0000","last_clean":"2024-03-16T19:29:52.582542+0000","last_became_active":"2024-03-16T19:29:44.413174+0000","last_became_peered":"2024-03-16T19:29:44.413174+0000","last_unstale":"2024-03-16T19:29:52.582542+0000","last_undegraded":"2024-03-16T19:29:52.582542+0000","last_fullsized":"2024-03-16T19:29:52.582542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:10:39.043849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.581775+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:29:52.581775+0000","last_peered":"2024-03-16T19:29:52.581775+0000","last_clean":"2024-03-16T19:29:52.581775+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:29:52.581775+0000","last_undegraded":"2024-03-16T19:29:52.581775+0000","last_fullsized":"2024-03-16T19:29:52.581775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.430058+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:29:52.430058+0000","last_peered":"2024-03-16T19:29:52.430058+0000","last_clean":"2024-03-16T19:29:52.430058+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:29:52.430058+0000","last_undegraded":"2024-03-16T19:29:52.430058+0000","last_fullsized":"2024-03-16T19:29:52.430058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582225+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:29:52.582225+0000","last_peered":"2024-03-16T19:29:52.582225+0000","last_clean":"2024-03-16T19:29:52.582225+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:29:52.582225+0000","last_undegraded":"2024-03-16T19:29:52.582225+0000","last_fullsized":"2024-03-16T19:29:52.582225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429556+0000","last_change":"2024-03-16T19:29:26.347388+0000","last_active":"2024-03-16T19:29:52.429556+0000","last_peered":"2024-03-16T19:29:52.429556+0000","last_clean":"2024-03-16T19:29:52.429556+0000","last_became_active":"2024-03-16T19:29:26.346582+0000","last_became_peered":"2024-03-16T19:29:26.346582+0000","last_unstale":"2024-03-16T19:29:52.429556+0000","last_undegraded":"2024-03-16T19:29:52.429556+0000","last_fullsized":"2024-03-16T19:29:52.429556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:05.294895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582450+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:29:52.582450+0000","last_peered":"2024-03-16T19:29:52.582450+0000","last_clean":"2024-03-16T19:29:52.582450+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:29:52.582450+0000","last_undegraded":"2024-03-16T19:29:52.582450+0000","last_fullsized":"2024-03-16T19:29:52.582450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429917+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:29:52.429917+0000","last_peered":"2024-03-16T19:29:52.429917+0000","last_clean":"2024-03-16T19:29:52.429917+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:29:52.429917+0000","last_undegraded":"2024-03-16T19:29:52.429917+0000","last_fullsized":"2024-03-16T19:29:52.429917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582440+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:29:52.582440+0000","last_peered":"2024-03-16T19:29:52.582440+0000","last_clean":"2024-03-16T19:29:52.582440+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:29:52.582440+0000","last_undegraded":"2024-03-16T19:29:52.582440+0000","last_fullsized":"2024-03-16T19:29:52.582440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.429155+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:29:52.429155+0000","last_peered":"2024-03-16T19:29:52.429155+0000","last_clean":"2024-03-16T19:29:52.429155+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:29:52.429155+0000","last_undegraded":"2024-03-16T19:29:52.429155+0000","last_fullsized":"2024-03-16T19:29:52.429155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582519+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:29:52.582519+0000","last_peered":"2024-03-16T19:29:52.582519+0000","last_clean":"2024-03-16T19:29:52.582519+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:29:52.582519+0000","last_undegraded":"2024-03-16T19:29:52.582519+0000","last_fullsized":"2024-03-16T19:29:52.582519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582052+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:29:52.582052+0000","last_peered":"2024-03-16T19:29:52.582052+0000","last_clean":"2024-03-16T19:29:52.582052+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:29:52.582052+0000","last_undegraded":"2024-03-16T19:29:52.582052+0000","last_fullsized":"2024-03-16T19:29:52.582052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582212+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:29:52.582212+0000","last_peered":"2024-03-16T19:29:52.582212+0000","last_clean":"2024-03-16T19:29:52.582212+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:29:52.582212+0000","last_undegraded":"2024-03-16T19:29:52.582212+0000","last_fullsized":"2024-03-16T19:29:52.582212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582433+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:29:52.582433+0000","last_peered":"2024-03-16T19:29:52.582433+0000","last_clean":"2024-03-16T19:29:52.582433+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:29:52.582433+0000","last_undegraded":"2024-03-16T19:29:52.582433+0000","last_fullsized":"2024-03-16T19:29:52.582433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582742+0000","last_change":"2024-03-16T19:29:44.413556+0000","last_active":"2024-03-16T19:29:52.582742+0000","last_peered":"2024-03-16T19:29:52.582742+0000","last_clean":"2024-03-16T19:29:52.582742+0000","last_became_active":"2024-03-16T19:29:44.413300+0000","last_became_peered":"2024-03-16T19:29:44.413300+0000","last_unstale":"2024-03-16T19:29:52.582742+0000","last_undegraded":"2024-03-16T19:29:52.582742+0000","last_fullsized":"2024-03-16T19:29:52.582742+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:27:31.056332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.430117+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:29:52.430117+0000","last_peered":"2024-03-16T19:29:52.430117+0000","last_clean":"2024-03-16T19:29:52.430117+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:29:52.430117+0000","last_undegraded":"2024-03-16T19:29:52.430117+0000","last_fullsized":"2024-03-16T19:29:52.430117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.430115+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:29:52.430115+0000","last_peered":"2024-03-16T19:29:52.430115+0000","last_clean":"2024-03-16T19:29:52.430115+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:29:52.430115+0000","last_undegraded":"2024-03-16T19:29:52.430115+0000","last_fullsized":"2024-03-16T19:29:52.430115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-16T19:29:52.582507+0000","last_change":"2024-03-16T19:29:44.414207+0000","last_active":"2024-03-16T19:29:52.582507+0000","last_peered":"2024-03-16T19:29:52.582507+0000","last_clean":"2024-03-16T19:29:52.582507+0000","last_became_active":"2024-03-16T19:29:44.413852+0000","last_became_peered":"2024-03-16T19:29:44.413852+0000","last_unstale":"2024-03-16T19:29:52.582507+0000","last_undegraded":"2024-03-16T19:29:52.582507+0000","last_fullsized":"2024-03-16T19:29:52.582507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:25.285467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5730,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":11460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":696316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":110298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":99515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":105492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":110298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":99425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":105526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":690431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:29:54.530 INFO:tasks.thrashosds.thrasher:Setting 3.20 to [7, 6, 1, 3] 2024-03-16T19:29:54.531 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.20', '7', '6', '1', '3'] 2024-03-16T19:29:54.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.20 7 6 1 3 2024-03-16T19:29:54.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.551+0000 7fdaae5d7640 1 Processor -- start 2024-03-16T19:29:54.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.551+0000 7fdaae5d7640 1 -- start start 2024-03-16T19:29:54.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaae5d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa806ba30 0x7fdaa8166280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaae5d7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa80719e0 con 0x7fdaa8157e50 2024-03-16T19:29:54.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaae5d7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa80c7b50 con 0x7fdaa8158230 2024-03-16T19:29:54.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaae5d7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa806aed0 con 0x7fdaa806ba30 2024-03-16T19:29:54.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaacb74640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa806ba30 0x7fdaa8166280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaa7fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdaa8157e50 0x7fdaa80ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:50028/0 (socket says 172.21.15.172:50028) 2024-03-16T19:29:54.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaa7fff640 1 -- 172.21.15.172:0/1141581127 learned_addr learned my addr 172.21.15.172:0/1141581127 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:54.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaa77fe640 1 --1- 172.21.15.172:0/1141581127 >> v1:172.21.15.174:6789/0 conn(0x7fdaa8158230 0x7fdaa806b320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:52874/0 (socket says 172.21.15.172:52874) 2024-03-16T19:29:54.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaa6ffd640 1 -- 172.21.15.172:0/1141581127 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348451882 0 0) 0x7fdaa80c7b50 con 0x7fdaa8158230 2024-03-16T19:29:54.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaacb74640 1 -- 172.21.15.172:0/1141581127 >> v1:172.21.15.174:6789/0 conn(0x7fdaa8158230 legacy=0x7fdaa806b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaacb74640 1 -- 172.21.15.172:0/1141581127 >> v1:172.21.15.172:6789/0 conn(0x7fdaa8157e50 legacy=0x7fdaa80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.552+0000 7fdaacb74640 1 -- 172.21.15.172:0/1141581127 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa80c7e10 con 0x7fdaa806ba30 2024-03-16T19:29:54.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.553+0000 7fdaacb74640 1 --2- 172.21.15.172:0/1141581127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa806ba30 0x7fdaa8166280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009dc0 tx=0x7fdaa002bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff193825a50984ca server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.553+0000 7fdaa6ffd640 1 -- 172.21.15.172:0/1141581127 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa002ca80 con 0x7fdaa806ba30 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.553+0000 7fdaa6ffd640 1 -- 172.21.15.172:0/1141581127 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa002cbe0 con 0x7fdaa806ba30 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.553+0000 7fdaa6ffd640 1 -- 172.21.15.172:0/1141581127 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0034440 con 0x7fdaa806ba30 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.554+0000 7fdaae5d7640 1 -- 172.21.15.172:0/1141581127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa806ba30 msgr2=0x7fdaa8166280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.554+0000 7fdaae5d7640 1 --2- 172.21.15.172:0/1141581127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa806ba30 0x7fdaa8166280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009dc0 tx=0x7fdaa002bd50 comp rx=0 tx=0).stop 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.554+0000 7fdaae5d7640 1 -- 172.21.15.172:0/1141581127 shutdown_connections 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.554+0000 7fdaae5d7640 1 --2- 172.21.15.172:0/1141581127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa806ba30 0x7fdaa8166280 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.554+0000 7fdaae5d7640 1 -- 172.21.15.172:0/1141581127 >> 172.21.15.172:0/1141581127 conn(0x7fdaa806db70 msgr2=0x7fdaa806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.554+0000 7fdaae5d7640 1 -- 172.21.15.172:0/1141581127 shutdown_connections 2024-03-16T19:29:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.554+0000 7fdaae5d7640 1 -- 172.21.15.172:0/1141581127 wait complete. 2024-03-16T19:29:54.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.555+0000 7fdaae5d7640 1 Processor -- start 2024-03-16T19:29:54.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.555+0000 7fdaae5d7640 1 -- start start 2024-03-16T19:29:54.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaae5d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa806ba30 0x7fdaa8150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaae5d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8157e50 0x7fdaa814edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaae5d7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa8158230 0x7fdaa815d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaae5d7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa8150f40 con 0x7fdaa8158230 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaae5d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa814e4a0 con 0x7fdaa806ba30 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaae5d7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa814f410 con 0x7fdaa8157e50 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaacb74640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa8158230 0x7fdaa815d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaa7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa806ba30 0x7fdaa8150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaa77fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8157e50 0x7fdaa814edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaacb74640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa8158230 0x7fdaa815d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54208/0 (socket says 172.21.15.172:54208) 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaa7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa806ba30 0x7fdaa8150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42970/0 (socket says 172.21.15.172:42970) 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaacb74640 1 -- 172.21.15.172:0/4264877473 learned_addr learned my addr 172.21.15.172:0/4264877473 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:54.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaa77fe640 1 -- 172.21.15.172:0/4264877473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa806ba30 msgr2=0x7fdaa8150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaa77fe640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa806ba30 0x7fdaa8150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaa77fe640 1 -- 172.21.15.172:0/4264877473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa8158230 msgr2=0x7fdaa815d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaa77fe640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa8158230 0x7fdaa815d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaa77fe640 1 -- 172.21.15.172:0/4264877473 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa0009a70 con 0x7fdaa8157e50 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.556+0000 7fdaa7fff640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa806ba30 0x7fdaa8150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.557+0000 7fdaa77fe640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8157e50 0x7fdaa814edc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00c3e0 tx=0x7fda9c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.557+0000 7fdaa4ff9640 1 -- 172.21.15.172:0/4264877473 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c0074e0 con 0x7fdaa8157e50 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.557+0000 7fdaa4ff9640 1 -- 172.21.15.172:0/4264877473 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c007660 con 0x7fdaa8157e50 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.557+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa814fcb0 con 0x7fdaa8157e50 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.557+0000 7fdaa4ff9640 1 -- 172.21.15.172:0/4264877473 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c004a20 con 0x7fdaa8157e50 2024-03-16T19:29:54.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.557+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa814e9d0 con 0x7fdaa8157e50 2024-03-16T19:29:54.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.558+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa814fe40 con 0x7fdaa8157e50 2024-03-16T19:29:54.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.558+0000 7fdaa4ff9640 1 -- 172.21.15.172:0/4264877473 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fda9c004b80 con 0x7fdaa8157e50 2024-03-16T19:29:54.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.559+0000 7fdaa4ff9640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fda9406b3b0 0x7fda9406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.559+0000 7fdaa4ff9640 1 -- 172.21.15.172:0/4264877473 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5741+0+0 (secure 0 0 0) 0x7fda9c089860 con 0x7fdaa8157e50 2024-03-16T19:29:54.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.559+0000 7fdaa7fff640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fda9406b3b0 0x7fda9406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.559+0000 7fdaa7fff640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fda9406b3b0 0x7fda9406d870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fda98000f60 tx=0x7fda98007e00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:54.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.563+0000 7fdaa4ff9640 1 -- 172.21.15.172:0/4264877473 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda9c0545e0 con 0x7fdaa8157e50 2024-03-16T19:29:54.574 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:54.591+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:54.603 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:54.603 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:54.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:29:54.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.659+0000 7f66a6fb6640 1 Processor -- start 2024-03-16T19:29:54.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.659+0000 7f66a6fb6640 1 -- start start 2024-03-16T19:29:54.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.660+0000 7f66a6fb6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a006dc80 0x7f66a0166420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.660+0000 7f66a6fb6640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a006d5c0 con 0x7f66a00cbac0 2024-03-16T19:29:54.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.660+0000 7f66a6fb6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a006c080 con 0x7f66a0158120 2024-03-16T19:29:54.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.660+0000 7f66a6fb6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a015b280 con 0x7f66a006dc80 2024-03-16T19:29:54.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.660+0000 7f669ffff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f66a00cbac0 0x7f66a0070bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:50030/0 (socket says 172.21.15.172:50030) 2024-03-16T19:29:54.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.660+0000 7f669ffff640 1 -- 172.21.15.172:0/4129417553 learned_addr learned my addr 172.21.15.172:0/4129417553 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:29:54.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.660+0000 7f66a5553640 1 --2- 172.21.15.172:0/4129417553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a006dc80 0x7f66a0166420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.661+0000 7f669f7fe640 1 -- 172.21.15.172:0/4129417553 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054650923 0 0) 0x7f66a006d5c0 con 0x7f66a00cbac0 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.661+0000 7f669f7fe640 1 -- 172.21.15.172:0/4129417553 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6688003520 con 0x7f66a00cbac0 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.661+0000 7f669f7fe640 1 -- 172.21.15.172:0/4129417553 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777889731 0 0) 0x7f66a006c080 con 0x7f66a0158120 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.661+0000 7f66a5553640 1 -- 172.21.15.172:0/4129417553 >> v1:172.21.15.174:6789/0 conn(0x7f66a0158120 legacy=0x7f66a015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.661+0000 7f66a5553640 1 -- 172.21.15.172:0/4129417553 >> v1:172.21.15.172:6789/0 conn(0x7f66a00cbac0 legacy=0x7f66a0070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.661+0000 7f66a5553640 1 -- 172.21.15.172:0/4129417553 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a015ec70 con 0x7f66a006dc80 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.661+0000 7f66a5553640 1 --2- 172.21.15.172:0/4129417553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a006dc80 0x7f66a0166420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66900097d0 tx=0x7f669002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1ec93ddde0fc9cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f669f7fe640 1 -- 172.21.15.172:0/4129417553 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690031890 con 0x7f66a006dc80 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f669f7fe640 1 -- 172.21.15.172:0/4129417553 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66900319f0 con 0x7f66a006dc80 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f669f7fe640 1 -- 172.21.15.172:0/4129417553 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690031d80 con 0x7f66a006dc80 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f66a6fb6640 1 -- 172.21.15.172:0/4129417553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a006dc80 msgr2=0x7f66a0166420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f66a6fb6640 1 --2- 172.21.15.172:0/4129417553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a006dc80 0x7f66a0166420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66900097d0 tx=0x7f669002b930 comp rx=0 tx=0).stop 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f66a6fb6640 1 -- 172.21.15.172:0/4129417553 shutdown_connections 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f66a6fb6640 1 --2- 172.21.15.172:0/4129417553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a006dc80 0x7f66a0166420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f66a6fb6640 1 -- 172.21.15.172:0/4129417553 >> 172.21.15.172:0/4129417553 conn(0x7f66a006f940 msgr2=0x7f66a006fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:54.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f66a6fb6640 1 -- 172.21.15.172:0/4129417553 shutdown_connections 2024-03-16T19:29:54.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.662+0000 7f66a6fb6640 1 -- 172.21.15.172:0/4129417553 wait complete. 2024-03-16T19:29:54.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.663+0000 7f66a6fb6640 1 Processor -- start 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.663+0000 7f66a6fb6640 1 -- start start 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.664+0000 7f66a6fb6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66a006dc80 0x7f66a00c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.664+0000 7f66a6fb6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66a0158120 0x7f66a00cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.664+0000 7f66a6fb6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a015bcd0 0x7f66a01f3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.664+0000 7f66a6fb6640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00c7740 con 0x7f66a0158120 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.664+0000 7f66a6fb6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a01f0b70 con 0x7f66a006dc80 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.664+0000 7f66a6fb6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00c8c40 con 0x7f66a015bcd0 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.664+0000 7f669ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66a0158120 0x7f66a00cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66a0158120 0x7f66a00cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54212/0 (socket says 172.21.15.172:54212) 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669ffff640 1 -- 172.21.15.172:0/3318134614 learned_addr learned my addr 172.21.15.172:0/3318134614 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f66a5553640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a015bcd0 0x7f66a01f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f66a4d52640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66a006dc80 0x7f66a00c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669ffff640 1 -- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a015bcd0 msgr2=0x7f66a01f3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669ffff640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a015bcd0 0x7f66a01f3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669ffff640 1 -- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66a006dc80 msgr2=0x7f66a00c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669ffff640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66a006dc80 0x7f66a00c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669ffff640 1 -- 172.21.15.172:0/3318134614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6690009400 con 0x7f66a0158120 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f66a4d52640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66a006dc80 0x7f66a00c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f66a5553640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a015bcd0 0x7f66a01f3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669ffff640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66a0158120 0x7f66a00cd530 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f669400c460 tx=0x7f669400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669d7fa640 1 -- 172.21.15.172:0/3318134614 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694005ce0 con 0x7f66a0158120 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669d7fa640 1 -- 172.21.15.172:0/3318134614 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6694005e40 con 0x7f66a0158120 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a015c690 con 0x7f66a0158120 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f669d7fa640 1 -- 172.21.15.172:0/3318134614 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694007290 con 0x7f66a0158120 2024-03-16T19:29:54.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.665+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a01f0520 con 0x7f66a0158120 2024-03-16T19:29:54.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.666+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a0068d20 con 0x7f66a0158120 2024-03-16T19:29:54.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.667+0000 7f669d7fa640 1 -- 172.21.15.172:0/3318134614 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f66940073f0 con 0x7f66a0158120 2024-03-16T19:29:54.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.667+0000 7f669d7fa640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f668c06b3b0 0x7f668c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:54.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.668+0000 7f669d7fa640 1 -- 172.21.15.172:0/3318134614 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f6694089200 con 0x7f66a0158120 2024-03-16T19:29:54.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.668+0000 7f66a4d52640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f668c06b3b0 0x7f668c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:54.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.668+0000 7f66a4d52640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f668c06b3b0 0x7f668c06d870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f66980012e0 tx=0x7f6698009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:54.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.672+0000 7f669d7fa640 1 -- 172.21.15.172:0/3318134614 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6694053e60 con 0x7f66a0158120 2024-03-16T19:29:54.672 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:54.691+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:54.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.755+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fdaa8075a70 con 0x7fdaa8157e50 2024-03-16T19:29:54.750 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:54.750 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:54.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:29:54.800 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:54.819+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:54.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:54.865+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "3.20", "id": [7, 6, 1, 3]} v 0) v1 -- 0x7f66a00681c0 con 0x7f66a0158120 2024-03-16T19:29:54.874 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:54.893+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:54.907 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:54.907 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:54.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:29:54.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:29:54.976 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:54.995+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:55.065 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:55.065 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:55.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:29:55.076 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:55.095+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:55.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.118+0000 7f73a468f640 1 Processor -- start 2024-03-16T19:29:55.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.118+0000 7f73a468f640 1 -- start start 2024-03-16T19:29:55.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.119+0000 7f73a468f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f739c154ed0 0x7f739c15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.119+0000 7f73a468f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f739c0b3630 0x7f739c0cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.119+0000 7f73a468f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739c0b4270 con 0x7f739c0cb5e0 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.119+0000 7f73a468f640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739c0cb9f0 con 0x7f739c0b3630 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.119+0000 7f73a468f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f739c1552b0 con 0x7f739c154ed0 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.119+0000 7f73a1c2a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f739c0b3630 0x7f739c0cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.119+0000 7f73a1c2a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f739c0b3630 0x7f739c0cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54224/0 (socket says 172.21.15.172:54224) 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.119+0000 7f73a1c2a640 1 -- 172.21.15.172:0/3866550858 learned_addr learned my addr 172.21.15.172:0/3866550858 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.119+0000 7f73a242b640 1 --2- 172.21.15.172:0/3866550858 >> v2:172.21.15.174:3300/0 conn(0x7f739c154ed0 0x7f739c15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.120+0000 7f73a1c2a640 1 -- 172.21.15.172:0/3866550858 >> v2:172.21.15.174:3300/0 conn(0x7f739c154ed0 msgr2=0x7f739c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.120+0000 7f73a1c2a640 1 --2- 172.21.15.172:0/3866550858 >> v2:172.21.15.174:3300/0 conn(0x7f739c154ed0 0x7f739c15ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.120+0000 7f73a1c2a640 1 -- 172.21.15.172:0/3866550858 >> v1:172.21.15.174:6789/0 conn(0x7f739c0cb5e0 legacy=0x7f739c0b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.120+0000 7f73a1c2a640 1 -- 172.21.15.172:0/3866550858 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c15b4b0 con 0x7f739c0b3630 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.120+0000 7f73a1c2a640 1 --2- 172.21.15.172:0/3866550858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 0x7f739c0cb0a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f739c0cde30 tx=0x7f738c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2644a3ed2fba540f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.121+0000 7f73a1429640 1 -- 172.21.15.172:0/3866550858 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c030860 con 0x7f739c0b3630 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.121+0000 7f73a1429640 1 -- 172.21.15.172:0/3866550858 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738c0309c0 con 0x7f739c0b3630 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.121+0000 7f73a1429640 1 -- 172.21.15.172:0/3866550858 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c030d50 con 0x7f739c0b3630 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.121+0000 7f73a468f640 1 -- 172.21.15.172:0/3866550858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 msgr2=0x7f739c0cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.121+0000 7f73a468f640 1 --2- 172.21.15.172:0/3866550858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 0x7f739c0cb0a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f739c0cde30 tx=0x7f738c02b9a0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.121+0000 7f73a468f640 1 -- 172.21.15.172:0/3866550858 shutdown_connections 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.121+0000 7f73a468f640 1 --2- 172.21.15.172:0/3866550858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 0x7f739c0cb0a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.121+0000 7f73a468f640 1 --2- 172.21.15.172:0/3866550858 >> v2:172.21.15.174:3300/0 conn(0x7f739c154ed0 0x7f739c15ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.121+0000 7f73a468f640 1 -- 172.21.15.172:0/3866550858 >> 172.21.15.172:0/3866550858 conn(0x7f739c06bf60 msgr2=0x7f739c06d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:55.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.122+0000 7f73a468f640 1 -- 172.21.15.172:0/3866550858 shutdown_connections 2024-03-16T19:29:55.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.122+0000 7f73a468f640 1 -- 172.21.15.172:0/3866550858 wait complete. 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a468f640 1 Processor -- start 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a468f640 1 -- start start 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a468f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 0x7f739c1f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a468f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f739c0cb5e0 0x7f739c15c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a468f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c154ed0 0x7f739c15a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a468f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c15c330 con 0x7f739c0b3630 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a468f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c15bd00 con 0x7f739c154ed0 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a468f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1f0f60 con 0x7f739c0cb5e0 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a242b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 0x7f739c1f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a242b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 0x7f739c1f0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54232/0 (socket says 172.21.15.172:54232) 2024-03-16T19:29:55.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.123+0000 7f73a242b640 1 -- 172.21.15.172:0/1761260608 learned_addr learned my addr 172.21.15.172:0/1761260608 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a1c2a640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f739c0cb5e0 0x7f739c15c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a2c2c640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c154ed0 0x7f739c15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a242b640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f739c0cb5e0 msgr2=0x7f739c15c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a242b640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f739c0cb5e0 0x7f739c15c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a242b640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c154ed0 msgr2=0x7f739c15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a242b640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c154ed0 0x7f739c15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a242b640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c0094d0 con 0x7f739c0b3630 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a242b640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 0x7f739c1f0a20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f739800e8a0 tx=0x7f739800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73980094f0 con 0x7f739c0b3630 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739800c940 con 0x7f739c0b3630 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739800ccf0 con 0x7f739c0b3630 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c15d1c0 con 0x7f739c0b3630 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.124+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c1f14d0 con 0x7f739c0b3630 2024-03-16T19:29:55.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.125+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f739c068d20 con 0x7f739c0b3630 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.126+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7398013470 con 0x7f739c0b3630 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.126+0000 7f7392ffd640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f738006b3e0 0x7f738006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.127+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f7398089cc0 con 0x7f739c0b3630 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.127+0000 7f73a1c2a640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f738006b3e0 0x7f738006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.127+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v26) v1 ==== 54+0+16 (secure 0 0 0) 0x7f739808c300 con 0x7f739c0b3630 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.127+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f739c0ca2e0 0x7f739c147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.127+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f739c15ba70 con 0x7f739c0ca2e0 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.127+0000 7f73a2c2c640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f739c0ca2e0 0x7f739c147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.127+0000 7f73a1c2a640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f738006b3e0 0x7f738006d8a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f738c036000 tx=0x7f738c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.127+0000 7f73a2c2c640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f739c0ca2e0 0x7f739c147330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.128+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7394001b20 con 0x7f739c0ca2e0 2024-03-16T19:29:55.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.143+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f739c0688d0 con 0x7f739c0ca2e0 2024-03-16T19:29:55.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.144+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f739c0688d0 con 0x7f739c0ca2e0 2024-03-16T19:29:55.126 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:29:55.127 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:29:55.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.145+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f739c1f3f90 0x7f739c1f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.145+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f739c1584c0 con 0x7f739c1f3f90 2024-03-16T19:29:55.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.145+0000 7f73a242b640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f739c1f3f90 0x7f739c1f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.145+0000 7f73a242b640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f739c1f3f90 0x7f739c1f4370 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.146+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f739c1584c0 con 0x7f739c1f3f90 2024-03-16T19:29:55.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.161+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f739c14b9d0 con 0x7f739c1f3f90 2024-03-16T19:29:55.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.164+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f739c14b9d0 con 0x7f739c1f3f90 2024-03-16T19:29:55.145 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:29:55.146 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:29:55.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.164+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f739c061930 0x7f739c061d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.164+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f739c1f4dc0 con 0x7f739c061930 2024-03-16T19:29:55.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.164+0000 7f73a1c2a640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f739c061930 0x7f739c061d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.164+0000 7f73a1c2a640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f739c061930 0x7f739c061d30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.165+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f739c1f4dc0 con 0x7f739c061930 2024-03-16T19:29:55.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.179+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f739c0b2be0 con 0x7f739c061930 2024-03-16T19:29:55.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.181+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f739c0b2be0 con 0x7f739c061930 2024-03-16T19:29:55.163 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:29:55.163 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:29:55.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.181+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f739c150cc0 0x7f739c151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.181+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f739c1525f0 con 0x7f739c150cc0 2024-03-16T19:29:55.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.181+0000 7f73a2c2c640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f739c150cc0 0x7f739c151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.182+0000 7f73a2c2c640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f739c150cc0 0x7f739c151100 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.182+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f739c1525f0 con 0x7f739c150cc0 2024-03-16T19:29:55.177 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:55.196+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:55.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.196+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f739c1f17b0 con 0x7f739c150cc0 2024-03-16T19:29:55.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.198+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f739c1f17b0 con 0x7f739c150cc0 2024-03-16T19:29:55.180 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:29:55.180 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:29:55.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.198+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f739c02bde0 0x7f739c02c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.198+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f739c15c000 con 0x7f739c02bde0 2024-03-16T19:29:55.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.198+0000 7f73a242b640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f739c02bde0 0x7f739c02c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.199+0000 7f73a242b640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f739c02bde0 0x7f739c02c240 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.200+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f739c15c000 con 0x7f739c02bde0 2024-03-16T19:29:55.187 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:55.187 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:55.187 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:29:55.187 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:29:55.187 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:55.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:29:55.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.214+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f739c15c000 con 0x7f739c02bde0 2024-03-16T19:29:55.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.216+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f739c15c000 con 0x7f739c02bde0 2024-03-16T19:29:55.198 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:29:55.199 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:29:55.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.216+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f739c02cc90 0x7f739c207320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.216+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f739c151b50 con 0x7f739c02cc90 2024-03-16T19:29:55.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.216+0000 7f73a1c2a640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f739c02cc90 0x7f739c207320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.217+0000 7f73a1c2a640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f739c02cc90 0x7f739c207320 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.218+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f739c151b50 con 0x7f739c02cc90 2024-03-16T19:29:55.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.232+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f739c151b50 con 0x7f739c02cc90 2024-03-16T19:29:55.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.234+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f739c151b50 con 0x7f739c02cc90 2024-03-16T19:29:55.216 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:29:55.216 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:29:55.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.234+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f739c02d0f0 0x7f739c209d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.234+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f739c151cc0 con 0x7f739c02d0f0 2024-03-16T19:29:55.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.235+0000 7f73a2c2c640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f739c02d0f0 0x7f739c209d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.235+0000 7f73a2c2c640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f739c02d0f0 0x7f739c209d80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.236+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7394019510 con 0x7f739c02d0f0 2024-03-16T19:29:55.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.250+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f739c1f17b0 con 0x7f739c02d0f0 2024-03-16T19:29:55.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.253+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f739c1f17b0 con 0x7f739c02d0f0 2024-03-16T19:29:55.235 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:29:55.235 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:29:55.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.253+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f739c20a7d0 0x7f739c20cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:55.236 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.253+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f739c20f990 con 0x7f739c20a7d0 2024-03-16T19:29:55.236 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.253+0000 7f73a242b640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f739c20a7d0 0x7f739c20cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:55.236 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.254+0000 7f73a242b640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f739c20a7d0 0x7f739c20cbc0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:55.236 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.255+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f739c20f990 con 0x7f739c20a7d0 2024-03-16T19:29:55.251 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.269+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f739c20d100 con 0x7f739c20a7d0 2024-03-16T19:29:55.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.271+0000 7f7392ffd640 1 -- 172.21.15.172:0/1761260608 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f739c20d100 con 0x7f739c20a7d0 2024-03-16T19:29:55.253 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f739c0ca2e0 msgr2=0x7f739c147330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f739c0ca2e0 0x7f739c147330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f739c1f3f90 msgr2=0x7f739c1f4370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f739c1f3f90 0x7f739c1f4370 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f739c061930 msgr2=0x7f739c061d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f739c061930 0x7f739c061d30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f739c150cc0 msgr2=0x7f739c151100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f739c150cc0 0x7f739c151100 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f739c02bde0 msgr2=0x7f739c02c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f739c02bde0 0x7f739c02c240 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f739c02cc90 msgr2=0x7f739c207320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f739c02cc90 0x7f739c207320 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f739c02d0f0 msgr2=0x7f739c209d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a2c2c640 1 -- 172.21.15.172:0/1761260608 reap_dead start 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f739c02d0f0 0x7f739c209d80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a2c2c640 1 -- 172.21.15.172:0/1761260608 reap_dead start 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f739c20a7d0 msgr2=0x7f739c20cbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f739c20a7d0 0x7f739c20cbc0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a2c2c640 1 -- 172.21.15.172:0/1761260608 reap_dead start 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f738006b3e0 msgr2=0x7f738006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f738006b3e0 0x7f738006d8a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f738c036000 tx=0x7f738c008030 comp rx=0 tx=0).stop 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 msgr2=0x7f739c1f0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 0x7f739c1f0a20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f739800e8a0 tx=0x7f739800ed60 comp rx=0 tx=0).stop 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 shutdown_connections 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f738006b3e0 0x7f738006d8a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 --2- 172.21.15.172:0/1761260608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f739c0b3630 0x7f739c1f0a20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 >> 172.21.15.172:0/1761260608 conn(0x7f739c06bf60 msgr2=0x7f739c0b5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:55.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 shutdown_connections 2024-03-16T19:29:55.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.272+0000 7f73a468f640 1 -- 172.21.15.172:0/1761260608 wait complete. 2024-03-16T19:29:55.278 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:55.298+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:55.307 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:55.307 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:29:55.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:29:55.307 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:29:55.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.307 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.968966676000001, 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:29:55.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.968952203000001, 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:29:55.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:29:55.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.968447714, 2024-03-16T19:29:55.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.968432318000001, 2024-03-16T19:29:55.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.967973635, 2024-03-16T19:29:55.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:29:55.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:29:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:29:55.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.967427465, 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.327 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.966870373999999, 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:29:55.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:29:55.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:29:55.335 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.965668625999999, 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:29:55.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:29:55.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.965089917, 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:29:55.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:29:55.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.965040389999999, 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.343 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:29:55.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:29:55.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:29:55.346 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.964079747, 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:29:55.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:29:55.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.960365548999999, 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:29:55.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:29:55.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:29:55.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.519964636999999, 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.954411634, 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.446812909, 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:29:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:29:55.366 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:55.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:29:55.410 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:55.430+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:55.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.435+0000 7f669d7fa640 1 -- 172.21.15.172:0/3318134614 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "3.20", "id": [7, 6, 1, 3]}]=0 set 3.20 pg_upmap_items mapping to [7->6,1->3] v27) v1 ==== 148+0+0 (secure 0 0 0) 0x7f6694017020 con 0x7f66a0158120 2024-03-16T19:29:55.417 INFO:teuthology.orchestra.run.smithi172.stderr:set 3.20 pg_upmap_items mapping to [7->6,1->3] 2024-03-16T19:29:55.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.437+0000 7fdaa4ff9640 1 -- 172.21.15.172:0/4264877473 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v27) v1 ==== 87+0+0 (secure 0 0 0) 0x7fda9c057ca0 con 0x7fdaa8157e50 2024-03-16T19:29:55.419 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.437+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f668c06b3b0 msgr2=0x7f668c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.437+0000 7f66a6fb6640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f668c06b3b0 0x7f668c06d870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f66980012e0 tx=0x7f6698009040 comp rx=0 tx=0).stop 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.437+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66a0158120 msgr2=0x7f66a00cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.437+0000 7f66a6fb6640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66a0158120 0x7f66a00cd530 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f669400c460 tx=0x7f669400c920 comp rx=0 tx=0).stop 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.438+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 shutdown_connections 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.438+0000 7f66a6fb6640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f668c06b3b0 0x7f668c06d870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.438+0000 7f66a6fb6640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66a015bcd0 0x7f66a01f3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.438+0000 7f66a6fb6640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66a0158120 0x7f66a00cd530 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.438+0000 7f66a6fb6640 1 --2- 172.21.15.172:0/3318134614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66a006dc80 0x7f66a00c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.438+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 >> 172.21.15.172:0/3318134614 conn(0x7f66a006f940 msgr2=0x7f66a00c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.438+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 shutdown_connections 2024-03-16T19:29:55.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.438+0000 7f66a6fb6640 1 -- 172.21.15.172:0/3318134614 wait complete. 2024-03-16T19:29:55.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fda9406b3b0 msgr2=0x7fda9406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fda9406b3b0 0x7fda9406d870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fda98000f60 tx=0x7fda98007e00 comp rx=0 tx=0).stop 2024-03-16T19:29:55.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8157e50 msgr2=0x7fdaa814edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:55.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8157e50 0x7fdaa814edc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00c3e0 tx=0x7fda9c00c8a0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 shutdown_connections 2024-03-16T19:29:55.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdaa8158230 0x7fdaa815d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fda9406b3b0 0x7fda9406d870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8157e50 0x7fdaa814edc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 --2- 172.21.15.172:0/4264877473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa806ba30 0x7fdaa8150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:55.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 >> 172.21.15.172:0/4264877473 conn(0x7fdaa806db70 msgr2=0x7fdaa81554d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:55.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.439+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 shutdown_connections 2024-03-16T19:29:55.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:55.440+0000 7fdaae5d7640 1 -- 172.21.15.172:0/4264877473 wait complete. 2024-03-16T19:29:55.479 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:55.499+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:55.486 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:55.487 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:55.487 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:29:55.487 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:55.495 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:29:55.580 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:55.600+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:55.642 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:55.642 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:55.642 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:29:55.642 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:29:55.642 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:55.652 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:29:55.681 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:55.700+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:55.782 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:55.801+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:55.802 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:55.802 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:29:55.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:29:55.802 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:29:55.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.803 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:55.803 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:29:55.803 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.462404578000001, 2024-03-16T19:29:55.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:29:55.803 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:55.803 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.803 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:55.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:55.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:29:55.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:55.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:29:55.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:55.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:29:55.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:29:55.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:55.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:29:55.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:29:55.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:55.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:29:55.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:29:55.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:29:55.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:29:55.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:55.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:29:55.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:29:55.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:55.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:29:55.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:29:55.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:55.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:29:55.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:29:55.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:55.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:29:55.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:29:55.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:55.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:29:55.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:29:55.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:55.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:29:55.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:29:55.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:55.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:29:55.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:29:55.813 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.814 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:55.814 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.814 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:29:55.814 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:29:55.815 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.461177156000002, 2024-03-16T19:29:55.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:29:55.815 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:55.815 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:29:55.815 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:55.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:55.816 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:29:55.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.816 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:55.816 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:29:55.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.817 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.817 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.817 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:55.817 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:29:55.817 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:29:55.817 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.818 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:55.818 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:29:55.818 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:29:55.818 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.819 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:55.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:29:55.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:29:55.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.819 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.819 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:29:55.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:29:55.820 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:55.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:29:55.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:29:55.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:55.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:29:55.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:29:55.821 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.821 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:55.822 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.822 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:55.822 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:29:55.822 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.460774449999999, 2024-03-16T19:29:55.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:29:55.823 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:55.823 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.823 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:55.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:55.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:29:55.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:55.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:29:55.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:55.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:29:55.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:29:55.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:55.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:29:55.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:29:55.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:55.827 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:29:55.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:29:55.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.827 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:29:55.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:29:55.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:55.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:29:55.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:29:55.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:55.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:29:55.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:29:55.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:55.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:29:55.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:29:55.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:55.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:29:55.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:29:55.831 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.831 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:55.831 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.831 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:55.832 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:29:55.832 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.460165699000001, 2024-03-16T19:29:55.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:29:55.832 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:55.832 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.832 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:55.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:55.834 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:29:55.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:29:55.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.834 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:55.834 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:29:55.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:29:55.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:55.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:29:55.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:29:55.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:55.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:29:55.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.459721829999999, 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:55.841 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.459579053999999, 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:29:55.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:29:55.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.459564513, 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:55.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:55.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:29:55.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:29:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.456686818000001, 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:55.849 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:29:55.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.852 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:55.853 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:55.853 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:55.854 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:29:55.894 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:55.914+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:55.983 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:56.002+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:56.000 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:56.000 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:56.000 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:29:56.000 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:56.008 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:29:56.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:29:56.072 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:29:56.115 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:56.115 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:56.115 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:29:56.115 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:29:56.115 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:56.116 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:56.135+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:56.124 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:29:56.184 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:56.204+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:56.273 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:56.273 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:29:56.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:29:56.273 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.934732253, 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.934653135000001, 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.278 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:29:56.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.934039010999999, 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:29:56.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:29:56.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.933598942, 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.287 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:29:56.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:29:56.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.933357352000002, 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:29:56.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.296 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.297 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:56.298 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:29:56.298 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:56.317+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:56.399 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:56.418+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:56.446 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:56.447 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:56.447 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:29:56.447 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:56.456 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:29:56.500 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:56.519+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:56.563 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:56.563 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:56.563 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:29:56.563 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:29:56.563 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:56.571 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:29:56.612 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:56.632+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:56.702 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:56.721+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.382703620000001, 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.722 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.382679447000001, 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:29:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:29:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:29:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.382673619999998, 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:29:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:29:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.382669708000002, 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:29:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.382663726000001, 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:29:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:29:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:29:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:29:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.382660059999999, 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:29:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:29:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.382649399999998, 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:29:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:29:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.382645511, 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.379128819999998, 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:29:56.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.764 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.765 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.379058385, 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.766 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:29:56.767 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.768 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.769 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.378676037999998, 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.770 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:29:56.771 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.772 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.773 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.774 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.376909755, 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:29:56.775 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:29:56.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.777 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.375096833000001, 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:29:56.778 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.779 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.780 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.781 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:25 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.374859053000002, 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189085, 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:29:56.782 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2099999999999996e-07 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365068+0000", 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9700000000000001e-06 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:29:56.783 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0020999999999999e-05 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7449999999999998e-06 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365136+0000", 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0973999999999997e-05 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.784 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365145+0000", 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9560000000000003e-06 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365253+0000", 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107798 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:29:56.785 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.374241084000001, 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.786 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.787 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.788 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.789 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.790 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.368380318, 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:29:56.791 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:29:56.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.794 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.93582911, 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:29:56.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:29:56.797 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:40 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.932548598, 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019577200000000001, 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1600000000000001e-07 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807377+0000", 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7839999999999999e-06 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807384+0000", 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8410000000000002e-06 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807388+0000", 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0550000000000001e-06 2024-03-16T19:29:56.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807431+0000", 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3271e-05 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807441+0000", 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2320000000000007e-06 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807570+0000", 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000129073 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.803 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.368335404, 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.808 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.860723153, 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:29:56.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:29:56.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:29:56.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:29:56.813 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.814 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.814 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.814 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:56.814 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:56.814 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:56.815 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:29:56.816 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:56.834+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:56.916 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:56.935+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:56.922 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:56.922 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:56.922 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:29:56.922 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:56.931 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:29:57.017 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:57.037+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:57.063 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:29:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:29:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:29:57.064 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:57.075 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:29:57.118 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:57.137+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:57.219 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:57.238+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:57.228 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:29:57.228 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:29:57.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:29:57.228 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:29:57.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 41.016697987000001, 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:29:57.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:29:57.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:29:57.234 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.640002440000004, 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:29:57.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:29:57.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.864025996000002, 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:29:57.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.244 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.863941582999999, 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:29:57.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:29:57.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:29:57.250 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.863934227000001, 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:29:57.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:29:57.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:29:57.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.671395035000003, 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.256 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:29:57.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:29:57.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:29:57.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.671389228000002, 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:29:57.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:29:57.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.671271095000002, 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:29:57.267 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:29:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:29:57.272 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.671175898000001, 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.273 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.671171887, 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:29:57.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:29:57.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.252240012999998, 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.284 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:29:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:29:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:29:57.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.252190644000002, 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:29:57.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:29:57.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.252111812000003, 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:29:57.295 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:29:57.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:29:57.300 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.252108601000003, 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.301 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.306 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.252105647999997, 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:29:57.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:29:57.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.251990673999998, 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.312 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:29:57.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.317 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.251987321999998, 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.318 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:29:57.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:29:57.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:29:57.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:29:57.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.251984206000003, 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:29:57.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.251923736999998, 2024-03-16T19:29:57.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:29:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:29:57.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:29:57.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.643243687000002, 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:29:57.335 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:29:57.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:29:57.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.340 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.341 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.341 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:29:57.341 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:29:57.341 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:29:57.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:29:57.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:29:57.343 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:57.361+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:57.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:29:57.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.461+0000 7f51ff8fb640 1 Processor -- start 2024-03-16T19:29:57.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.461+0000 7f51ff8fb640 1 -- start start 2024-03-16T19:29:57.443 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:57.462+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:57.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.462+0000 7f51ff8fb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f51f8154ed0 0x7f51f815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.462+0000 7f51ff8fb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f51f80b4700 0x7f51f80b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.462+0000 7f51ff8fb640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f80b4270 con 0x7f51f80b3630 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.462+0000 7f51ff8fb640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f80b5050 con 0x7f51f80b4700 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.462+0000 7f51ff8fb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f81552b0 con 0x7f51f8154ed0 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.462+0000 7f51fde98640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f51f80b4700 0x7f51f80b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.462+0000 7f51fde98640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f51f80b4700 0x7f51f80b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54246/0 (socket says 172.21.15.172:54246) 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.462+0000 7f51fde98640 1 -- 172.21.15.172:0/4197761305 learned_addr learned my addr 172.21.15.172:0/4197761305 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.462+0000 7f51fd697640 1 --2- 172.21.15.172:0/4197761305 >> v2:172.21.15.174:3300/0 conn(0x7f51f8154ed0 0x7f51f815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.463+0000 7f51fde98640 1 -- 172.21.15.172:0/4197761305 >> v2:172.21.15.174:3300/0 conn(0x7f51f8154ed0 msgr2=0x7f51f815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.463+0000 7f51fde98640 1 --2- 172.21.15.172:0/4197761305 >> v2:172.21.15.174:3300/0 conn(0x7f51f8154ed0 0x7f51f815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.463+0000 7f51fde98640 1 -- 172.21.15.172:0/4197761305 >> v1:172.21.15.174:6789/0 conn(0x7f51f80b3630 legacy=0x7f51f80cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.463+0000 7f51fde98640 1 -- 172.21.15.172:0/4197761305 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f815b4b0 con 0x7f51f80b4700 2024-03-16T19:29:57.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.464+0000 7f51fde98640 1 --2- 172.21.15.172:0/4197761305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f80b4700 0x7f51f80b4b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f51f80cde30 tx=0x7f51f002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46922ff1b4cbbf4b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.464+0000 7f51effff640 1 -- 172.21.15.172:0/4197761305 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f002cb10 con 0x7f51f80b4700 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.464+0000 7f51effff640 1 -- 172.21.15.172:0/4197761305 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f002cc70 con 0x7f51f80b4700 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.464+0000 7f51effff640 1 -- 172.21.15.172:0/4197761305 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f00343f0 con 0x7f51f80b4700 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.464+0000 7f51ff8fb640 1 -- 172.21.15.172:0/4197761305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f80b4700 msgr2=0x7f51f80b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.464+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/4197761305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f80b4700 0x7f51f80b4b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f51f80cde30 tx=0x7f51f002bf70 comp rx=0 tx=0).stop 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.465+0000 7f51ff8fb640 1 -- 172.21.15.172:0/4197761305 shutdown_connections 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.465+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/4197761305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f80b4700 0x7f51f80b4b10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.465+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/4197761305 >> v2:172.21.15.174:3300/0 conn(0x7f51f8154ed0 0x7f51f815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.465+0000 7f51ff8fb640 1 -- 172.21.15.172:0/4197761305 >> 172.21.15.172:0/4197761305 conn(0x7f51f806bf60 msgr2=0x7f51f806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.465+0000 7f51ff8fb640 1 -- 172.21.15.172:0/4197761305 shutdown_connections 2024-03-16T19:29:57.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.465+0000 7f51ff8fb640 1 -- 172.21.15.172:0/4197761305 wait complete. 2024-03-16T19:29:57.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.466+0000 7f51ff8fb640 1 Processor -- start 2024-03-16T19:29:57.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.466+0000 7f51ff8fb640 1 -- start start 2024-03-16T19:29:57.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.466+0000 7f51ff8fb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f80b3630 0x7f51f815d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.466+0000 7f51ff8fb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f80b4700 0x7f51f81f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.466+0000 7f51ff8fb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f8154ed0 0x7f51f8147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51ff8fb640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8147ea0 con 0x7f51f8154ed0 2024-03-16T19:29:57.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51ff8fb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81f1410 con 0x7f51f80b3630 2024-03-16T19:29:57.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51ff8fb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f80c6de0 con 0x7f51f80b4700 2024-03-16T19:29:57.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fde98640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f8154ed0 0x7f51f8147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fd697640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f80b3630 0x7f51f815d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fde98640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f8154ed0 0x7f51f8147960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54260/0 (socket says 172.21.15.172:54260) 2024-03-16T19:29:57.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fde98640 1 -- 172.21.15.172:0/633731613 learned_addr learned my addr 172.21.15.172:0/633731613 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:57.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fce96640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f80b4700 0x7f51f81f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fde98640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f80b4700 msgr2=0x7f51f81f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fde98640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f80b4700 0x7f51f81f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fde98640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f80b3630 msgr2=0x7f51f815d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fde98640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f80b3630 0x7f51f815d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fde98640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0009b30 con 0x7f51f8154ed0 2024-03-16T19:29:57.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51fde98640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f8154ed0 0x7f51f8147960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009f40 tx=0x7f51f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f0009510 con 0x7f51f8154ed0 2024-03-16T19:29:57.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f003e450 con 0x7f51f8154ed0 2024-03-16T19:29:57.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f815cb50 con 0x7f51f8154ed0 2024-03-16T19:29:57.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.467+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f003e800 con 0x7f51f8154ed0 2024-03-16T19:29:57.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.468+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f815d110 con 0x7f51f8154ed0 2024-03-16T19:29:57.452 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.469+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f51f8068d20 con 0x7f51f8154ed0 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.469+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f51f003e960 con 0x7f51f8154ed0 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.470+0000 7f51edffb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51dc06b3b0 0x7f51dc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.470+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f51f00acd70 con 0x7f51f8154ed0 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.470+0000 7f51fd697640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51dc06b3b0 0x7f51dc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.470+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v27) v1 ==== 54+0+16 (secure 0 0 0) 0x7f51f00ad1c0 con 0x7f51f8154ed0 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.470+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51f815c5f0 0x7f51f8153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.470+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51f81f1260 con 0x7f51f815c5f0 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.470+0000 7f51fce96640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51f815c5f0 0x7f51f8153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.470+0000 7f51fd697640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51dc06b3b0 0x7f51dc06d870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009b40 tx=0x7f51f4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.471+0000 7f51fce96640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51f815c5f0 0x7f51f8153150 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.472+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51f81f1260 con 0x7f51f815c5f0 2024-03-16T19:29:57.454 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:29:57.454 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:29:57.454 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:29:57.454 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:29:57.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:29:57.469 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.487+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51f80b3c20 con 0x7f51f815c5f0 2024-03-16T19:29:57.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.489+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51e8001810 con 0x7f51f815c5f0 2024-03-16T19:29:57.471 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:29:57.471 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:29:57.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.489+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51f80d0d00 0x7f51f80c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.489+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51f806df10 con 0x7f51f80d0d00 2024-03-16T19:29:57.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.489+0000 7f51fd697640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51f80d0d00 0x7f51f80c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.489+0000 7f51fd697640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51f80d0d00 0x7f51f80c6660 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.490+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51f806df10 con 0x7f51f80d0d00 2024-03-16T19:29:57.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.505+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51f80681c0 con 0x7f51f80d0d00 2024-03-16T19:29:57.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.558+0000 7f71cf155640 1 Processor -- start 2024-03-16T19:29:57.540 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.558+0000 7f71cf155640 1 -- start start 2024-03-16T19:29:57.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71cf155640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f71c8157ec0 0x7f71c815b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71cf155640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c806dc20 0x7f71c815b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71cf155640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f71c80c9080 0x7f71c80c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71cf155640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c80cba70 con 0x7f71c8157ec0 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71cf155640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c806bfe0 con 0x7f71c80c9080 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71cf155640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c80700f0 con 0x7f71c806dc20 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71ccef1640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f71c8157ec0 0x7f71c815b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71ccef1640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f71c8157ec0 0x7f71c815b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54262/0 (socket says 172.21.15.172:54262) 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71ccef1640 1 -- 172.21.15.172:0/1332447665 learned_addr learned my addr 172.21.15.172:0/1332447665 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71c7fff640 1 --2- 172.21.15.172:0/1332447665 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c806dc20 0x7f71c815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.559+0000 7f71cd6f2640 1 --2- 172.21.15.172:0/1332447665 >> v2:172.21.15.174:3300/0 conn(0x7f71c80c9080 0x7f71c80c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.560+0000 7f71ccef1640 1 -- 172.21.15.172:0/1332447665 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c806dc20 msgr2=0x7f71c815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.560+0000 7f71ccef1640 1 --2- 172.21.15.172:0/1332447665 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c806dc20 0x7f71c815b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.560+0000 7f71ccef1640 1 -- 172.21.15.172:0/1332447665 >> v2:172.21.15.174:3300/0 conn(0x7f71c80c9080 msgr2=0x7f71c80c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:29:57.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.560+0000 7f71ccef1640 1 --2- 172.21.15.172:0/1332447665 >> v2:172.21.15.174:3300/0 conn(0x7f71c80c9080 0x7f71c80c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.560+0000 7f71ccef1640 1 -- 172.21.15.172:0/1332447665 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c80c8ac0 con 0x7f71c8157ec0 2024-03-16T19:29:57.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.560+0000 7f71ccef1640 1 --2- 172.21.15.172:0/1332447665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c8157ec0 0x7f71c815b000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f71c80cf160 tx=0x7f71c002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ddfbb86bfa2babb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.560+0000 7f71c77fe640 1 -- 172.21.15.172:0/1332447665 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c002cb00 con 0x7f71c8157ec0 2024-03-16T19:29:57.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.560+0000 7f71c77fe640 1 -- 172.21.15.172:0/1332447665 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c002cc60 con 0x7f71c8157ec0 2024-03-16T19:29:57.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.560+0000 7f71c77fe640 1 -- 172.21.15.172:0/1332447665 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c00343f0 con 0x7f71c8157ec0 2024-03-16T19:29:57.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.561+0000 7f71cf155640 1 -- 172.21.15.172:0/1332447665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c8157ec0 msgr2=0x7f71c815b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.561+0000 7f71cf155640 1 --2- 172.21.15.172:0/1332447665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c8157ec0 0x7f71c815b000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f71c80cf160 tx=0x7f71c002bf60 comp rx=0 tx=0).stop 2024-03-16T19:29:57.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.561+0000 7f71cf155640 1 -- 172.21.15.172:0/1332447665 shutdown_connections 2024-03-16T19:29:57.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.561+0000 7f71cf155640 1 --2- 172.21.15.172:0/1332447665 >> v2:172.21.15.174:3300/0 conn(0x7f71c80c9080 0x7f71c80c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.561+0000 7f71cf155640 1 --2- 172.21.15.172:0/1332447665 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c806dc20 0x7f71c815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.561+0000 7f71cf155640 1 --2- 172.21.15.172:0/1332447665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c8157ec0 0x7f71c815b000 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.561+0000 7f71cf155640 1 -- 172.21.15.172:0/1332447665 >> 172.21.15.172:0/1332447665 conn(0x7f71c8070810 msgr2=0x7f71c8070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:57.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.561+0000 7f71cf155640 1 -- 172.21.15.172:0/1332447665 shutdown_connections 2024-03-16T19:29:57.545 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.561+0000 7f71cf155640 1 -- 172.21.15.172:0/1332447665 wait complete. 2024-03-16T19:29:57.545 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:57.562+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:57.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.562+0000 7f71cf155640 1 Processor -- start 2024-03-16T19:29:57.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.562+0000 7f71cf155640 1 -- start start 2024-03-16T19:29:57.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71cf155640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c806dc20 0x7f71c8151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71cf155640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c80c9080 0x7f71c81535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71cf155640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c8157ec0 0x7f71c80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71cf155640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c81f1900 con 0x7f71c80c9080 2024-03-16T19:29:57.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71cf155640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8153110 con 0x7f71c806dc20 2024-03-16T19:29:57.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71cf155640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8153ae0 con 0x7f71c8157ec0 2024-03-16T19:29:57.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71c7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c80c9080 0x7f71c81535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71c7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c80c9080 0x7f71c81535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54266/0 (socket says 172.21.15.172:54266) 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71c7fff640 1 -- 172.21.15.172:0/3525889390 learned_addr learned my addr 172.21.15.172:0/3525889390 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71ccef1640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c806dc20 0x7f71c8151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.563+0000 7f71cd6f2640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c8157ec0 0x7f71c80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71c7fff640 1 -- 172.21.15.172:0/3525889390 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c8157ec0 msgr2=0x7f71c80cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71c7fff640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c8157ec0 0x7f71c80cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71c7fff640 1 -- 172.21.15.172:0/3525889390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c806dc20 msgr2=0x7f71c8151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71c7fff640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c806dc20 0x7f71c8151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71c7fff640 1 -- 172.21.15.172:0/3525889390 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c0009b30 con 0x7f71c80c9080 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71c7fff640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c80c9080 0x7f71c81535a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f71b000e870 tx=0x7f71b000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71c57fa640 1 -- 172.21.15.172:0/3525889390 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b00094f0 con 0x7f71c80c9080 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71c57fa640 1 -- 172.21.15.172:0/3525889390 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b000c940 con 0x7f71c80c9080 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c81f1620 con 0x7f71c80c9080 2024-03-16T19:29:57.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71c57fa640 1 -- 172.21.15.172:0/3525889390 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b000ccf0 con 0x7f71c80c9080 2024-03-16T19:29:57.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.564+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c8152ac0 con 0x7f71c80c9080 2024-03-16T19:29:57.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.565+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c81532c0 con 0x7f71c80c9080 2024-03-16T19:29:57.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.566+0000 7f71c57fa640 1 -- 172.21.15.172:0/3525889390 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f71b0013470 con 0x7f71c80c9080 2024-03-16T19:29:57.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.566+0000 7f71c57fa640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f71a806b3b0 0x7f71a806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.566+0000 7f71c57fa640 1 -- 172.21.15.172:0/3525889390 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f71b008a0a0 con 0x7f71c80c9080 2024-03-16T19:29:57.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.566+0000 7f71ccef1640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f71a806b3b0 0x7f71a806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.567+0000 7f71ccef1640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f71a806b3b0 0x7f71a806d870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f71c0035000 tx=0x7f71c0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.575+0000 7f71c57fa640 1 -- 172.21.15.172:0/3525889390 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71c81532c0 con 0x7f71c80c9080 2024-03-16T19:29:57.582 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:57.582 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:57.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:29:57.676 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:57.694+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:57.741 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:57.741 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:57.745 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:57.764+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:57.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.765+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f71c81532c0 con 0x7f71c80c9080 2024-03-16T19:29:57.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:29:57.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.852+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51f80681c0 con 0x7f51f80d0d00 2024-03-16T19:29:57.834 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:29:57.835 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:29:57.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.852+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f51f81f4b90 0x7f51f8061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.853+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51f81f1550 con 0x7f51f81f4b90 2024-03-16T19:29:57.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.853+0000 7f51fce96640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f51f81f4b90 0x7f51f8061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.853+0000 7f51fce96640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f51f81f4b90 0x7f51f8061930 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.854+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51f81f1550 con 0x7f51f81f4b90 2024-03-16T19:29:57.846 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:57.865+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:57.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.868+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51f815a6d0 con 0x7f51f81f4b90 2024-03-16T19:29:57.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.869+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51f815a6d0 con 0x7f51f81f4b90 2024-03-16T19:29:57.851 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:29:57.852 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:29:57.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.869+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f51f80ccb90 0x7f51f80ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.869+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51f80b28f0 con 0x7f51f80ccb90 2024-03-16T19:29:57.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.870+0000 7f51fde98640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f51f80ccb90 0x7f51f80ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.870+0000 7f51fde98640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f51f80ccb90 0x7f51f80ccff0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.871+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51f80b28f0 con 0x7f51f80ccb90 2024-03-16T19:29:57.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.885+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51f80cbbe0 con 0x7f51f80ccb90 2024-03-16T19:29:57.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.888+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51f80cbbe0 con 0x7f51f80ccb90 2024-03-16T19:29:57.869 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:29:57.870 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:29:57.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.888+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51f802bde0 0x7f51f802c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.888+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51f806d860 con 0x7f51f802bde0 2024-03-16T19:29:57.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.888+0000 7f51fd697640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51f802bde0 0x7f51f802c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.888+0000 7f51fd697640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51f802bde0 0x7f51f802c200 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.889+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51f806d860 con 0x7f51f802bde0 2024-03-16T19:29:57.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.903+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51f806d860 con 0x7f51f802bde0 2024-03-16T19:29:57.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.905+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51f806d860 con 0x7f51f802bde0 2024-03-16T19:29:57.887 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:29:57.888 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:29:57.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.905+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f51f802cc50 0x7f51f8207650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.905+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51f8150c60 con 0x7f51f802cc50 2024-03-16T19:29:57.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.905+0000 7f51fce96640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f51f802cc50 0x7f51f8207650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.906+0000 7f51fce96640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f51f802cc50 0x7f51f8207650 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.906+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51f8150c60 con 0x7f51f802cc50 2024-03-16T19:29:57.916 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:57.916 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:57.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.938+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51f8150c60 con 0x7f51f802cc50 2024-03-16T19:29:57.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.940+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51f8150c60 con 0x7f51f802cc50 2024-03-16T19:29:57.922 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:29:57.923 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:29:57.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.940+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f51f802d0b0 0x7f51f820a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.940+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51f8150980 con 0x7f51f802d0b0 2024-03-16T19:29:57.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.940+0000 7f51fde98640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f51f802d0b0 0x7f51f820a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.941+0000 7f51fde98640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f51f802d0b0 0x7f51f820a0b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.941+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51f8150980 con 0x7f51f802d0b0 2024-03-16T19:29:57.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:29:57.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.956+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51f8150980 con 0x7f51f802d0b0 2024-03-16T19:29:57.939 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:29:57.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.957+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51f8150980 con 0x7f51f802d0b0 2024-03-16T19:29:57.940 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:29:57.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.957+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f51f820ab00 0x7f51f820cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:29:57.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.957+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51f8150980 con 0x7f51f820ab00 2024-03-16T19:29:57.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.958+0000 7f51fd697640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f51f820ab00 0x7f51f820cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:29:57.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.958+0000 7f51fd697640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f51f820ab00 0x7f51f820cef0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:29:57.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.959+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51f8150980 con 0x7f51f820ab00 2024-03-16T19:29:57.947 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:57.965+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:57.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.973+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f51f8150980 con 0x7f51f820ab00 2024-03-16T19:29:57.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51edffb640 1 -- 172.21.15.172:0/633731613 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f51f8150980 con 0x7f51f820ab00 2024-03-16T19:29:57.957 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:29:57.958 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:29:57.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51f815c5f0 msgr2=0x7f51f8153150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f51f815c5f0 0x7f51f8153150 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51f80d0d00 msgr2=0x7f51f80c6660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f51f80d0d00 0x7f51f80c6660 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f51f81f4b90 msgr2=0x7f51f8061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f51f81f4b90 0x7f51f8061930 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f51f80ccb90 msgr2=0x7f51f80ccff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f51f80ccb90 0x7f51f80ccff0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51fde98640 1 -- 172.21.15.172:0/633731613 reap_dead start 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51f802bde0 msgr2=0x7f51f802c200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f51f802bde0 0x7f51f802c200 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51fde98640 1 -- 172.21.15.172:0/633731613 reap_dead start 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f51f802cc50 msgr2=0x7f51f8207650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f51f802cc50 0x7f51f8207650 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f51f802d0b0 msgr2=0x7f51f820a0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f51f802d0b0 0x7f51f820a0b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f51f820ab00 msgr2=0x7f51f820cef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f51f820ab00 0x7f51f820cef0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51dc06b3b0 msgr2=0x7f51dc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51dc06b3b0 0x7f51dc06d870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009b40 tx=0x7f51f4009190 comp rx=0 tx=0).stop 2024-03-16T19:29:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f8154ed0 msgr2=0x7f51f8147960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 --2- 172.21.15.172:0/633731613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51f8154ed0 0x7f51f8147960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009f40 tx=0x7f51f0004a40 comp rx=0 tx=0).stop 2024-03-16T19:29:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51fde98640 1 -- 172.21.15.172:0/633731613 reap_dead start 2024-03-16T19:29:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 shutdown_connections 2024-03-16T19:29:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 >> 172.21.15.172:0/633731613 conn(0x7f51f806bf60 msgr2=0x7f51f8154120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 shutdown_connections 2024-03-16T19:29:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:57.975+0000 7f51ff8fb640 1 -- 172.21.15.172:0/633731613 wait complete. 2024-03-16T19:29:58.044 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:58.044 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:58.047 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:58.067+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:58.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:29:58.149 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:58.167+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:58.160 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:58.160 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:58.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:29:58.253 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:58.272+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:58.317 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:58.317 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:58.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:29:58.366 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:29:58.385+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:29:58.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.445+0000 7f71c57fa640 1 -- 172.21.15.172:0/3525889390 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7f71c81532c0 con 0x7f71c80c9080 2024-03-16T19:29:58.427 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.447+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f71a806b3b0 msgr2=0x7f71a806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.447+0000 7f71cf155640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f71a806b3b0 0x7f71a806d870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f71c0035000 tx=0x7f71c0030000 comp rx=0 tx=0).stop 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.447+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c80c9080 msgr2=0x7f71c81535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.447+0000 7f71cf155640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c80c9080 0x7f71c81535a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f71b000e870 tx=0x7f71b000ed30 comp rx=0 tx=0).stop 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.448+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 shutdown_connections 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.448+0000 7f71cf155640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f71a806b3b0 0x7f71a806d870 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.448+0000 7f71cf155640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f71c8157ec0 0x7f71c80cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.448+0000 7f71cf155640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f71c80c9080 0x7f71c81535a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.448+0000 7f71cf155640 1 --2- 172.21.15.172:0/3525889390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71c806dc20 0x7f71c8151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.448+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 >> 172.21.15.172:0/3525889390 conn(0x7f71c8070810 msgr2=0x7f71c80c94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.448+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 shutdown_connections 2024-03-16T19:29:58.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:29:58.448+0000 7f71cf155640 1 -- 172.21.15.172:0/3525889390 wait complete. 2024-03-16T19:29:58.451 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:58.471+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:58.476 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:58.476 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:58.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:29:58.569 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:58.588+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:58.632 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:58.632 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:58.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:29:58.653 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:58.672+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:58.754 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:58.774+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:58.789 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:58.789 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:58.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:29:58.855 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:58.873+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:58.902 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:58.902 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:58.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:29:58.955 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:58.974+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:59.055 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:29:59.074+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:59.069 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:29:59.156 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:29:59.175+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:29:59.177 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:59.177 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:59.186 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:29:59.257 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:59.276+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:59.296 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:59.297 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:59.306 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:29:59.357 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:59.376+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:59.414 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:59.414 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:59.423 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:29:59.465 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:29:59.483+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:29:59.559 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:29:59.578+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:29:59.571 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:59.571 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:59.581 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:29:59.660 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:59.679+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:59.686 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:59.687 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:59.696 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:29:59.761 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:29:59.780+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:29:59.803 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:59.804 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:59.812 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:29:59.862 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:29:59.881+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:29:59.963 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:29:59.981+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:29:59.963 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:29:59.964 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:29:59.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:29:59.973 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:00.064 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:00.083+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:00.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.095+0000 7f4e9e091640 1 Processor -- start 2024-03-16T19:30:00.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.095+0000 7f4e9e091640 1 -- start start 2024-03-16T19:30:00.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e9e091640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4e98154ed0 0x7f4e9815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e9e091640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e980b4270 con 0x7f4e980b4700 2024-03-16T19:30:00.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e9e091640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e981552b0 con 0x7f4e980b3630 2024-03-16T19:30:00.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e9e091640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9815b4b0 con 0x7f4e98154ed0 2024-03-16T19:30:00.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e977fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4e98154ed0 0x7f4e9815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e97fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4e980b4700 0x7f4e981654c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39724/0 (socket says 172.21.15.172:39724) 2024-03-16T19:30:00.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e977fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4e98154ed0 0x7f4e9815ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47090/0 (socket says 172.21.15.172:47090) 2024-03-16T19:30:00.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e97fff640 1 -- 172.21.15.172:0/768370813 learned_addr learned my addr 172.21.15.172:0/768370813 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:00.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e977fe640 1 -- 172.21.15.172:0/768370813 >> v1:172.21.15.174:6789/0 conn(0x7f4e980b3630 legacy=0x7f4e980cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e977fe640 1 -- 172.21.15.172:0/768370813 >> v1:172.21.15.172:6789/0 conn(0x7f4e980b4700 legacy=0x7f4e981654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e977fe640 1 -- 172.21.15.172:0/768370813 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9815b790 con 0x7f4e98154ed0 2024-03-16T19:30:00.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.096+0000 7f4e967fc640 1 -- 172.21.15.172:0/768370813 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791952911 0 0) 0x7f4e980b4270 con 0x7f4e980b4700 2024-03-16T19:30:00.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e977fe640 1 --2- 172.21.15.172:0/768370813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e98154ed0 0x7f4e9815ab40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0097a0 tx=0x7f4e8c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e53d3550961f31b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e967fc640 1 -- 172.21.15.172:0/768370813 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c030610 con 0x7f4e98154ed0 2024-03-16T19:30:00.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e967fc640 1 -- 172.21.15.172:0/768370813 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c030770 con 0x7f4e98154ed0 2024-03-16T19:30:00.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e967fc640 1 -- 172.21.15.172:0/768370813 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c030b00 con 0x7f4e98154ed0 2024-03-16T19:30:00.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e9e091640 1 -- 172.21.15.172:0/768370813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e98154ed0 msgr2=0x7f4e9815ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e9e091640 1 --2- 172.21.15.172:0/768370813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e98154ed0 0x7f4e9815ab40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0097a0 tx=0x7f4e8c02b960 comp rx=0 tx=0).stop 2024-03-16T19:30:00.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e9e091640 1 -- 172.21.15.172:0/768370813 shutdown_connections 2024-03-16T19:30:00.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e9e091640 1 --2- 172.21.15.172:0/768370813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e98154ed0 0x7f4e9815ab40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e9e091640 1 -- 172.21.15.172:0/768370813 >> 172.21.15.172:0/768370813 conn(0x7f4e9806bf60 msgr2=0x7f4e9806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:00.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.097+0000 7f4e9e091640 1 -- 172.21.15.172:0/768370813 shutdown_connections 2024-03-16T19:30:00.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.098+0000 7f4e9e091640 1 -- 172.21.15.172:0/768370813 wait complete. 2024-03-16T19:30:00.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.099+0000 7f4e9e091640 1 Processor -- start 2024-03-16T19:30:00.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.099+0000 7f4e9e091640 1 -- start start 2024-03-16T19:30:00.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.099+0000 7f4e9e091640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e980b3630 0x7f4e98151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.099+0000 7f4e9e091640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e980b4700 0x7f4e980ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.099+0000 7f4e9e091640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e98154ed0 0x7f4e980c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.099+0000 7f4e9e091640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9815e560 con 0x7f4e980b3630 2024-03-16T19:30:00.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.099+0000 7f4e9e091640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9815e9f0 con 0x7f4e98154ed0 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.099+0000 7f4e9e091640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9815ca50 con 0x7f4e980b4700 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e977fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e980b3630 0x7f4e98151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e977fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e980b3630 0x7f4e98151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47104/0 (socket says 172.21.15.172:47104) 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e977fe640 1 -- 172.21.15.172:0/2075584405 learned_addr learned my addr 172.21.15.172:0/2075584405 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e98154ed0 0x7f4e980c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e96ffd640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e980b4700 0x7f4e980ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e977fe640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e980b4700 msgr2=0x7f4e980ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e977fe640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e980b4700 0x7f4e980ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e977fe640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e98154ed0 msgr2=0x7f4e980c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e977fe640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e98154ed0 0x7f4e980c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e977fe640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c009400 con 0x7f4e980b3630 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e977fe640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e980b3630 0x7f4e98151ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0098c0 tx=0x7f4e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c03e070 con 0x7f4e980b3630 2024-03-16T19:30:00.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c039a10 con 0x7f4e980b3630 2024-03-16T19:30:00.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9815d950 con 0x7f4e980b3630 2024-03-16T19:30:00.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c039da0 con 0x7f4e980b3630 2024-03-16T19:30:00.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.100+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9815df10 con 0x7f4e980b3630 2024-03-16T19:30:00.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.101+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4e98068d20 con 0x7f4e980b3630 2024-03-16T19:30:00.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.102+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4e8c04b400 con 0x7f4e980b3630 2024-03-16T19:30:00.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.102+0000 7f4e7ffff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4e8406b3b0 0x7f4e8406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.102+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f4e8c0abf00 con 0x7f4e980b3630 2024-03-16T19:30:00.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.102+0000 7f4e96ffd640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4e8406b3b0 0x7f4e8406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.102+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v28) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4e8c0ae540 con 0x7f4e980b3630 2024-03-16T19:30:00.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.102+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4e9815e0a0 0x7f4e981f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.102+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e980b2e20 con 0x7f4e9815e0a0 2024-03-16T19:30:00.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.103+0000 7f4e96ffd640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4e8406b3b0 0x7f4e8406d870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4e98152430 tx=0x7f4e88009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.103+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4e9815e0a0 0x7f4e981f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.103+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4e9815e0a0 0x7f4e981f36b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.104+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4e90004e60 con 0x7f4e9815e0a0 2024-03-16T19:30:00.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.118+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e9815ba70 con 0x7f4e9815e0a0 2024-03-16T19:30:00.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.120+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4e90001c90 con 0x7f4e9815e0a0 2024-03-16T19:30:00.101 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:00.102 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:00.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.120+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f4e980c7da0 0x7f4e981f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.120+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e9814b9d0 con 0x7f4e980c7da0 2024-03-16T19:30:00.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.120+0000 7f4e977fe640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f4e980c7da0 0x7f4e981f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.120+0000 7f4e977fe640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f4e980c7da0 0x7f4e981f4100 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.121+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4e9814b9d0 con 0x7f4e980c7da0 2024-03-16T19:30:00.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.135+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e980cbbe0 con 0x7f4e980c7da0 2024-03-16T19:30:00.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.137+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4e980cbbe0 con 0x7f4e980c7da0 2024-03-16T19:30:00.119 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:00.119 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:00.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.137+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f4e98147440 0x7f4e98147820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.137+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e98147d60 con 0x7f4e98147440 2024-03-16T19:30:00.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.137+0000 7f4e96ffd640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f4e98147440 0x7f4e98147820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.137+0000 7f4e96ffd640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f4e98147440 0x7f4e98147820 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.138+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4e98147d60 con 0x7f4e98147440 2024-03-16T19:30:00.126 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:00.126 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:00.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.152+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e980ca590 con 0x7f4e98147440 2024-03-16T19:30:00.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.153+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4e980ca590 con 0x7f4e98147440 2024-03-16T19:30:00.135 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:00.136 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.153+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e98062090 0x7f4e980624b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.153+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e98202c30 con 0x7f4e98062090 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e98062090 0x7f4e980624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e98062090 0x7f4e980624b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e97fff640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e98062090 msgr2=0x7f4e980624b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 24 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e97fff640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e98062090 msgr2=0x7f4e980624b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e98062090 0x7f4e980624b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e98062090 0x7f4e980624b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e98062090 msgr2=0x7f4e980624b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e7ffff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e98062090 0x7f4e980624b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e7ffff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e84075370 0x7f4e84077760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e980ca590 con 0x7f4e84075370 2024-03-16T19:30:00.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=29}) v3 -- 0x7f4e84077ca0 con 0x7f4e980b3630 2024-03-16T19:30:00.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e84075370 0x7f4e84077760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.154+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e84075370 0x7f4e84077760 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.155+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4e980ca590 con 0x7f4e84075370 2024-03-16T19:30:00.139 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:00.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.169+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e98062740 con 0x7f4e84075370 2024-03-16T19:30:00.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.170+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4e98062740 con 0x7f4e84075370 2024-03-16T19:30:00.152 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:00.153 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:00.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.170+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f4e98152d70 0x7f4e98153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.170+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e98153690 con 0x7f4e98152d70 2024-03-16T19:30:00.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.171+0000 7f4e977fe640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f4e98152d70 0x7f4e98153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.171+0000 7f4e977fe640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f4e98152d70 0x7f4e98153150 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.172+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4e98153690 con 0x7f4e98152d70 2024-03-16T19:30:00.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.185+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e98153690 con 0x7f4e98152d70 2024-03-16T19:30:00.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.187+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4e98153690 con 0x7f4e98152d70 2024-03-16T19:30:00.169 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:00.170 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:00.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.187+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f4e9802c2f0 0x7f4e9802c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.187+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e9802cc90 con 0x7f4e9802c2f0 2024-03-16T19:30:00.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.188+0000 7f4e96ffd640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f4e9802c2f0 0x7f4e9802c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.188+0000 7f4e96ffd640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f4e9802c2f0 0x7f4e9802c750 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.189+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4e9802cc90 con 0x7f4e9802c2f0 2024-03-16T19:30:00.183 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:00.202+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:00.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.202+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e9802cc90 con 0x7f4e9802c2f0 2024-03-16T19:30:00.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.205+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4e9802cc90 con 0x7f4e9802c2f0 2024-03-16T19:30:00.186 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:00.186 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:00.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.205+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4e98204db0 0x7f4e982071a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.205+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e98209ff0 con 0x7f4e98204db0 2024-03-16T19:30:00.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.205+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4e98204db0 0x7f4e982071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.205+0000 7f4e97fff640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4e98204db0 0x7f4e982071a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.205+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4e98209ff0 con 0x7f4e98204db0 2024-03-16T19:30:00.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.219+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e98207760 con 0x7f4e98204db0 2024-03-16T19:30:00.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.221+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4e98207760 con 0x7f4e98204db0 2024-03-16T19:30:00.202 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:00.203 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:00.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.221+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4e982095a0 0x7f4e982099c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.221+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e9820d350 con 0x7f4e982095a0 2024-03-16T19:30:00.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.221+0000 7f4e977fe640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4e982095a0 0x7f4e982099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.221+0000 7f4e977fe640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4e982095a0 0x7f4e982099c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.222+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4e9820d350 con 0x7f4e982095a0 2024-03-16T19:30:00.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.236+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e98207930 con 0x7f4e982095a0 2024-03-16T19:30:00.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e7ffff640 1 -- 172.21.15.172:0/2075584405 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4e98207930 con 0x7f4e982095a0 2024-03-16T19:30:00.220 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:00.222 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:00.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4e9815e0a0 msgr2=0x7f4e981f36b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4e9815e0a0 0x7f4e981f36b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f4e980c7da0 msgr2=0x7f4e981f4100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f4e980c7da0 0x7f4e981f4100 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f4e98147440 msgr2=0x7f4e98147820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f4e98147440 0x7f4e98147820 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e84075370 msgr2=0x7f4e84077760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4e84075370 0x7f4e84077760 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f4e98152d70 msgr2=0x7f4e98153150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f4e98152d70 0x7f4e98153150 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f4e9802c2f0 msgr2=0x7f4e9802c750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f4e9802c2f0 0x7f4e9802c750 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e97fff640 1 -- 172.21.15.172:0/2075584405 reap_dead start 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4e98204db0 msgr2=0x7f4e982071a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4e98204db0 0x7f4e982071a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e97fff640 1 -- 172.21.15.172:0/2075584405 reap_dead start 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e97fff640 1 -- 172.21.15.172:0/2075584405 reap_dead start 2024-03-16T19:30:00.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e97fff640 1 -- 172.21.15.172:0/2075584405 reap_dead start 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4e982095a0 msgr2=0x7f4e982099c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4e982095a0 0x7f4e982099c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4e8406b3b0 msgr2=0x7f4e8406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4e8406b3b0 0x7f4e8406d870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4e98152430 tx=0x7f4e88009040 comp rx=0 tx=0).stop 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e980b3630 msgr2=0x7f4e98151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e980b3630 0x7f4e98151ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0098c0 tx=0x7f4e8c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 shutdown_connections 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4e982095a0 0x7f4e982099c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4e98204db0 0x7f4e982071a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4e8406b3b0 0x7f4e8406d870 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 --2- 172.21.15.172:0/2075584405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e980b3630 0x7f4e98151ef0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 >> 172.21.15.172:0/2075584405 conn(0x7f4e9806bf60 msgr2=0x7f4e980cef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 shutdown_connections 2024-03-16T19:30:00.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.240+0000 7f4e9e091640 1 -- 172.21.15.172:0/2075584405 wait complete. 2024-03-16T19:30:00.251 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:00.251 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:00.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:00.251 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:00.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.251 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:00.251 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.912364271000001, 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.912340098000001, 2024-03-16T19:30:00.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:00.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.912334270999999, 2024-03-16T19:30:00.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:30:00.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:30:00.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.912330358999998, 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.266 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.912324377000001, 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:30:00.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:30:00.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:30:00.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.275 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.912320711, 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.280 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.912310050999999, 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.912306162, 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:30:00.284 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:00.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:30:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.287 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.908789470999999, 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:00.291 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.908719036000001, 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.908336688999999, 2024-03-16T19:30:00.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:30:00.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.906570406, 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:00.299 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:30:00.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:30:00.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.904757484000001, 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:25 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:00.306 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.904519703999998, 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189085, 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2099999999999996e-07 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365068+0000", 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9700000000000001e-06 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0020999999999999e-05 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7449999999999998e-06 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365136+0000", 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0973999999999997e-05 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365145+0000", 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9560000000000003e-06 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365253+0000", 2024-03-16T19:30:00.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107798 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.903901735000002, 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:30:00.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.314 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.898040969, 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.465489761000001, 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:00.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:30:00.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:30:00.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:40 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.462209249000001, 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019577200000000001, 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:00.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1600000000000001e-07 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807377+0000", 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7839999999999999e-06 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807384+0000", 2024-03-16T19:30:00.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8410000000000002e-06 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807388+0000", 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0550000000000001e-06 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807431+0000", 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3271e-05 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807441+0000", 2024-03-16T19:30:00.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2320000000000007e-06 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807570+0000", 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000129073 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.897996055, 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.327 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:30:00.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:30:00.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.390383803999999, 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.332 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:30:00.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.335 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.337 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:00.339 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:00.339 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:00.357+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:00.432 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:30:00.432 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:30:00.432 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-16T19:30:00.432 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-16T19:30:00.432 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-16T19:30:00.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-03-16T19:30:00.439 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:00.457+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:00.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:30:00.488 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:00.488 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:00.488 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:00.489 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:00.497 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:00.539 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:00.558+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:00.542 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:00.542 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-16T19:30:00.542 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:00.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.598+0000 7f56e3ba1640 1 Processor -- start 2024-03-16T19:30:00.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.598+0000 7f56e3ba1640 1 -- start start 2024-03-16T19:30:00.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.599+0000 7f56e3ba1640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f56dc0cb560 0x7f56dc15e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.599+0000 7f56e3ba1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 0x7f56dc06c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.599+0000 7f56e3ba1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc0cb360 con 0x7f56dc154b90 2024-03-16T19:30:00.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.599+0000 7f56e3ba1640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56dc15ede0 con 0x7f56dc0cb560 2024-03-16T19:30:00.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.599+0000 7f56e3ba1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc0cb9a0 con 0x7f56dc06bc60 2024-03-16T19:30:00.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.599+0000 7f56e113c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f56dc0cb560 0x7f56dc15e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.599+0000 7f56e113c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f56dc0cb560 0x7f56dc15e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47106/0 (socket says 172.21.15.172:47106) 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.599+0000 7f56e113c640 1 -- 172.21.15.172:0/987669913 learned_addr learned my addr 172.21.15.172:0/987669913 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.599+0000 7f56e213e640 1 --2- 172.21.15.172:0/987669913 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 0x7f56dc06c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.600+0000 7f56e113c640 1 -- 172.21.15.172:0/987669913 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 msgr2=0x7f56dc06c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.600+0000 7f56e113c640 1 --2- 172.21.15.172:0/987669913 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 0x7f56dc06c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.600+0000 7f56e113c640 1 -- 172.21.15.172:0/987669913 >> v1:172.21.15.174:6789/0 conn(0x7f56dc154b90 legacy=0x7f56dc0cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.600+0000 7f56e093b640 1 -- 172.21.15.172:0/987669913 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448945796 0 0) 0x7f56dc0cb360 con 0x7f56dc154b90 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.600+0000 7f56e113c640 1 -- 172.21.15.172:0/987669913 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc15a950 con 0x7f56dc0cb560 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.600+0000 7f56e113c640 1 --2- 172.21.15.172:0/987669913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 0x7f56dc15e8a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009870 tx=0x7f56cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6ae83521bbe456 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.600+0000 7f56e093b640 1 -- 172.21.15.172:0/987669913 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc030840 con 0x7f56dc0cb560 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e093b640 1 -- 172.21.15.172:0/987669913 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc0309a0 con 0x7f56dc0cb560 2024-03-16T19:30:00.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e093b640 1 -- 172.21.15.172:0/987669913 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc030d30 con 0x7f56dc0cb560 2024-03-16T19:30:00.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e3ba1640 1 -- 172.21.15.172:0/987669913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 msgr2=0x7f56dc15e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e3ba1640 1 --2- 172.21.15.172:0/987669913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 0x7f56dc15e8a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009870 tx=0x7f56cc02b980 comp rx=0 tx=0).stop 2024-03-16T19:30:00.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e3ba1640 1 -- 172.21.15.172:0/987669913 shutdown_connections 2024-03-16T19:30:00.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e3ba1640 1 --2- 172.21.15.172:0/987669913 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 0x7f56dc06c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e3ba1640 1 --2- 172.21.15.172:0/987669913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 0x7f56dc15e8a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e3ba1640 1 -- 172.21.15.172:0/987669913 >> 172.21.15.172:0/987669913 conn(0x7f56dc066e00 msgr2=0x7f56dc067210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:00.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e3ba1640 1 -- 172.21.15.172:0/987669913 shutdown_connections 2024-03-16T19:30:00.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.601+0000 7f56e3ba1640 1 -- 172.21.15.172:0/987669913 wait complete. 2024-03-16T19:30:00.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.602+0000 7f56e3ba1640 1 Processor -- start 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.602+0000 7f56e3ba1640 1 -- start start 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e3ba1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 0x7f56dc14f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e3ba1640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 0x7f56dc15da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e3ba1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56dc154b90 0x7f56dc14c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e3ba1640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc1f3810 con 0x7f56dc0cb560 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e3ba1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc15df70 con 0x7f56dc154b90 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e3ba1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc1f4600 con 0x7f56dc06bc60 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e113c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 0x7f56dc15da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e193d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 0x7f56dc14f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e113c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 0x7f56dc15da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47112/0 (socket says 172.21.15.172:47112) 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e213e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56dc154b90 0x7f56dc14c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e113c640 1 -- 172.21.15.172:0/3318867660 learned_addr learned my addr 172.21.15.172:0/3318867660 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e113c640 1 -- 172.21.15.172:0/3318867660 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 msgr2=0x7f56dc14f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e113c640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 0x7f56dc14f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e113c640 1 -- 172.21.15.172:0/3318867660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56dc154b90 msgr2=0x7f56dc14c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:00.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e113c640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56dc154b90 0x7f56dc14c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:00.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.603+0000 7f56e113c640 1 -- 172.21.15.172:0/3318867660 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc0094d0 con 0x7f56dc0cb560 2024-03-16T19:30:00.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.604+0000 7f56e113c640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 0x7f56dc15da30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f56cc000c00 tx=0x7f56cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.604+0000 7f56d27fc640 1 -- 172.21.15.172:0/3318867660 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc03e040 con 0x7f56dc0cb560 2024-03-16T19:30:00.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.604+0000 7f56d27fc640 1 -- 172.21.15.172:0/3318867660 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc039c90 con 0x7f56dc0cb560 2024-03-16T19:30:00.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.604+0000 7f56d27fc640 1 -- 172.21.15.172:0/3318867660 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc004030 con 0x7f56dc0cb560 2024-03-16T19:30:00.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.604+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc1f39f0 con 0x7f56dc0cb560 2024-03-16T19:30:00.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.604+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56dc1f3fb0 con 0x7f56dc0cb560 2024-03-16T19:30:00.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.605+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc075a70 con 0x7f56dc0cb560 2024-03-16T19:30:00.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.605+0000 7f56d27fc640 1 -- 172.21.15.172:0/3318867660 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f56cc004190 con 0x7f56dc0cb560 2024-03-16T19:30:00.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.606+0000 7f56d27fc640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f56c006b3e0 0x7f56c006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:00.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.606+0000 7f56d27fc640 1 -- 172.21.15.172:0/3318867660 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f56cc0ac070 con 0x7f56dc0cb560 2024-03-16T19:30:00.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.606+0000 7f56e193d640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f56c006b3e0 0x7f56c006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:00.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.606+0000 7f56e193d640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f56c006b3e0 0x7f56c006d8a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f56d8000f60 tx=0x7f56d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:00.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.610+0000 7f56d27fc640 1 -- 172.21.15.172:0/3318867660 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56cc076f10 con 0x7f56dc0cb560 2024-03-16T19:30:00.639 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:00.658+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:00.647 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:00.647 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:00.647 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:00.647 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:00.647 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:00.656 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:00.740 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:00.759+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:00.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:00.790+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f56dc0b6180 con 0x7f56dc0cb560 2024-03-16T19:30:00.791 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:00.791 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:00.791 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:00.791 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:00.791 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.791 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.579901739999997, 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:00.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:30:00.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.203206193, 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.427229748999999, 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.801 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.427145336000002, 2024-03-16T19:30:00.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:30:00.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:30:00.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.427137979999998, 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.234598788, 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.817 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:30:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:30:00.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:30:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.234592980999999, 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:30:00.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:30:00.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:30:00.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:30:00.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.234474847999998, 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.828 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:30:00.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:30:00.831 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:30:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.833 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.234379650999998, 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.838 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.234375640000003, 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:30:00.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.815443766000001, 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.844 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.815394396999999, 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:30:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:30:00.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.854 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.815315564999999, 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:30:00.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:30:00.859 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.815312354, 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.860 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:30:00.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:30:00.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:30:00.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.865 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.815309401, 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.815194427000002, 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:30:00.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:30:00.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.815191075000001, 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.876 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:00.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:30:00.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.815187958999999, 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.881 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:30:00.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:30:00.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.815127490000002, 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.886 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:30:00.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:30:00.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.206447439999998, 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:00.891 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:00.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:30:00.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:00.896 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:00.897 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:00.916+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:00.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:00.999 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:01.017+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:01.005 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:01.005 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:01.005 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:01.005 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:01.014 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:01.100 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:01.118+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:01.162 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:01.163 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:01.163 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:01.163 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:01.163 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:01.172 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:01.200 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:01.219+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:01.281 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:01.281 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:01.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:01.281 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.706320212999998, 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:30:01.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:30:01.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.916094041999997, 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:30:01.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.915333807000003, 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:01.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:01.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.914519851999998, 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.297 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:30:01.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.913879926999996, 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:30:01.302 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:30:01.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.913221829999998, 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:30:01.307 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:30:01.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:30:01.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:30:01.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:30:01.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:30:01.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:01.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.912689254999997, 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:30:01.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:30:01.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.854517352000002, 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:01.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:30:01.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.853915158, 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:30:01.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.853272212999997, 2024-03-16T19:30:01.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.852768251999997, 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:30:01.335 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:30:01.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.694621986000001, 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:30:01.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:30:01.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.69409185, 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:30:01.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:30:01.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.942122591, 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:30:01.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.356 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.942098054999999, 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:30:01.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:01.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.942093525000001, 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:01.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:30:01.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:30:01.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.368 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.942083712999999, 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:01.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:30:01.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.94207986, 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:30:01.375 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:30:01.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:30:01.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.381 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.942076543999999, 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.940880961000001, 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:30:01.386 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:30:01.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:30:01.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:30:01.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.391 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.391 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.391 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:01.393 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:01.393 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:01.411+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:01.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.457+0000 7f56d27fc640 1 -- 172.21.15.172:0/3318867660 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v29) v1 ==== 91+0+0 (secure 0 0 0) 0x7f56cc07a5d0 con 0x7f56dc0cb560 2024-03-16T19:30:01.439 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:30:01.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f56c006b3e0 msgr2=0x7f56c006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:01.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f56c006b3e0 0x7f56c006d8a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f56d8000f60 tx=0x7f56d8009040 comp rx=0 tx=0).stop 2024-03-16T19:30:01.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 msgr2=0x7f56dc15da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:01.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 0x7f56dc15da30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f56cc000c00 tx=0x7f56cc004730 comp rx=0 tx=0).stop 2024-03-16T19:30:01.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 shutdown_connections 2024-03-16T19:30:01.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56dc154b90 0x7f56dc14c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:01.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56dc0cb560 0x7f56dc15da30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:01.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f56c006b3e0 0x7f56c006d8a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:01.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 --2- 172.21.15.172:0/3318867660 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56dc06bc60 0x7f56dc14f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:01.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 >> 172.21.15.172:0/3318867660 conn(0x7f56dc066e00 msgr2=0x7f56dc0c5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:01.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 shutdown_connections 2024-03-16T19:30:01.442 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:01.459+0000 7f56e3ba1640 1 -- 172.21.15.172:0/3318867660 wait complete. 2024-03-16T19:30:01.493 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:01.512+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:01.504 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:01.504 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:01.504 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:01.504 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:01.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:01.594 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:01.613+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:01.659 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:01.659 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:01.659 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:01.659 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:01.659 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:01.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:01.711 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:01.730+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.437272298, 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.776 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:30:01.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:30:01.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:30:01.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.437255184000001, 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.781 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:01.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:30:01.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:30:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.437247903999999, 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:30:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:30:01.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.437234037, 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:01.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:30:01.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.436561202, 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.800 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.434262952000001, 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:01.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.434217851, 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.434189795999998, 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.434168381999999, 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:30:01.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:30:01.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:30:01.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.434146031000001, 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.817 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:30:01.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.434125632000001, 2024-03-16T19:30:01.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:30:01.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.825 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.434019639999999, 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:30:01.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:30:01.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.432227929, 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.432180427999999, 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:30:01.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:30:01.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.431941176999999, 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:30 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:01.841 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.431785598000001, 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126666, 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362824+0000", 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7999999999999995e-07 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362828+0000", 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0879999999999997e-06 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362837+0000", 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.704e-06 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362843+0000", 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9390000000000002e-06 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362877+0000", 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4779e-05 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362883+0000", 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9009999999999999e-06 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362950+0000", 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6674999999999994e-05 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.423443032000002, 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.422103155999999, 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:01.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:30:01.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:01.853 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:01.855 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:01.855 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:01.874+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:01.956 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:01.974+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:02.002 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:02.002 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:02.002 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:02.002 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:02.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:02.057 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:02.075+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:02.118 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:02.119 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:02.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:02.158 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:02.177+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:02.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:02.236 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:02.236 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897535402999999, 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053568110000000004, 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357146+0000", 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5860000000000001e-06 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357154+0000", 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5439999999999999e-06 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357217+0000", 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2986999999999999e-05 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357229+0000", 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1853e-05 2024-03-16T19:30:02.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357343+0000", 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113663 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357504+0000", 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016176 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:02.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013908599999999999 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360705+0000", 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030619419999999998 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362409+0000", 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001704021 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362456+0000", 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6140000000000002e-05 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362496+0000", 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0228999999999997e-05 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897515376000001, 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078702890000000008, 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4299999999999998e-07 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357163+0000", 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2559999999999998e-06 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357254+0000", 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1701999999999997e-05 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357258+0000", 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5719999999999999e-06 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357931+0000", 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067308000000000005 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357999+0000", 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8323000000000003e-05 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358113+0000", 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011366699999999999 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362505+0000", 2024-03-16T19:30:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004391509 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024372970000000002 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364992+0000", 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9741999999999999e-05 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365029+0000", 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7697999999999997e-05 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897508574, 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079944349999999994, 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6e-07 2024-03-16T19:30:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357172+0000", 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7960000000000001e-06 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357285+0000", 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113183 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357290+0000", 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6190000000000002e-06 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358306+0000", 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010160670000000001 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358400+0000", 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3854e-05 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358492+0000", 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2954000000000005e-05 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362510+0000", 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040175469999999998 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025680310000000001 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365123+0000", 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4515000000000003e-05 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365160+0000", 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7608999999999999e-05 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897501678000001, 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080787759999999993, 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.36e-07 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1270000000000001e-06 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357297+0000", 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121885 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357300+0000", 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049999999999999e-06 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358709+0000", 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001409083 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358776+0000", 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6489999999999995e-05 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358885+0000", 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109271 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362514+0000", 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036292199999999998 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365197+0000", 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026832100000000001 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365226+0000", 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9136999999999999e-05 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365251+0000", 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5012e-05 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897498693999999, 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081787609999999997, 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357177+0000", 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5090000000000001e-06 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357306+0000", 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012856900000000001 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357308+0000", 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3700000000000002e-06 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359071+0000", 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017631210000000001 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359182+0000", 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110265 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359280+0000", 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8595e-05 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362520+0000", 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032394120000000001 2024-03-16T19:30:02.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365297+0000", 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002777542 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365328+0000", 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0899999999999999e-05 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365354+0000", 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6276000000000001e-05 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.263 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897496006000001, 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083197830000000007, 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357179+0000", 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357187+0000", 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357314+0000", 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126978 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357316+0000", 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3800000000000001e-06 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359512+0000", 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021950799999999999 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359585+0000", 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3168000000000001e-05 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359702+0000", 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011741 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362525+0000", 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028225419999999999 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365408+0000", 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028829010000000002 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365460+0000", 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839e-05 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365498+0000", 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7673999999999999e-05 2024-03-16T19:30:02.268 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897484881, 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084493169999999996, 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357190+0000", 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-07 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357191+0000", 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7069999999999999e-06 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357325+0000", 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133585 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357330+0000", 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0799999999999996e-06 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359915+0000", 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025849290000000001 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9741999999999995e-05 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360121+0000", 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362826+0000", 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027046819999999999 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365564+0000", 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027387179999999998 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365606+0000", 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1715999999999998e-05 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365639+0000", 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660999999999998e-05 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897482212, 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085533069999999996, 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.274 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:02.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0699999999999999e-07 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3030000000000001e-06 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357339+0000", 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014513199999999999 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:02.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000003e-06 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030087619999999999 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360414+0000", 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1297999999999995e-05 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:02.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360498+0000", 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4538999999999998e-05 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362832+0000", 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023338410000000001 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365670+0000", 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028379550000000001 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365724+0000", 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3533999999999999e-05 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365745+0000", 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1725999999999998e-05 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.279 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897478988, 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052850789999999998, 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357196+0000", 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.173e-06 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357359+0000", 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016222400000000001 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357365+0000", 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7320000000000003e-06 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357451+0000", 2024-03-16T19:30:02.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6494000000000002e-05 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357555+0000", 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010401200000000001 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8224000000000001e-05 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362294+0000", 2024-03-16T19:30:02.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046502399999999999 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362388+0000", 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4448000000000001e-05 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362433+0000", 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4888000000000001e-05 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362480+0000", 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7441000000000002e-05 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.897476951000002, 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053762239999999998, 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357198+0000", 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:02.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357199+0000", 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357415+0000", 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021638699999999999 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:02.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0069e-05 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358132+0000", 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070650599999999997 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358207+0000", 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5197999999999994e-05 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:02.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358342+0000", 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013503199999999999 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362377+0000", 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040354600000000003 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362493+0000", 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115951 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362536+0000", 2024-03-16T19:30:02.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3189999999999998e-05 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362574+0000", 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7113999999999997e-05 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:6 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.894418049999999, 2024-03-16T19:30:02.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045449999999999999, 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360257+0000", 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7400000000000002e-07 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360260+0000", 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3550000000000001e-06 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360270+0000", 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360277+0000", 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5760000000000002e-06 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360697+0000", 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042039200000000001 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360702+0000", 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7140000000000001e-06 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360711+0000", 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1209999999999996e-06 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:22 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.892473875, 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100673, 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.294 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362201+0000", 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5799999999999998e-07 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362206+0000", 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8579999999999999e-06 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362219+0000", 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.239e-05 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362226+0000", 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2710000000000003e-06 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362281+0000", 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4756000000000002e-05 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362288+0000", 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.075e-06 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362301+0000", 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3365e-05 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:23 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.892362605999999, 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017205200000000001, 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4499999999999997e-07 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362315+0000", 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1350000000000001e-06 2024-03-16T19:30:02.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362322+0000", 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9469999999999997e-06 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362328+0000", 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362366+0000", 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8160000000000001e-05 2024-03-16T19:30:02.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362373+0000", 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5799999999999997e-06 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362484+0000", 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111173 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:7 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.301 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.889818733999999, 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018177499999999999, 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:02.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364861+0000", 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3699999999999997e-06 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364875+0000", 2024-03-16T19:30:02.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4059e-05 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364881+0000", 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5270000000000004e-06 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364929+0000", 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000002e-05 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:02.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3699999999999997e-06 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365037+0000", 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100819 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:8 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.889761676999999, 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025451199999999998, 2024-03-16T19:30:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364916+0000", 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8109999999999999e-06 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364924+0000", 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1419999999999993e-06 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364930+0000", 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2260000000000002e-06 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365071+0000", 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014030100000000001 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365075+0000", 2024-03-16T19:30:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365167+0000", 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2196999999999998e-05 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:9 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.889735436999999, 2024-03-16T19:30:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032082900000000001, 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.22e-07 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3980000000000002e-06 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364947+0000", 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3090000000000003e-06 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364951+0000", 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1309999999999999e-06 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365191+0000", 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023992600000000001 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365195+0000", 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.704e-06 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365260+0000", 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5139000000000001e-05 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:10 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.889707050999998, 2024-03-16T19:30:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039415499999999998, 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364968+0000", 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364970+0000", 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4650000000000001e-06 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364977+0000", 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.443e-06 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364981+0000", 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2320000000000003e-06 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365289+0000", 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030856900000000002 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365294+0000", 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6920000000000001e-06 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365361+0000", 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7358000000000001e-05 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:11 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.889686551, 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051803199999999998, 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9299999999999999e-07 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364993+0000", 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9080000000000003e-06 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4240000000000001e-06 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365002+0000", 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7570000000000002e-06 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365395+0000", 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039237199999999999 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365401+0000", 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9380000000000003e-06 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365506+0000", 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010433999999999999 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:12 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.889664546999999, 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063430500000000005, 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9500000000000001e-07 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365012+0000", 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7129999999999999e-06 2024-03-16T19:30:02.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365015+0000", 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2629999999999999e-06 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365017+0000", 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1749999999999999e-06 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365544+0000", 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052690099999999998 2024-03-16T19:30:02.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365559+0000", 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4610000000000001e-05 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365644+0000", 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5447999999999999e-05 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.324 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:13 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.889654049000001, 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00073113500000000005, 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:02.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1000000000000001e-07 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365021+0000", 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.23e-07 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:02.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365024+0000", 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7369999999999998e-06 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365026+0000", 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1289999999999999e-06 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365663+0000", 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063679600000000002 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365667+0000", 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.233e-06 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365751+0000", 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4307000000000004e-05 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:02.328 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:02.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:02.330 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:02.349+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:02.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.352+0000 7fd5d5064640 1 Processor -- start 2024-03-16T19:30:02.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.352+0000 7fd5d5064640 1 -- start start 2024-03-16T19:30:02.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5d5064640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d00c5190 0x7fd5d00cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5d5064640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d015b900 con 0x7fd5d0154a70 2024-03-16T19:30:02.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5d5064640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d00cad10 con 0x7fd5d0154e50 2024-03-16T19:30:02.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5d5064640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d015ad80 con 0x7fd5d00c5190 2024-03-16T19:30:02.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5ced9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd5d0154a70 0x7fd5d006c490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39726/0 (socket says 172.21.15.172:39726) 2024-03-16T19:30:02.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5ced9d640 1 -- 172.21.15.172:0/2511872771 learned_addr learned my addr 172.21.15.172:0/2511872771 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:02.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5cf59e640 1 --2- 172.21.15.172:0/2511872771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d00c5190 0x7fd5d00cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5ce59c640 1 -- 172.21.15.172:0/2511872771 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985132818 0 0) 0x7fd5d015b900 con 0x7fd5d0154a70 2024-03-16T19:30:02.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5ce59c640 1 -- 172.21.15.172:0/2511872771 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5bc003520 con 0x7fd5d0154a70 2024-03-16T19:30:02.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5ce59c640 1 -- 172.21.15.172:0/2511872771 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871099790 0 0) 0x7fd5d00cad10 con 0x7fd5d0154e50 2024-03-16T19:30:02.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5ce59c640 1 -- 172.21.15.172:0/2511872771 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d015b900 con 0x7fd5d0154e50 2024-03-16T19:30:02.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5cf59e640 1 -- 172.21.15.172:0/2511872771 >> v1:172.21.15.174:6789/0 conn(0x7fd5d0154e50 legacy=0x7fd5d006b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5cf59e640 1 -- 172.21.15.172:0/2511872771 >> v1:172.21.15.172:6789/0 conn(0x7fd5d0154a70 legacy=0x7fd5d006c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.353+0000 7fd5cf59e640 1 -- 172.21.15.172:0/2511872771 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d015b060 con 0x7fd5d00c5190 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.354+0000 7fd5cf59e640 1 --2- 172.21.15.172:0/2511872771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d00c5190 0x7fd5d00cb1a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8009dc0 tx=0x7fd5c802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9c2efd253158a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.354+0000 7fd5ce59c640 1 -- 172.21.15.172:0/2511872771 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c802cc40 con 0x7fd5d00c5190 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.354+0000 7fd5ce59c640 1 -- 172.21.15.172:0/2511872771 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c802cda0 con 0x7fd5d00c5190 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.354+0000 7fd5ce59c640 1 -- 172.21.15.172:0/2511872771 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8034660 con 0x7fd5d00c5190 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.355+0000 7fd5d5064640 1 -- 172.21.15.172:0/2511872771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d00c5190 msgr2=0x7fd5d00cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.355+0000 7fd5d5064640 1 --2- 172.21.15.172:0/2511872771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d00c5190 0x7fd5d00cb1a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8009dc0 tx=0x7fd5c802be90 comp rx=0 tx=0).stop 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.355+0000 7fd5d5064640 1 -- 172.21.15.172:0/2511872771 shutdown_connections 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.355+0000 7fd5d5064640 1 --2- 172.21.15.172:0/2511872771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d00c5190 0x7fd5d00cb1a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.355+0000 7fd5d5064640 1 -- 172.21.15.172:0/2511872771 >> 172.21.15.172:0/2511872771 conn(0x7fd5d0067250 msgr2=0x7fd5d00d01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.355+0000 7fd5d5064640 1 -- 172.21.15.172:0/2511872771 shutdown_connections 2024-03-16T19:30:02.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.355+0000 7fd5d5064640 1 -- 172.21.15.172:0/2511872771 wait complete. 2024-03-16T19:30:02.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.356+0000 7fd5d5064640 1 Processor -- start 2024-03-16T19:30:02.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.356+0000 7fd5d5064640 1 -- start start 2024-03-16T19:30:02.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5d5064640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5d00c5190 0x7fd5d01554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5d5064640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d0154a70 0x7fd5d01574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5d5064640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd5d0154e50 0x7fd5d0157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5d5064640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00ca390 con 0x7fd5d00c5190 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5d5064640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00c9ca0 con 0x7fd5d0154e50 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5d5064640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00c9580 con 0x7fd5d0154a70 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5ced9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5d00c5190 0x7fd5d01554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5ced9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5d00c5190 0x7fd5d01554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47116/0 (socket says 172.21.15.172:47116) 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5ced9d640 1 -- 172.21.15.172:0/4007014233 learned_addr learned my addr 172.21.15.172:0/4007014233 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5c7fff640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d0154a70 0x7fd5d01574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5cf59e640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd5d0154e50 0x7fd5d0157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5ced9d640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d0154a70 msgr2=0x7fd5d01574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5ced9d640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5d0154a70 0x7fd5d01574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5ced9d640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd5d0154e50 msgr2=0x7fd5d0157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5ced9d640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd5d0154e50 0x7fd5d0157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.357+0000 7fd5ced9d640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c8009a70 con 0x7fd5d00c5190 2024-03-16T19:30:02.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.358+0000 7fd5ced9d640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5d00c5190 0x7fd5d01554a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c530 tx=0x7fd5c000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.358+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0005ce0 con 0x7fd5d00c5190 2024-03-16T19:30:02.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.358+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c0005e60 con 0x7fd5d00c5190 2024-03-16T19:30:02.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.358+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c00072b0 con 0x7fd5d00c5190 2024-03-16T19:30:02.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.358+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d00c92e0 con 0x7fd5d00c5190 2024-03-16T19:30:02.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.358+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d00ca0b0 con 0x7fd5d00c5190 2024-03-16T19:30:02.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.359+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd5d00681c0 con 0x7fd5d00c5190 2024-03-16T19:30:02.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.359+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd5c0007410 con 0x7fd5d00c5190 2024-03-16T19:30:02.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.360+0000 7fd5c77fe640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd5b006b3b0 0x7fd5b006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.360+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5741+0+0 (secure 0 0 0) 0x7fd5c0089240 con 0x7fd5d00c5190 2024-03-16T19:30:02.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.360+0000 7fd5c7fff640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd5b006b3b0 0x7fd5b006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.360+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v29) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd5c0090070 con 0x7fd5d00c5190 2024-03-16T19:30:02.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.360+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd5d00c8190 0x7fd5d01f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.360+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5d00b4e80 con 0x7fd5d00c8190 2024-03-16T19:30:02.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.360+0000 7fd5cf59e640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd5d00c8190 0x7fd5d01f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.360+0000 7fd5c7fff640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd5b006b3b0 0x7fd5b006d870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8007fd0 tx=0x7fd5b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.361+0000 7fd5cf59e640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd5d00c8190 0x7fd5d01f3270 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.361+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd5d00b4e80 con 0x7fd5d00c8190 2024-03-16T19:30:02.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.376+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd5d00c61a0 con 0x7fd5d00c8190 2024-03-16T19:30:02.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.378+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd5d00c61a0 con 0x7fd5d00c8190 2024-03-16T19:30:02.358 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:02.359 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:02.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.378+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd5d00c8ac0 0x7fd5d01f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.378+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5d01559e0 con 0x7fd5d00c8ac0 2024-03-16T19:30:02.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.378+0000 7fd5ced9d640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd5d00c8ac0 0x7fd5d01f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.378+0000 7fd5ced9d640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd5d00c8ac0 0x7fd5d01f3cc0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.379+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd5d01559e0 con 0x7fd5d00c8ac0 2024-03-16T19:30:02.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.395+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd5d00c9790 con 0x7fd5d00c8ac0 2024-03-16T19:30:02.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.396+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd5d00c9790 con 0x7fd5d00c8ac0 2024-03-16T19:30:02.377 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:02.377 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:02.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.396+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fd5d014acb0 0x7fd5d014b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.396+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5d0159520 con 0x7fd5d014acb0 2024-03-16T19:30:02.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.397+0000 7fd5c7fff640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fd5d014acb0 0x7fd5d014b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.397+0000 7fd5c7fff640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fd5d014acb0 0x7fd5d014b0b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.398+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd5d0159520 con 0x7fd5d014acb0 2024-03-16T19:30:02.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.412+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd5d0159970 con 0x7fd5d014acb0 2024-03-16T19:30:02.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.413+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd5d0159970 con 0x7fd5d014acb0 2024-03-16T19:30:02.394 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:02.394 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:02.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.413+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd5d0061930 0x7fd5d0061d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.413+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5d015a410 con 0x7fd5d0061930 2024-03-16T19:30:02.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.414+0000 7fd5cf59e640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd5d0061930 0x7fd5d0061d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.414+0000 7fd5cf59e640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd5d0061930 0x7fd5d0061d30 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.415+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd5d015a410 con 0x7fd5d0061930 2024-03-16T19:30:02.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.429+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd5d00624c0 con 0x7fd5d0061930 2024-03-16T19:30:02.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.430+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd5d00624c0 con 0x7fd5d0061930 2024-03-16T19:30:02.411 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:02.412 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:02.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.431+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd5d0062690 0x7fd5d002c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.431+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5d002c830 con 0x7fd5d0062690 2024-03-16T19:30:02.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.431+0000 7fd5ced9d640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd5d0062690 0x7fd5d002c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.431+0000 7fd5ced9d640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd5d0062690 0x7fd5d002c2f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.432+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd5d002c830 con 0x7fd5d0062690 2024-03-16T19:30:02.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.446+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd5d002c830 con 0x7fd5d0062690 2024-03-16T19:30:02.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.448+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd5d002c830 con 0x7fd5d0062690 2024-03-16T19:30:02.428 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:02.429 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:02.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.448+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd5d002ceb0 0x7fd5d0206880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.448+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5d002d310 con 0x7fd5d002ceb0 2024-03-16T19:30:02.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.448+0000 7fd5c7fff640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd5d002ceb0 0x7fd5d0206880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.448+0000 7fd5c7fff640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd5d002ceb0 0x7fd5d0206880 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.449+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd5d002d310 con 0x7fd5d002ceb0 2024-03-16T19:30:02.430 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:02.449+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:02.443 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:02.443 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:02.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.464+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd5d002d310 con 0x7fd5d002ceb0 2024-03-16T19:30:02.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.466+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd5d002d310 con 0x7fd5d002ceb0 2024-03-16T19:30:02.447 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:02.447 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:02.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.466+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd5d02072d0 0x7fd5d02096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.466+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5d020c490 con 0x7fd5d02072d0 2024-03-16T19:30:02.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.466+0000 7fd5cf59e640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd5d02072d0 0x7fd5d02096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.467+0000 7fd5cf59e640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd5d02072d0 0x7fd5d02096c0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.468+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd5d020c490 con 0x7fd5d02072d0 2024-03-16T19:30:02.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:02.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.482+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd5d0209c00 con 0x7fd5d02072d0 2024-03-16T19:30:02.464 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:02.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.483+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd5d0209c00 con 0x7fd5d02072d0 2024-03-16T19:30:02.465 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:02.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.484+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd5d020a280 0x7fd5d020c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:02.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.484+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5d020f510 con 0x7fd5d020a280 2024-03-16T19:30:02.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.484+0000 7fd5ced9d640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd5d020a280 0x7fd5d020c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:02.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.484+0000 7fd5ced9d640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd5d020a280 0x7fd5d020c670 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:02.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.485+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd5d020f510 con 0x7fd5d020a280 2024-03-16T19:30:02.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.499+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd5d0062350 con 0x7fd5d020a280 2024-03-16T19:30:02.481 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5c77fe640 1 -- 172.21.15.172:0/4007014233 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd5d0062350 con 0x7fd5d020a280 2024-03-16T19:30:02.482 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:02.483 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:02.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd5d00c8190 msgr2=0x7fd5d01f3270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd5d00c8190 0x7fd5d01f3270 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd5d00c8ac0 msgr2=0x7fd5d01f3cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd5d00c8ac0 0x7fd5d01f3cc0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fd5d014acb0 msgr2=0x7fd5d014b0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fd5d014acb0 0x7fd5d014b0b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd5d0061930 msgr2=0x7fd5d0061d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd5d0061930 0x7fd5d0061d30 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5cf59e640 1 -- 172.21.15.172:0/4007014233 reap_dead start 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd5d0062690 msgr2=0x7fd5d002c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd5d0062690 0x7fd5d002c2f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5cf59e640 1 -- 172.21.15.172:0/4007014233 reap_dead start 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd5d002ceb0 msgr2=0x7fd5d0206880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd5d002ceb0 0x7fd5d0206880 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd5d02072d0 msgr2=0x7fd5d02096c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd5d02072d0 0x7fd5d02096c0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd5d020a280 msgr2=0x7fd5d020c670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd5d020a280 0x7fd5d020c670 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd5b006b3b0 msgr2=0x7fd5b006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd5b006b3b0 0x7fd5b006d870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8007fd0 tx=0x7fd5b8009040 comp rx=0 tx=0).stop 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5d00c5190 msgr2=0x7fd5d01554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5cf59e640 1 -- 172.21.15.172:0/4007014233 reap_dead start 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5d00c5190 0x7fd5d01554a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c530 tx=0x7fd5c000c9f0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 shutdown_connections 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 --2- 172.21.15.172:0/4007014233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5d00c5190 0x7fd5d01554a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 >> 172.21.15.172:0/4007014233 conn(0x7fd5d0067250 msgr2=0x7fd5d006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 shutdown_connections 2024-03-16T19:30:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:02.501+0000 7fd5d5064640 1 -- 172.21.15.172:0/4007014233 wait complete. 2024-03-16T19:30:02.534 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:02.554+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:02.569 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:02.569 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:02.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:02.662 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:02.682+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:02.726 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:02.726 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:02.733 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:02.752+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:02.735 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:02.834 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:02.852+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:02.840 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:02.841 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:02.849 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:02.935 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:02.954+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:02.997 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:02.997 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:03.006 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:03.036 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:03.055+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:03.136 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:03.155+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:03.139 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:03.140 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:03.151 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:03.237 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:03.256+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:03.299 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:03.299 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:03.308 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:03.339 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:03.358+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:03.439 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:03.459+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:03.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:30:03.456 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:03.456 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:03.466 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:03.540 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:03.559+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:03.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.581+0000 7f35a3064640 1 Processor -- start 2024-03-16T19:30:03.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.581+0000 7f35a3064640 1 -- start start 2024-03-16T19:30:03.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.581+0000 7f35a3064640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f359c154e50 0x7f359c0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.581+0000 7f35a3064640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c06b110 con 0x7f359c0caf40 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.581+0000 7f35a3064640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c15b8d0 con 0x7f359c0b5f10 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.581+0000 7f35a3064640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359c15ad50 con 0x7f359c154e50 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f35a0e00640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f359c154e50 0x7f359c0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f35a0e00640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f359c154e50 0x7f359c0cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:36246/0 (socket says 172.21.15.172:36246) 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f35a1601640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f359c0caf40 0x7f359c165260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39734/0 (socket says 172.21.15.172:39734) 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f35a0e00640 1 -- 172.21.15.172:0/1304743816 learned_addr learned my addr 172.21.15.172:0/1304743816 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f359b7fe640 1 -- 172.21.15.172:0/1304743816 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775854696 0 0) 0x7f359c06b110 con 0x7f359c0caf40 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f359b7fe640 1 -- 172.21.15.172:0/1304743816 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3584003520 con 0x7f359c0caf40 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f359b7fe640 1 -- 172.21.15.172:0/1304743816 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402120810 0 0) 0x7f359c15b8d0 con 0x7f359c0b5f10 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f35a0e00640 1 -- 172.21.15.172:0/1304743816 >> v1:172.21.15.174:6789/0 conn(0x7f359c0b5f10 legacy=0x7f359c06b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:03.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f35a0e00640 1 -- 172.21.15.172:0/1304743816 >> v1:172.21.15.172:6789/0 conn(0x7f359c0caf40 legacy=0x7f359c165260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.582+0000 7f35a0e00640 1 -- 172.21.15.172:0/1304743816 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c15b030 con 0x7f359c154e50 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.583+0000 7f35a0e00640 1 --2- 172.21.15.172:0/1304743816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f359c154e50 0x7f359c0cbc30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f358c009e50 tx=0x7f358c02c090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=114b16776c33f86f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.583+0000 7f359b7fe640 1 -- 172.21.15.172:0/1304743816 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c02cc30 con 0x7f359c154e50 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.583+0000 7f359b7fe640 1 -- 172.21.15.172:0/1304743816 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c02cd90 con 0x7f359c154e50 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.583+0000 7f359b7fe640 1 -- 172.21.15.172:0/1304743816 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c0345f0 con 0x7f359c154e50 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.584+0000 7f35a3064640 1 -- 172.21.15.172:0/1304743816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f359c154e50 msgr2=0x7f359c0cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.584+0000 7f35a3064640 1 --2- 172.21.15.172:0/1304743816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f359c154e50 0x7f359c0cbc30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f358c009e50 tx=0x7f358c02c090 comp rx=0 tx=0).stop 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.584+0000 7f35a3064640 1 -- 172.21.15.172:0/1304743816 shutdown_connections 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.584+0000 7f35a3064640 1 --2- 172.21.15.172:0/1304743816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f359c154e50 0x7f359c0cbc30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.584+0000 7f35a3064640 1 -- 172.21.15.172:0/1304743816 >> 172.21.15.172:0/1304743816 conn(0x7f359c066c20 msgr2=0x7f359c067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.584+0000 7f35a3064640 1 -- 172.21.15.172:0/1304743816 shutdown_connections 2024-03-16T19:30:03.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.584+0000 7f35a3064640 1 -- 172.21.15.172:0/1304743816 wait complete. 2024-03-16T19:30:03.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.585+0000 7f35a3064640 1 Processor -- start 2024-03-16T19:30:03.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.585+0000 7f35a3064640 1 -- start start 2024-03-16T19:30:03.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a3064640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f359c0b5f10 0x7f359c0c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:03.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a3064640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f359c0caf40 0x7f359c1f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:03.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a3064640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f359c154e50 0x7f359c1503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a3064640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c0c6f60 con 0x7f359c154e50 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a3064640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1f12d0 con 0x7f359c0caf40 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a3064640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c0c4300 con 0x7f359c0b5f10 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a1601640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f359c154e50 0x7f359c1503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a1601640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f359c154e50 0x7f359c1503b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47118/0 (socket says 172.21.15.172:47118) 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a0e00640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f359c0b5f10 0x7f359c0c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a1601640 1 -- 172.21.15.172:0/3538597706 learned_addr learned my addr 172.21.15.172:0/3538597706 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f359bfff640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f359c0caf40 0x7f359c1f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a1601640 1 -- 172.21.15.172:0/3538597706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f359c0b5f10 msgr2=0x7f359c0c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a1601640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f359c0b5f10 0x7f359c0c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a1601640 1 -- 172.21.15.172:0/3538597706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f359c0caf40 msgr2=0x7f359c1f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a1601640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f359c0caf40 0x7f359c1f0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a1601640 1 -- 172.21.15.172:0/3538597706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c009b30 con 0x7f359c154e50 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a1601640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f359c154e50 0x7f359c1503b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f359400c430 tx=0x7f359400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35997fa640 1 -- 172.21.15.172:0/3538597706 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35940074e0 con 0x7f359c154e50 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35997fa640 1 -- 172.21.15.172:0/3538597706 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594007640 con 0x7f359c154e50 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35997fa640 1 -- 172.21.15.172:0/3538597706 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594004a20 con 0x7f359c154e50 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.586+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c15ab20 con 0x7f359c154e50 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.587+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c0c48c0 con 0x7f359c154e50 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.588+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c0c9ab0 con 0x7f359c154e50 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.588+0000 7f35997fa640 1 -- 172.21.15.172:0/3538597706 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3594004b80 con 0x7f359c154e50 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.588+0000 7f35997fa640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f358806b3b0 0x7f358806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.589+0000 7f35997fa640 1 -- 172.21.15.172:0/3538597706 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f3594089620 con 0x7f359c154e50 2024-03-16T19:30:03.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.589+0000 7f35a0e00640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f358806b3b0 0x7f358806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:03.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.589+0000 7f35a0e00640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f358806b3b0 0x7f358806d870 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f358c035000 tx=0x7f358c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:03.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.593+0000 7f35997fa640 1 -- 172.21.15.172:0/3538597706 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3594054280 con 0x7f359c154e50 2024-03-16T19:30:03.593 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:03.593 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:03.602 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:03.641 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:03.661+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:03.742 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:03.761+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:03.749 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:03.749 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:03.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:03.774+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f359c075a70 con 0x7f359c154e50 2024-03-16T19:30:03.758 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:03.843 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:03.862+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:03.881 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:03.881 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:03.892 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:03.976 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:03.995+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:04.040 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:04.040 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:04.045 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:04.065+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:04.049 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:04.146 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:04.165+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:04.199 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:04.199 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:04.207 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:04.248 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:04.267+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:04.348 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:04.367+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:04.363 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:04.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.464+0000 7f35997fa640 1 -- 172.21.15.172:0/3538597706 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v30) v1 ==== 101+0+0 (secure 0 0 0) 0x7f3594057940 con 0x7f359c154e50 2024-03-16T19:30:04.445 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:30:04.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f358806b3b0 msgr2=0x7f358806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f358806b3b0 0x7f358806d870 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f358c035000 tx=0x7f358c030000 comp rx=0 tx=0).stop 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f359c154e50 msgr2=0x7f359c1503b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f359c154e50 0x7f359c1503b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f359400c430 tx=0x7f359400c8f0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 shutdown_connections 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f359c154e50 0x7f359c1503b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f359c0caf40 0x7f359c1f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f358806b3b0 0x7f358806d870 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 --2- 172.21.15.172:0/3538597706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f359c0b5f10 0x7f359c0c6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 >> 172.21.15.172:0/3538597706 conn(0x7f359c066c20 msgr2=0x7f359c0cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 shutdown_connections 2024-03-16T19:30:04.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.467+0000 7f35a3064640 1 -- 172.21.15.172:0/3538597706 wait complete. 2024-03-16T19:30:04.449 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:04.468+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:04.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:04.520 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:04.521 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:04.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:04.587 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:04.607+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:04.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.624+0000 7f3788c5c640 1 Processor -- start 2024-03-16T19:30:04.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.624+0000 7f3788c5c640 1 -- start start 2024-03-16T19:30:04.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.625+0000 7f3788c5c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37840b4700 0x7f37840b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.625+0000 7f3788c5c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37840b4270 con 0x7f37840b3630 2024-03-16T19:30:04.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.625+0000 7f3788c5c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378415b250 con 0x7f3784154ed0 2024-03-16T19:30:04.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.625+0000 7f3788c5c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37840b5050 con 0x7f37840b4700 2024-03-16T19:30:04.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.626+0000 7f3782d9d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f37840b4700 0x7f37840b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.626+0000 7f3781d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f37840b3630 0x7f37840cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39740/0 (socket says 172.21.15.172:39740) 2024-03-16T19:30:04.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.626+0000 7f3781d9b640 1 -- 172.21.15.172:0/169416031 learned_addr learned my addr 172.21.15.172:0/169416031 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:04.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.626+0000 7f3782d9d640 1 -- 172.21.15.172:0/169416031 >> v1:172.21.15.174:6789/0 conn(0x7f3784154ed0 legacy=0x7f378415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.626+0000 7f3782d9d640 1 -- 172.21.15.172:0/169416031 >> v1:172.21.15.172:6789/0 conn(0x7f37840b3630 legacy=0x7f37840cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.626+0000 7f378159a640 1 -- 172.21.15.172:0/169416031 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762007352 0 0) 0x7f378415b250 con 0x7f3784154ed0 2024-03-16T19:30:04.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.626+0000 7f3782d9d640 1 -- 172.21.15.172:0/169416031 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37841552b0 con 0x7f37840b4700 2024-03-16T19:30:04.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f3782d9d640 1 --2- 172.21.15.172:0/169416031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37840b4700 0x7f37840b4b10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f37740097a0 tx=0x7f377402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356e2b67520f3c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f378159a640 1 -- 172.21.15.172:0/169416031 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774030710 con 0x7f37840b4700 2024-03-16T19:30:04.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f378159a640 1 -- 172.21.15.172:0/169416031 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377402e050 con 0x7f37840b4700 2024-03-16T19:30:04.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f378159a640 1 -- 172.21.15.172:0/169416031 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774030a90 con 0x7f37840b4700 2024-03-16T19:30:04.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f3788c5c640 1 -- 172.21.15.172:0/169416031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37840b4700 msgr2=0x7f37840b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f3788c5c640 1 --2- 172.21.15.172:0/169416031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37840b4700 0x7f37840b4b10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f37740097a0 tx=0x7f377402b790 comp rx=0 tx=0).stop 2024-03-16T19:30:04.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f3788c5c640 1 -- 172.21.15.172:0/169416031 shutdown_connections 2024-03-16T19:30:04.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f3788c5c640 1 --2- 172.21.15.172:0/169416031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37840b4700 0x7f37840b4b10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f3788c5c640 1 -- 172.21.15.172:0/169416031 >> 172.21.15.172:0/169416031 conn(0x7f378406bf60 msgr2=0x7f378406d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:04.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.627+0000 7f3788c5c640 1 -- 172.21.15.172:0/169416031 shutdown_connections 2024-03-16T19:30:04.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.628+0000 7f3788c5c640 1 -- 172.21.15.172:0/169416031 wait complete. 2024-03-16T19:30:04.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.629+0000 7f3788c5c640 1 Processor -- start 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.629+0000 7f3788c5c640 1 -- start start 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.629+0000 7f3788c5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37840b3630 0x7f37840c5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3788c5c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37840b4700 0x7f37841e57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3788c5c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3784154ed0 0x7f37841e6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3788c5c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37840c5b40 con 0x7f3784154ed0 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3788c5c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378414f230 con 0x7f37840b4700 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3788c5c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840c3e80 con 0x7f37840b3630 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3782d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3784154ed0 0x7f37841e6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3781d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37840b4700 0x7f37841e57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3782d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3784154ed0 0x7f37841e6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47138/0 (socket says 172.21.15.172:47138) 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3782d9d640 1 -- 172.21.15.172:0/1907633011 learned_addr learned my addr 172.21.15.172:0/1907633011 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:04.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f378259c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37840b3630 0x7f37840c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3782d9d640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37840b3630 msgr2=0x7f37840c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37840b3630 0x7f37840c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3782d9d640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37840b4700 msgr2=0x7f37841e57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37840b4700 0x7f37841e57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3782d9d640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3774009400 con 0x7f3784154ed0 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3784154ed0 0x7f37841e6660 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f37740097a0 tx=0x7f37740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774032910 con 0x7f3784154ed0 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3774032a70 con 0x7f3784154ed0 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37841f0500 con 0x7f3784154ed0 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774032e20 con 0x7f3784154ed0 2024-03-16T19:30:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.630+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37840c4420 con 0x7f3784154ed0 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.631+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f37840b3c20 con 0x7f3784154ed0 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.632+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3774039040 con 0x7f3784154ed0 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.632+0000 7f3772ffd640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f376406b3b0 0x7f376406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.632+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5741+0+0 (secure 0 0 0) 0x7f37740ab800 con 0x7f3784154ed0 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.632+0000 7f378259c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f376406b3b0 0x7f376406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.632+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v30) v1 ==== 54+0+16 (secure 0 0 0) 0x7f37740abba0 con 0x7f3784154ed0 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.633+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3784159f90 0x7f3784146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.633+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37840c4b50 con 0x7f3784159f90 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.633+0000 7f378259c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f376406b3b0 0x7f376406d870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f377c0012e0 tx=0x7f377c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.633+0000 7f3781d9b640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3784159f90 0x7f3784146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.633+0000 7f3781d9b640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3784159f90 0x7f3784146fe0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.634+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f37840c4b50 con 0x7f3784159f90 2024-03-16T19:30:04.629 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.648+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3784068d20 con 0x7f3784159f90 2024-03-16T19:30:04.631 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.650+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3784068d20 con 0x7f3784159f90 2024-03-16T19:30:04.631 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:04.631 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:04.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.650+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f378415a370 0x7f37841f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.650+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37840cbf90 con 0x7f378415a370 2024-03-16T19:30:04.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.651+0000 7f378259c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f378415a370 0x7f37841f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.651+0000 7f378259c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f378415a370 0x7f37841f0a00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.651+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f37840cbf90 con 0x7f378415a370 2024-03-16T19:30:04.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.667+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f378415b790 con 0x7f378415a370 2024-03-16T19:30:04.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.668+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f377c00fe00 con 0x7f378415a370 2024-03-16T19:30:04.649 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:04.650 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:04.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.669+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f37841f3f80 0x7f37841f4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.669+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37841f48a0 con 0x7f37841f3f80 2024-03-16T19:30:04.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.669+0000 7f3781d9b640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f37841f3f80 0x7f37841f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.669+0000 7f3781d9b640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f37841f3f80 0x7f37841f4360 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.670+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f37841f48a0 con 0x7f37841f3f80 2024-03-16T19:30:04.651 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:04.671+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:04.657 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:04.657 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:04.657 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:04.657 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:04.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.685+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f37840cbbe0 con 0x7f37841f3f80 2024-03-16T19:30:04.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:04.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.686+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f37840cbbe0 con 0x7f37841f3f80 2024-03-16T19:30:04.667 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:04.668 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:04.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.687+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3784062200 0x7f37840ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.687+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378406d7b0 con 0x7f3784062200 2024-03-16T19:30:04.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.687+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3784062200 0x7f37840ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.687+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3784062200 0x7f37840ccaf0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.752 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:04.771+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:04.773 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:04.773 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:04.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:04.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.834+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f378406d7b0 con 0x7f3784062200 2024-03-16T19:30:04.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.848+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3784062910 con 0x7f3784062200 2024-03-16T19:30:04.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.850+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3784062910 con 0x7f3784062200 2024-03-16T19:30:04.831 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:04.831 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:04.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.850+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f37840cd540 0x7f378402bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.850+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3784159790 con 0x7f37840cd540 2024-03-16T19:30:04.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.851+0000 7f378259c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f37840cd540 0x7f378402bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.851+0000 7f378259c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f37840cd540 0x7f378402bde0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.852+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3784159790 con 0x7f37840cd540 2024-03-16T19:30:04.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.866+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3784159790 con 0x7f37840cd540 2024-03-16T19:30:04.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.867+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3784159790 con 0x7f37840cd540 2024-03-16T19:30:04.848 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:04.849 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:04.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.867+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f378402c830 0x7f378402cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.867+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37840cda90 con 0x7f378402c830 2024-03-16T19:30:04.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.868+0000 7f3781d9b640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f378402c830 0x7f378402cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.868+0000 7f3781d9b640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f378402c830 0x7f378402cc30 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.869+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f37840cda90 con 0x7f378402c830 2024-03-16T19:30:04.853 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:04.872+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:04.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.883+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f37840cda90 con 0x7f378402c830 2024-03-16T19:30:04.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.885+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f37840cda90 con 0x7f378402c830 2024-03-16T19:30:04.865 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:04.866 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:04.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.885+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3784207310 0x7f3784209700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.885+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378420c4d0 con 0x7f3784207310 2024-03-16T19:30:04.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.885+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3784207310 0x7f3784209700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.885+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3784207310 0x7f3784209700 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.886+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f378420c4d0 con 0x7f3784207310 2024-03-16T19:30:04.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3784209f50 con 0x7f3784207310 2024-03-16T19:30:04.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3782d9d640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3784207310 msgr2=0x7f3784209700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 27 2024-03-16T19:30:04.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3782d9d640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3784207310 msgr2=0x7f3784209700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:30:04.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3784207310 0x7f3784209700 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:30:04.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3784207310 0x7f3784209700 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3784207310 msgr2=0x7f3784209700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:30:04.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3772ffd640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3784207310 0x7f3784209700 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3772ffd640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3764075460 0x7f3764077850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f378406d7b0 con 0x7f3764075460 2024-03-16T19:30:04.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=31}) v3 -- 0x7f3764077d90 con 0x7f3784154ed0 2024-03-16T19:30:04.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3764075460 0x7f3764077850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.901+0000 7f3782d9d640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3764075460 0x7f3764077850 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.903+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f378406d7b0 con 0x7f3764075460 2024-03-16T19:30:04.884 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:04.884 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:04.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.903+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3784207fc0 0x7f37842083c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:04.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.903+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3784209b00 con 0x7f3784207fc0 2024-03-16T19:30:04.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.903+0000 7f378259c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3784207fc0 0x7f37842083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:04.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.904+0000 7f378259c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3784207fc0 0x7f37842083c0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:04.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.904+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3784209b00 con 0x7f3784207fc0 2024-03-16T19:30:04.892 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:04.892 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:04.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:04.892 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:04.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.892 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:30:04.892 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.317402453, 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:04.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:30:04.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.527176281999999, 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.897 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:04.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:30:04.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.526416046999998, 2024-03-16T19:30:04.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:30:04.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:30:04.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.525602092, 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:30:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:30:04.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:30:04.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.524962166999998, 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:30:04.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.524304069999999, 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:30:04.914 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:30:04.918 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.523771494999998, 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:30:04.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.922 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.465599591999997, 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:30:04.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:30:04.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.464997398000001, 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:30:04.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:30:04.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.464354452999999, 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:30:04.931 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:30:04.935 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.463850491999999, 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:30:04.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.305704226000003, 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:30:04.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:30:04.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.305174090000001, 2024-03-16T19:30:04.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:30:04.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:30:04.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.553204831000002, 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:30:04.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:30:04.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:30:04.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.553180294999997, 2024-03-16T19:30:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:30:04.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.959 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.553175764999999, 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:30:04.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:30:04.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:30:04.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:30:04.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.553165952999997, 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:04.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.553162100000002, 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:30:04.970 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:30:04.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:30:04.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:30:04.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.553158783999997, 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.979 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.551963201, 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:04.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.983 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:04.984 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:04.984 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:04.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.918+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3784208900 con 0x7f3784207fc0 2024-03-16T19:30:04.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.920+0000 7f3772ffd640 1 -- 172.21.15.172:0/1907633011 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3784208900 con 0x7f3784207fc0 2024-03-16T19:30:04.985 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:04.986 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:04.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3784159f90 msgr2=0x7f3784146fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3784159f90 0x7f3784146fe0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f378415a370 msgr2=0x7f37841f0a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f378415a370 0x7f37841f0a00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f37841f3f80 msgr2=0x7f37841f4360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f37841f3f80 0x7f37841f4360 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3784062200 msgr2=0x7f37840ccaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3784062200 0x7f37840ccaf0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3782d9d640 1 -- 172.21.15.172:0/1907633011 reap_dead start 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f37840cd540 msgr2=0x7f378402bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f37840cd540 0x7f378402bde0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f378402c830 msgr2=0x7f378402cc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3782d9d640 1 -- 172.21.15.172:0/1907633011 reap_dead start 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f378402c830 0x7f378402cc30 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3764075460 msgr2=0x7f3764077850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3764075460 0x7f3764077850 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3784207fc0 msgr2=0x7f37842083c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3784207fc0 0x7f37842083c0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:04.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f376406b3b0 msgr2=0x7f376406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f376406b3b0 0x7f376406d870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f377c0012e0 tx=0x7f377c009040 comp rx=0 tx=0).stop 2024-03-16T19:30:04.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3784154ed0 msgr2=0x7f37841e6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:04.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 --2- 172.21.15.172:0/1907633011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3784154ed0 0x7f37841e6660 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f37740097a0 tx=0x7f37740046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:04.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3782d9d640 1 -- 172.21.15.172:0/1907633011 reap_dead start 2024-03-16T19:30:04.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 shutdown_connections 2024-03-16T19:30:04.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 >> 172.21.15.172:0/1907633011 conn(0x7f378406bf60 msgr2=0x7f37840cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:04.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 shutdown_connections 2024-03-16T19:30:04.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:04.921+0000 7f3788c5c640 1 -- 172.21.15.172:0/1907633011 wait complete. 2024-03-16T19:30:04.990 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:05.004+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:05.086 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:05.105+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:05.135 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:05.136 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:05.136 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:05.136 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:05.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:05.189 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:05.209+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:05.251 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:05.251 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:05.251 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:05.252 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:05.252 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:05.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:05.302 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:05.321+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:05.389 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:05.407+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.071683573999998, 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.071666460000003, 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:30:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:30:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.071659179999997, 2024-03-16T19:30:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:30:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:30:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.071645312999998, 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:30:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:30:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:30:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:30:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.070972478000002, 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.068674227999999, 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:30:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.068629127000001, 2024-03-16T19:30:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:30:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:30:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.068601072, 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:30:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:30:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.068579657999997, 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:30:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:30:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.068557306999999, 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:30:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:30:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.068536907999999, 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:30:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:30:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.068430915999997, 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:30:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.066639205000001, 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:30:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.066591703999997, 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:30:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.066352453, 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:30 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.066196873999999, 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126666, 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362824+0000", 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7999999999999995e-07 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362828+0000", 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0879999999999997e-06 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362837+0000", 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.704e-06 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362843+0000", 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9390000000000002e-06 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362877+0000", 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4779e-05 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362883+0000", 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9009999999999999e-06 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362950+0000", 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6674999999999994e-05 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.057854308, 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:30:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:30:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.056514432, 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:30:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.492 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:05.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:05.494 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:05.512+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:05.552 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:30:05.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:30:05.553 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-03-16T19:30:05.553 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-16T19:30:05.553 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-16T19:30:05.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-03-16T19:30:05.594 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:05.612+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:05.643 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:05.644 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:05.644 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:05.644 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:05.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:05.687 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:05.687 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-16T19:30:05.687 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:05.695 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:05.715+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:05.760 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:05.760 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:05.760 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:05.761 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:05.761 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:05.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:05.796 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:05.814+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:05.878 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:05.878 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:05.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:05.878 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:05.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539617311000001, 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053568110000000004, 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357146+0000", 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5860000000000001e-06 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357154+0000", 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5439999999999999e-06 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357217+0000", 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2986999999999999e-05 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357229+0000", 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1853e-05 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357343+0000", 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113663 2024-03-16T19:30:05.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357504+0000", 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016176 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013908599999999999 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360705+0000", 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030619419999999998 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362409+0000", 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001704021 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362456+0000", 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6140000000000002e-05 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362496+0000", 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0228999999999997e-05 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539597284000003, 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078702890000000008, 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4299999999999998e-07 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357163+0000", 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2559999999999998e-06 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357254+0000", 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1701999999999997e-05 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357258+0000", 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5719999999999999e-06 2024-03-16T19:30:05.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357931+0000", 2024-03-16T19:30:05.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067308000000000005 2024-03-16T19:30:05.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357999+0000", 2024-03-16T19:30:05.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8323000000000003e-05 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358113+0000", 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011366699999999999 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362505+0000", 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004391509 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024372970000000002 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364992+0000", 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9741999999999999e-05 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365029+0000", 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7697999999999997e-05 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.889 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539590482000001, 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079944349999999994, 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6e-07 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357172+0000", 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7960000000000001e-06 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357285+0000", 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113183 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357290+0000", 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6190000000000002e-06 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358306+0000", 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010160670000000001 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358400+0000", 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3854e-05 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358492+0000", 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2954000000000005e-05 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362510+0000", 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040175469999999998 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025680310000000001 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365123+0000", 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4515000000000003e-05 2024-03-16T19:30:05.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365160+0000", 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7608999999999999e-05 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539583585999999, 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080787759999999993, 2024-03-16T19:30:05.895 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.36e-07 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1270000000000001e-06 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357297+0000", 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121885 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357300+0000", 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049999999999999e-06 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358709+0000", 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001409083 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358776+0000", 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6489999999999995e-05 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358885+0000", 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109271 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362514+0000", 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036292199999999998 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365197+0000", 2024-03-16T19:30:05.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026832100000000001 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365226+0000", 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9136999999999999e-05 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365251+0000", 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5012e-05 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539580602000001, 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081787609999999997, 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357177+0000", 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5090000000000001e-06 2024-03-16T19:30:05.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357306+0000", 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012856900000000001 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357308+0000", 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3700000000000002e-06 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359071+0000", 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017631210000000001 2024-03-16T19:30:05.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359182+0000", 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110265 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359280+0000", 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8595e-05 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362520+0000", 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032394120000000001 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365297+0000", 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002777542 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365328+0000", 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0899999999999999e-05 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365354+0000", 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6276000000000001e-05 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539577913999999, 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083197830000000007, 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.906 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357179+0000", 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:05.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357187+0000", 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357314+0000", 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126978 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357316+0000", 2024-03-16T19:30:05.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3800000000000001e-06 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359512+0000", 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021950799999999999 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359585+0000", 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3168000000000001e-05 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359702+0000", 2024-03-16T19:30:05.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011741 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362525+0000", 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028225419999999999 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365408+0000", 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028829010000000002 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365460+0000", 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839e-05 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365498+0000", 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7673999999999999e-05 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539566788999998, 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084493169999999996, 2024-03-16T19:30:05.911 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357190+0000", 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-07 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357191+0000", 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7069999999999999e-06 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357325+0000", 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133585 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357330+0000", 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0799999999999996e-06 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359915+0000", 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025849290000000001 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9741999999999995e-05 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360121+0000", 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362826+0000", 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027046819999999999 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365564+0000", 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027387179999999998 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365606+0000", 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1715999999999998e-05 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365639+0000", 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660999999999998e-05 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539564120000001, 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085533069999999996, 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0699999999999999e-07 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3030000000000001e-06 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357339+0000", 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014513199999999999 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000003e-06 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030087619999999999 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360414+0000", 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1297999999999995e-05 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360498+0000", 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4538999999999998e-05 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362832+0000", 2024-03-16T19:30:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023338410000000001 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365670+0000", 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028379550000000001 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365724+0000", 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3533999999999999e-05 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365745+0000", 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1725999999999998e-05 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539560895999998, 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052850789999999998, 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357196+0000", 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.173e-06 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357359+0000", 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016222400000000001 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357365+0000", 2024-03-16T19:30:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7320000000000003e-06 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357451+0000", 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6494000000000002e-05 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357555+0000", 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010401200000000001 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8224000000000001e-05 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362294+0000", 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046502399999999999 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362388+0000", 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4448000000000001e-05 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362433+0000", 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4888000000000001e-05 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362480+0000", 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7441000000000002e-05 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.539558859000003, 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053762239999999998, 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357198+0000", 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357199+0000", 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357415+0000", 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021638699999999999 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0069e-05 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358132+0000", 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070650599999999997 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358207+0000", 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5197999999999994e-05 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358342+0000", 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013503199999999999 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362377+0000", 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040354600000000003 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362493+0000", 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115951 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362536+0000", 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3189999999999998e-05 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362574+0000", 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7113999999999997e-05 2024-03-16T19:30:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:6 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.536499958, 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045449999999999999, 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360257+0000", 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7400000000000002e-07 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360260+0000", 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3550000000000001e-06 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360270+0000", 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360277+0000", 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5760000000000002e-06 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360697+0000", 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042039200000000001 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360702+0000", 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7140000000000001e-06 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360711+0000", 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1209999999999996e-06 2024-03-16T19:30:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:22 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.534555783000002, 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100673, 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362201+0000", 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5799999999999998e-07 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362206+0000", 2024-03-16T19:30:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8579999999999999e-06 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362219+0000", 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.239e-05 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362226+0000", 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2710000000000003e-06 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362281+0000", 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4756000000000002e-05 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362288+0000", 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.075e-06 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362301+0000", 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3365e-05 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:23 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.534444514, 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017205200000000001, 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4499999999999997e-07 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362315+0000", 2024-03-16T19:30:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1350000000000001e-06 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362322+0000", 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9469999999999997e-06 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362328+0000", 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362366+0000", 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8160000000000001e-05 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362373+0000", 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5799999999999997e-06 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362484+0000", 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111173 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:7 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.531900641999997, 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018177499999999999, 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364861+0000", 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3699999999999997e-06 2024-03-16T19:30:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364875+0000", 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4059e-05 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364881+0000", 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5270000000000004e-06 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364929+0000", 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000002e-05 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3699999999999997e-06 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365037+0000", 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100819 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:8 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.531843584999997, 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025451199999999998, 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364916+0000", 2024-03-16T19:30:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8109999999999999e-06 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364924+0000", 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1419999999999993e-06 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364930+0000", 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2260000000000002e-06 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365071+0000", 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014030100000000001 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365075+0000", 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365167+0000", 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2196999999999998e-05 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:9 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.531817345, 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032082900000000001, 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.22e-07 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3980000000000002e-06 2024-03-16T19:30:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364947+0000", 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3090000000000003e-06 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364951+0000", 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1309999999999999e-06 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365191+0000", 2024-03-16T19:30:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023992600000000001 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365195+0000", 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.704e-06 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365260+0000", 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5139000000000001e-05 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:10 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.531788958999996, 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039415499999999998, 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364968+0000", 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364970+0000", 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4650000000000001e-06 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364977+0000", 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.443e-06 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364981+0000", 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2320000000000003e-06 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365289+0000", 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030856900000000002 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365294+0000", 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6920000000000001e-06 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365361+0000", 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7358000000000001e-05 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:11 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.531768458999998, 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051803199999999998, 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9299999999999999e-07 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364993+0000", 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9080000000000003e-06 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4240000000000001e-06 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365002+0000", 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7570000000000002e-06 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365395+0000", 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039237199999999999 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365401+0000", 2024-03-16T19:30:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9380000000000003e-06 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365506+0000", 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010433999999999999 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:12 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.531746454999997, 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063430500000000005, 2024-03-16T19:30:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9500000000000001e-07 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365012+0000", 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7129999999999999e-06 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365015+0000", 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2629999999999999e-06 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365017+0000", 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1749999999999999e-06 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365544+0000", 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052690099999999998 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365559+0000", 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4610000000000001e-05 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365644+0000", 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5447999999999999e-05 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:13 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.531735957000002, 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00073113500000000005, 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1000000000000001e-07 2024-03-16T19:30:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365021+0000", 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.23e-07 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365024+0000", 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7369999999999998e-06 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365026+0000", 2024-03-16T19:30:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1289999999999999e-06 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365663+0000", 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063679600000000002 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365667+0000", 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.233e-06 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365751+0000", 2024-03-16T19:30:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4307000000000004e-05 2024-03-16T19:30:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:05.972 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:05.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:05.974 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:05.993+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:06.074 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:06.093+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:06.120 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:06.121 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:06.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:06.175 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:06.194+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:06.238 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:06.239 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:06.239 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:06.239 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:06.239 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:06.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:06.276 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:06.294+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:06.357 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.019215676999998, 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:06.358 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:30:06.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:30:06.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:30:06.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.019201203999998, 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:30:06.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:30:06.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.018696714999997, 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.366 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:30:06.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:30:06.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:06.370 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.018681319000002, 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:30:06.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.018222635999997, 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.017676465999998, 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:06.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:30:06.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:30:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.017119375, 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:30:06.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.015917627, 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:30:06.387 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:06.391 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.015338917999998, 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:06.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.015289391000003, 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:06.395 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:30:06.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:30:06.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.014328747999997, 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:30:06.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.01061455, 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.570213637999998, 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:30:06.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.004660635, 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:30:06.411 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:30:06.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.497061909999999, 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:30:06.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:30:06.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:30:06.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.420 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:06.420 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:06.420 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:06.421 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:06.440+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:06.422 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:06.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:30:06.522 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:06.541+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:06.531 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:06.531 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:06.531 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:06.532 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:06.540 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:06.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.584+0000 7f2603dd0640 1 Processor -- start 2024-03-16T19:30:06.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.584+0000 7f2603dd0640 1 -- start start 2024-03-16T19:30:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.584+0000 7f2603dd0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc158180 0x7f25fc15b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f2603dd0640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f25fc0ce200 0x7f25fc0ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f2603dd0640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc15ba20 con 0x7f25fc06fe50 2024-03-16T19:30:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f2603dd0640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25fc06cf90 con 0x7f25fc0ce200 2024-03-16T19:30:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f2603dd0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc06c5b0 con 0x7f25fc158180 2024-03-16T19:30:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f260236d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f25fc0ce200 0x7f25fc0ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f260136b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f25fc06fe50 0x7f25fc0c9210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39742/0 (socket says 172.21.15.172:39742) 2024-03-16T19:30:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f260136b640 1 -- 172.21.15.172:0/209100203 learned_addr learned my addr 172.21.15.172:0/209100203 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:06.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f2601b6c640 1 --2- 172.21.15.172:0/209100203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc158180 0x7f25fc15b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:06.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f260236d640 1 -- 172.21.15.172:0/209100203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc158180 msgr2=0x7f25fc15b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:06.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f260236d640 1 --2- 172.21.15.172:0/209100203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc158180 0x7f25fc15b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:06.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f260236d640 1 -- 172.21.15.172:0/209100203 >> v1:172.21.15.172:6789/0 conn(0x7f25fc06fe50 legacy=0x7f25fc0c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:06.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f260236d640 1 -- 172.21.15.172:0/209100203 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc158560 con 0x7f25fc0ce200 2024-03-16T19:30:06.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.585+0000 7f2600b6a640 1 -- 172.21.15.172:0/209100203 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350705531 0 0) 0x7f25ec001110 con 0x7f25fc06fe50 2024-03-16T19:30:06.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f260236d640 1 --2- 172.21.15.172:0/209100203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc0ce200 0x7f25fc0ce610 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009d90 tx=0x7f25f402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7529ecb7524213f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:06.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f2600b6a640 1 -- 172.21.15.172:0/209100203 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f402cce0 con 0x7f25fc0ce200 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f2600b6a640 1 -- 172.21.15.172:0/209100203 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f402ce40 con 0x7f25fc0ce200 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f2600b6a640 1 -- 172.21.15.172:0/209100203 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f4034660 con 0x7f25fc0ce200 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f2603dd0640 1 -- 172.21.15.172:0/209100203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc0ce200 msgr2=0x7f25fc0ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f2603dd0640 1 --2- 172.21.15.172:0/209100203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc0ce200 0x7f25fc0ce610 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009d90 tx=0x7f25f402bf30 comp rx=0 tx=0).stop 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f2603dd0640 1 -- 172.21.15.172:0/209100203 shutdown_connections 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f2603dd0640 1 --2- 172.21.15.172:0/209100203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc0ce200 0x7f25fc0ce610 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f2603dd0640 1 --2- 172.21.15.172:0/209100203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc158180 0x7f25fc15b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.586+0000 7f2603dd0640 1 -- 172.21.15.172:0/209100203 >> 172.21.15.172:0/209100203 conn(0x7f25fc066c20 msgr2=0x7f25fc067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.587+0000 7f2603dd0640 1 -- 172.21.15.172:0/209100203 shutdown_connections 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.587+0000 7f2603dd0640 1 -- 172.21.15.172:0/209100203 wait complete. 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f2603dd0640 1 Processor -- start 2024-03-16T19:30:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f2603dd0640 1 -- start start 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f2603dd0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc06fe50 0x7f25fc0cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f2603dd0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc0ce200 0x7f25fc1f08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f2603dd0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc158180 0x7f25fc1f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f2603dd0640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc1591c0 con 0x7f25fc158180 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f2603dd0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc1594a0 con 0x7f25fc0ce200 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f2603dd0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc159780 con 0x7f25fc06fe50 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f260236d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc158180 0x7f25fc1f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f260236d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc158180 0x7f25fc1f0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47164/0 (socket says 172.21.15.172:47164) 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.588+0000 7f260236d640 1 -- 172.21.15.172:0/1557419399 learned_addr learned my addr 172.21.15.172:0/1557419399 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f2601b6c640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc06fe50 0x7f25fc0cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f260136b640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc0ce200 0x7f25fc1f08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f260236d640 1 -- 172.21.15.172:0/1557419399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc06fe50 msgr2=0x7f25fc0cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f260236d640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc06fe50 0x7f25fc0cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f260236d640 1 -- 172.21.15.172:0/1557419399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc0ce200 msgr2=0x7f25fc1f08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f260236d640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc0ce200 0x7f25fc1f08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f260236d640 1 -- 172.21.15.172:0/1557419399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f4009a70 con 0x7f25fc158180 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f260236d640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc158180 0x7f25fc1f0e10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009d90 tx=0x7f25f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f25f27fc640 1 -- 172.21.15.172:0/1557419399 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f4003b90 con 0x7f25fc158180 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f25f27fc640 1 -- 172.21.15.172:0/1557419399 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f4003d10 con 0x7f25fc158180 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25fc159a60 con 0x7f25fc158180 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f25f27fc640 1 -- 172.21.15.172:0/1557419399 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f4004270 con 0x7f25fc158180 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.589+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25fc1f0340 con 0x7f25fc158180 2024-03-16T19:30:06.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.590+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25fc1f05c0 con 0x7f25fc158180 2024-03-16T19:30:06.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.590+0000 7f25f27fc640 1 -- 172.21.15.172:0/1557419399 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f25f403b040 con 0x7f25fc158180 2024-03-16T19:30:06.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.591+0000 7f25f27fc640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f25e006b3b0 0x7f25e006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:06.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.591+0000 7f25f27fc640 1 -- 172.21.15.172:0/1557419399 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f25f40ab9e0 con 0x7f25fc158180 2024-03-16T19:30:06.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.591+0000 7f2601b6c640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f25e006b3b0 0x7f25e006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:06.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.591+0000 7f2601b6c640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f25e006b3b0 0x7f25e006d870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009b40 tx=0x7f25f8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:06.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.595+0000 7f25f27fc640 1 -- 172.21.15.172:0/1557419399 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25f40765c0 con 0x7f25fc158180 2024-03-16T19:30:06.624 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:06.643+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:06.686 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:06.687 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:06.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:06.724 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:06.744+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:06.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:06.775+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f25fc075a70 con 0x7f25fc158180 2024-03-16T19:30:06.825 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:06.844+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:06.843 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:06.843 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:06.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:06.843 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.505523779999997, 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:06.844 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:06.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:06.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:06.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:06.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.504296357999998, 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:06.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:06.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.503893652000002, 2024-03-16T19:30:06.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:06.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:06.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:06.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.860 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.503284901000001, 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:06.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:06.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.502841031999999, 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:06.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:06.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:06.869 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.502698256000002, 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:06.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:06.873 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.502683715000003, 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:06.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.499806020000001, 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:06.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:06.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:06.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:06.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:06.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:06.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:06.883 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.883 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:06.883 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.883 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:06.883 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:06.883 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:06.884 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:06.925 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:06.944+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:06.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:07.026 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:07.045+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:07.031 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:07.031 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:07.032 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:07.032 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:07.041 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:07.091 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.110+0000 7fb55edd0640 1 Processor -- start 2024-03-16T19:30:07.091 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.110+0000 7fb55edd0640 1 -- start start 2024-03-16T19:30:07.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55edd0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb558154bb0 0x7fb5580c3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55edd0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb5580c3e70 0x7fb55815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55edd0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55806fde0 con 0x7fb5580c8ba0 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55edd0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb558070310 con 0x7fb5580c3e70 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55edd0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580c31c0 con 0x7fb558154bb0 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55cb6c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb558154bb0 0x7fb5580c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb557fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fb5580c8ba0 0x7fb55815a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:40380/0 (socket says 172.21.15.172:40380) 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb557fff640 1 -- 172.21.15.172:0/3538523976 learned_addr learned my addr 172.21.15.172:0/3538523976 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55d36d640 1 --2- 172.21.15.172:0/3538523976 >> v2:172.21.15.174:3300/0 conn(0x7fb5580c3e70 0x7fb55815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55cb6c640 1 -- 172.21.15.172:0/3538523976 >> v2:172.21.15.174:3300/0 conn(0x7fb5580c3e70 msgr2=0x7fb55815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/3538523976 >> v2:172.21.15.174:3300/0 conn(0x7fb5580c3e70 0x7fb55815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55cb6c640 1 -- 172.21.15.172:0/3538523976 >> v1:172.21.15.174:6789/0 conn(0x7fb5580c8ba0 legacy=0x7fb55815a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.111+0000 7fb55cb6c640 1 -- 172.21.15.172:0/3538523976 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580c4300 con 0x7fb558154bb0 2024-03-16T19:30:07.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.112+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/3538523976 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb558154bb0 0x7fb5580c3930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb5400097a0 tx=0x7fb54002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fd7ca02a0c25d5f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.112+0000 7fb5577fe640 1 -- 172.21.15.172:0/3538523976 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540030610 con 0x7fb558154bb0 2024-03-16T19:30:07.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.112+0000 7fb5577fe640 1 -- 172.21.15.172:0/3538523976 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540030770 con 0x7fb558154bb0 2024-03-16T19:30:07.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.112+0000 7fb5577fe640 1 -- 172.21.15.172:0/3538523976 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540030b00 con 0x7fb558154bb0 2024-03-16T19:30:07.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.113+0000 7fb55edd0640 1 -- 172.21.15.172:0/3538523976 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb558154bb0 msgr2=0x7fb5580c3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.113+0000 7fb55edd0640 1 --2- 172.21.15.172:0/3538523976 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb558154bb0 0x7fb5580c3930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb5400097a0 tx=0x7fb54002b960 comp rx=0 tx=0).stop 2024-03-16T19:30:07.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.113+0000 7fb55edd0640 1 -- 172.21.15.172:0/3538523976 shutdown_connections 2024-03-16T19:30:07.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.113+0000 7fb55edd0640 1 --2- 172.21.15.172:0/3538523976 >> v2:172.21.15.174:3300/0 conn(0x7fb5580c3e70 0x7fb55815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.113+0000 7fb55edd0640 1 --2- 172.21.15.172:0/3538523976 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb558154bb0 0x7fb5580c3930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.113+0000 7fb55edd0640 1 -- 172.21.15.172:0/3538523976 >> 172.21.15.172:0/3538523976 conn(0x7fb5580708a0 msgr2=0x7fb558070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.113+0000 7fb55edd0640 1 -- 172.21.15.172:0/3538523976 shutdown_connections 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.113+0000 7fb55edd0640 1 -- 172.21.15.172:0/3538523976 wait complete. 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.114+0000 7fb55edd0640 1 Processor -- start 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.114+0000 7fb55edd0640 1 -- start start 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55edd0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5580c3e70 0x7fb558157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55edd0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5580c8ba0 0x7fb558156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55edd0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb558154bb0 0x7fb5581f0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55edd0640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5581594a0 con 0x7fb5580c3e70 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55edd0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb558071560 con 0x7fb558154bb0 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55edd0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb558158890 con 0x7fb5580c8ba0 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55cb6c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5580c3e70 0x7fb558157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55cb6c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5580c3e70 0x7fb558157ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47178/0 (socket says 172.21.15.172:47178) 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb557fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5580c8ba0 0x7fb558156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55cb6c640 1 -- 172.21.15.172:0/4108278289 learned_addr learned my addr 172.21.15.172:0/4108278289 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55d36d640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb558154bb0 0x7fb5581f0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55cb6c640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5580c8ba0 msgr2=0x7fb558156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5580c8ba0 0x7fb558156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55cb6c640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb558154bb0 msgr2=0x7fb5581f0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb558154bb0 0x7fb5581f0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.115+0000 7fb55cb6c640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb540009400 con 0x7fb5580c3e70 2024-03-16T19:30:07.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.116+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5580c3e70 0x7fb558157ac0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb5400098c0 tx=0x7fb5400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.116+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54003e070 con 0x7fb5580c3e70 2024-03-16T19:30:07.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.116+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540030e10 con 0x7fb5580c3e70 2024-03-16T19:30:07.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.116+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb558158b70 con 0x7fb5580c3e70 2024-03-16T19:30:07.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.116+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540039a40 con 0x7fb5580c3e70 2024-03-16T19:30:07.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.116+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb558159130 con 0x7fb5580c3e70 2024-03-16T19:30:07.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.117+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb558158350 con 0x7fb5580c3e70 2024-03-16T19:30:07.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.117+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb540042070 con 0x7fb5580c3e70 2024-03-16T19:30:07.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.118+0000 7fb5557fa640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb53806b3b0 0x7fb53806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.118+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fb5400abbd0 con 0x7fb5580c3e70 2024-03-16T19:30:07.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.118+0000 7fb557fff640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb53806b3b0 0x7fb53806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.118+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v31) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb5400ac0c0 con 0x7fb5580c3e70 2024-03-16T19:30:07.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.118+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb5580cc080 0x7fb55813e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.118+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55806c180 con 0x7fb5580cc080 2024-03-16T19:30:07.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.118+0000 7fb557fff640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb53806b3b0 0x7fb53806d870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb5580cb340 tx=0x7fb548001a30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.118+0000 7fb55d36d640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb5580cc080 0x7fb55813e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.118+0000 7fb55d36d640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb5580cc080 0x7fb55813e6a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.119+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb550009870 con 0x7fb5580cc080 2024-03-16T19:30:07.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.134+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb5580c3410 con 0x7fb5580cc080 2024-03-16T19:30:07.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.135+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb550009dc0 con 0x7fb5580cc080 2024-03-16T19:30:07.116 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:07.117 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:07.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.135+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fb5580cc460 0x7fb5581f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.135+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5580681c0 con 0x7fb5580cc460 2024-03-16T19:30:07.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.135+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fb5580cc460 0x7fb5581f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.136+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fb5580cc460 0x7fb5581f3740 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.136+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb5580681c0 con 0x7fb5580cc460 2024-03-16T19:30:07.128 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:07.147+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:07.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.151+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb5580cdf40 con 0x7fb5580cc460 2024-03-16T19:30:07.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.153+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb5580cdf40 con 0x7fb5580cc460 2024-03-16T19:30:07.134 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:07.135 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:07.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.153+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fb5581f4190 0x7fb558061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.153+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5581f4590 con 0x7fb5581f4190 2024-03-16T19:30:07.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.153+0000 7fb557fff640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fb5581f4190 0x7fb558061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.153+0000 7fb557fff640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fb5581f4190 0x7fb558061930 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.154+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb5581f4590 con 0x7fb5581f4190 2024-03-16T19:30:07.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.168+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb5580620c0 con 0x7fb5581f4190 2024-03-16T19:30:07.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.169+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb5580620c0 con 0x7fb5581f4190 2024-03-16T19:30:07.150 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:07.151 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:07.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.170+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fb558062650 0x7fb55806e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.170+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55806f730 con 0x7fb558062650 2024-03-16T19:30:07.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.170+0000 7fb55d36d640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fb558062650 0x7fb55806e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.170+0000 7fb55d36d640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fb558062650 0x7fb55806e880 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.171+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb55806f730 con 0x7fb558062650 2024-03-16T19:30:07.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.185+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb55806f010 con 0x7fb558062650 2024-03-16T19:30:07.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.186+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb55806f010 con 0x7fb558062650 2024-03-16T19:30:07.167 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:07.167 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:07.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.186+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb55802c2f0 0x7fb55802c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.186+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55806edc0 con 0x7fb55802c2f0 2024-03-16T19:30:07.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.187+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb55802c2f0 0x7fb55802c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.187+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb55802c2f0 0x7fb55802c750 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.188+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb55806edc0 con 0x7fb55802c2f0 2024-03-16T19:30:07.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.201+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb55806efc0 con 0x7fb55802c2f0 2024-03-16T19:30:07.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.203+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb55806efc0 con 0x7fb55802c2f0 2024-03-16T19:30:07.184 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:07.185 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:07.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.203+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fb558204ac0 0x7fb558206eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.203+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55806f180 con 0x7fb558204ac0 2024-03-16T19:30:07.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.204+0000 7fb557fff640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fb558204ac0 0x7fb558206eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.204+0000 7fb557fff640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fb558204ac0 0x7fb558206eb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.205+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb55806f180 con 0x7fb558204ac0 2024-03-16T19:30:07.189 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:07.189 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:07.198 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:07.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.219+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb55806f180 con 0x7fb558204ac0 2024-03-16T19:30:07.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.221+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb55806f180 con 0x7fb558204ac0 2024-03-16T19:30:07.202 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:07.202 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:07.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.221+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb558207900 0x7fb558209cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.221+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55802d1a0 con 0x7fb558207900 2024-03-16T19:30:07.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.222+0000 7fb55d36d640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb558207900 0x7fb558209cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.222+0000 7fb55d36d640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb558207900 0x7fb558209cf0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.223+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb55802d1a0 con 0x7fb558207900 2024-03-16T19:30:07.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.236+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb55802d1a0 con 0x7fb558207900 2024-03-16T19:30:07.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.238+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb55802d1a0 con 0x7fb558207900 2024-03-16T19:30:07.219 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:07.219 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:07.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.238+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fb55820a740 0x7fb55820cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:07.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.238+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55820f900 con 0x7fb55820a740 2024-03-16T19:30:07.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.238+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fb55820a740 0x7fb55820cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:07.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.239+0000 7fb55cb6c640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fb55820a740 0x7fb55820cb30 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:07.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.239+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb55820f900 con 0x7fb55820a740 2024-03-16T19:30:07.229 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:07.248+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:07.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.253+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb55820d070 con 0x7fb55820a740 2024-03-16T19:30:07.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb5557fa640 1 -- 172.21.15.172:0/4108278289 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb55820d070 con 0x7fb55820a740 2024-03-16T19:30:07.235 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:07.237 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:07.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb5580cc080 msgr2=0x7fb55813e6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb5580cc080 0x7fb55813e6a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fb5580cc460 msgr2=0x7fb5581f3740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fb5580cc460 0x7fb5581f3740 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fb5581f4190 msgr2=0x7fb558061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fb5581f4190 0x7fb558061930 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fb558062650 msgr2=0x7fb55806e880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fb558062650 0x7fb55806e880 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55d36d640 1 -- 172.21.15.172:0/4108278289 reap_dead start 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb55802c2f0 msgr2=0x7fb55802c750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb55802c2f0 0x7fb55802c750 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55d36d640 1 -- 172.21.15.172:0/4108278289 reap_dead start 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fb558204ac0 msgr2=0x7fb558206eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fb558204ac0 0x7fb558206eb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb558207900 msgr2=0x7fb558209cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb558207900 0x7fb558209cf0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fb55820a740 msgr2=0x7fb55820cb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fb55820a740 0x7fb55820cb30 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb53806b3b0 msgr2=0x7fb53806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb53806b3b0 0x7fb53806d870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb5580cb340 tx=0x7fb548001a30 comp rx=0 tx=0).stop 2024-03-16T19:30:07.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5580c3e70 msgr2=0x7fb558157ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 --2- 172.21.15.172:0/4108278289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb5580c3e70 0x7fb558157ac0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb5400098c0 tx=0x7fb5400046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55d36d640 1 -- 172.21.15.172:0/4108278289 reap_dead start 2024-03-16T19:30:07.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 shutdown_connections 2024-03-16T19:30:07.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 >> 172.21.15.172:0/4108278289 conn(0x7fb5580708a0 msgr2=0x7fb5580cd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:07.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 shutdown_connections 2024-03-16T19:30:07.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.255+0000 7fb55edd0640 1 -- 172.21.15.172:0/4108278289 wait complete. 2024-03-16T19:30:07.305 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:07.305 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:07.314 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:07.359 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:07.378+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:07.423 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:07.423 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:07.430 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:07.449+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:07.431 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:07.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.470+0000 7f25f27fc640 1 -- 172.21.15.172:0/1557419399 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v32) v1 ==== 87+0+0 (secure 0 0 0) 0x7f25f4039020 con 0x7f25fc158180 2024-03-16T19:30:07.451 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:30:07.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f25e006b3b0 msgr2=0x7f25e006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f25e006b3b0 0x7f25e006d870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009b40 tx=0x7f25f8009190 comp rx=0 tx=0).stop 2024-03-16T19:30:07.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc158180 msgr2=0x7f25fc1f0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:07.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc158180 0x7f25fc1f0e10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009d90 tx=0x7f25f4004a40 comp rx=0 tx=0).stop 2024-03-16T19:30:07.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 shutdown_connections 2024-03-16T19:30:07.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25fc158180 0x7f25fc1f0e10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc0ce200 0x7f25fc1f08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f25e006b3b0 0x7f25e006d870 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 --2- 172.21.15.172:0/1557419399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25fc06fe50 0x7f25fc0cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:07.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 >> 172.21.15.172:0/1557419399 conn(0x7f25fc066c20 msgr2=0x7f25fc0cab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:07.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 shutdown_connections 2024-03-16T19:30:07.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:07.473+0000 7f2603dd0640 1 -- 172.21.15.172:0/1557419399 wait complete. 2024-03-16T19:30:07.531 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:07.549+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:07.580 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:07.581 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:07.589 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:07.632 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:07.651+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:07.732 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:07.752+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:07.737 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:07.737 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:07.746 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:07.833 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:07.852+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:07.851 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:07.851 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:07.860 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:07.945 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:07.964+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:08.009 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:08.010 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:08.018 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:08.059 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:08.078+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:08.137 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:08.155+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:08.166 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:08.166 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:08.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:08.237 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:08.257+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:08.285 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:08.285 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:08.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:08.338 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:08.356+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:08.438 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:08.458+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:08.440 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:08.440 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:08.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:08.539 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:08.558+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:08.597 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:08.597 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:08.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:08.649 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:08.669+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:08.713 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:08.714 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:08.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:08.764 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:08.783+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:08.842 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:08.860+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:08.869 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:08.869 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:08.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:08.942 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:08.961+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:08.984 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:08.984 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:08.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:09.043 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:09.062+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:09.098 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:09.099 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:09.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:09.144 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:09.162+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:09.214 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:09.214 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:09.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:09.244 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:09.263+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:09.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:09.333 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:09.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:09.345 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:09.364+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:09.367 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.387+0000 7f11b91f4640 1 Processor -- start 2024-03-16T19:30:09.367 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.387+0000 7f11b91f4640 1 -- start start 2024-03-16T19:30:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b91f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b406aed0 0x7f11b415c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b91f4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f11b40ca6d0 0x7f11b4071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b91f4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b40c7a40 con 0x7f11b4152910 2024-03-16T19:30:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b91f4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b40c7d20 con 0x7f11b40ca6d0 2024-03-16T19:30:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b91f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b415bd30 con 0x7f11b406aed0 2024-03-16T19:30:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b2d9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f11b4152910 0x7f11b406ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:57206/0 (socket says 172.21.15.172:57206) 2024-03-16T19:30:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b2d9d640 1 -- 172.21.15.172:0/1328889157 learned_addr learned my addr 172.21.15.172:0/1328889157 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b259c640 1 --2- 172.21.15.172:0/1328889157 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b406aed0 0x7f11b415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b359e640 1 --2- 172.21.15.172:0/1328889157 >> v2:172.21.15.174:3300/0 conn(0x7f11b40ca6d0 0x7f11b4071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b1d9b640 1 -- 172.21.15.172:0/1328889157 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654527462 0 0) 0x7f11b40c7a40 con 0x7f11b4152910 2024-03-16T19:30:09.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b1d9b640 1 -- 172.21.15.172:0/1328889157 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f119c003520 con 0x7f11b4152910 2024-03-16T19:30:09.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b259c640 1 -- 172.21.15.172:0/1328889157 >> v2:172.21.15.174:3300/0 conn(0x7f11b40ca6d0 msgr2=0x7f11b4071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:09.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b259c640 1 --2- 172.21.15.172:0/1328889157 >> v2:172.21.15.174:3300/0 conn(0x7f11b40ca6d0 0x7f11b4071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b259c640 1 -- 172.21.15.172:0/1328889157 >> v1:172.21.15.172:6789/0 conn(0x7f11b4152910 legacy=0x7f11b406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b259c640 1 -- 172.21.15.172:0/1328889157 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b415c010 con 0x7f11b406aed0 2024-03-16T19:30:09.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.388+0000 7f11b359e640 1 --2- 172.21.15.172:0/1328889157 >> v2:172.21.15.174:3300/0 conn(0x7f11b40ca6d0 0x7f11b4071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.389+0000 7f11b259c640 1 --2- 172.21.15.172:0/1328889157 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b406aed0 0x7f11b415c530 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11a80097a0 tx=0x7f11a802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daf68e79f6310e5b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.389+0000 7f11b1d9b640 1 -- 172.21.15.172:0/1328889157 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8030610 con 0x7f11b406aed0 2024-03-16T19:30:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.389+0000 7f11b1d9b640 1 -- 172.21.15.172:0/1328889157 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a8030770 con 0x7f11b406aed0 2024-03-16T19:30:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.389+0000 7f11b1d9b640 1 -- 172.21.15.172:0/1328889157 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8030b00 con 0x7f11b406aed0 2024-03-16T19:30:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.390+0000 7f11b91f4640 1 -- 172.21.15.172:0/1328889157 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b406aed0 msgr2=0x7f11b415c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.390+0000 7f11b91f4640 1 --2- 172.21.15.172:0/1328889157 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b406aed0 0x7f11b415c530 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11a80097a0 tx=0x7f11a802b960 comp rx=0 tx=0).stop 2024-03-16T19:30:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.390+0000 7f11b91f4640 1 -- 172.21.15.172:0/1328889157 shutdown_connections 2024-03-16T19:30:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.390+0000 7f11b91f4640 1 --2- 172.21.15.172:0/1328889157 >> v2:172.21.15.174:3300/0 conn(0x7f11b40ca6d0 0x7f11b4071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.390+0000 7f11b91f4640 1 --2- 172.21.15.172:0/1328889157 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b406aed0 0x7f11b415c530 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.390+0000 7f11b91f4640 1 -- 172.21.15.172:0/1328889157 >> 172.21.15.172:0/1328889157 conn(0x7f11b4066c20 msgr2=0x7f11b4067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.390+0000 7f11b91f4640 1 -- 172.21.15.172:0/1328889157 shutdown_connections 2024-03-16T19:30:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.390+0000 7f11b91f4640 1 -- 172.21.15.172:0/1328889157 wait complete. 2024-03-16T19:30:09.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.391+0000 7f11b91f4640 1 Processor -- start 2024-03-16T19:30:09.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.391+0000 7f11b91f4640 1 -- start start 2024-03-16T19:30:09.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b91f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11b406aed0 0x7f11b41e3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b91f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b40ca6d0 0x7f11b41ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b91f4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11b4152910 0x7f11b41ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b91f4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b41f1940 con 0x7f11b4152910 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b91f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b41f24c0 con 0x7f11b406aed0 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b91f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b40c2ea0 con 0x7f11b40ca6d0 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b359e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11b4152910 0x7f11b41ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b359e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11b4152910 0x7f11b41ee910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39614/0 (socket says 172.21.15.172:39614) 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b359e640 1 -- 172.21.15.172:0/866670092 learned_addr learned my addr 172.21.15.172:0/866670092 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b2d9d640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11b406aed0 0x7f11b41e3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b259c640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b40ca6d0 0x7f11b41ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b359e640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b40ca6d0 msgr2=0x7f11b41ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b359e640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b40ca6d0 0x7f11b41ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b359e640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11b406aed0 msgr2=0x7f11b41e3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b359e640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11b406aed0 0x7f11b41e3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b359e640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac009a70 con 0x7f11b4152910 2024-03-16T19:30:09.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.392+0000 7f11b259c640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11b40ca6d0 0x7f11b41ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:09.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.393+0000 7f11b359e640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11b4152910 0x7f11b41ee910 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009eb0 tx=0x7f11ac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.393+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac009d50 con 0x7f11b4152910 2024-03-16T19:30:09.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.393+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac00f040 con 0x7f11b4152910 2024-03-16T19:30:09.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.393+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a8009400 con 0x7f11b4152910 2024-03-16T19:30:09.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.393+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0146a0 con 0x7f11b4152910 2024-03-16T19:30:09.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.393+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b41f1e70 con 0x7f11b4152910 2024-03-16T19:30:09.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.394+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f11b40688d0 con 0x7f11b4152910 2024-03-16T19:30:09.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.394+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f11ac014800 con 0x7f11b4152910 2024-03-16T19:30:09.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.395+0000 7f119b7fe640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f11a006b3e0 0x7f11a006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.395+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f11ac08aad0 con 0x7f11b4152910 2024-03-16T19:30:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.395+0000 7f11b2d9d640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f11a006b3e0 0x7f11a006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.395+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v32) v1 ==== 54+0+16 (secure 0 0 0) 0x7f11ac08f240 con 0x7f11b4152910 2024-03-16T19:30:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.395+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f11b4152f90 0x7f11b41505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.395+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b40c8000 con 0x7f11b4152f90 2024-03-16T19:30:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.395+0000 7f11b2d9d640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f11a006b3e0 0x7f11a006d8a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f11a4005880 tx=0x7f11a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.395+0000 7f11b259c640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f11b4152f90 0x7f11b41505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.396+0000 7f11b259c640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f11b4152f90 0x7f11b41505d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.396+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f11a8039450 con 0x7f11b4152f90 2024-03-16T19:30:09.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.411+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11b41f2600 con 0x7f11b4152f90 2024-03-16T19:30:09.393 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.413+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f11b41f2600 con 0x7f11b4152f90 2024-03-16T19:30:09.393 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:09.394 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:09.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.413+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f11b4151020 0x7f11b414f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.413+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b4151400 con 0x7f11b4151020 2024-03-16T19:30:09.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.413+0000 7f11b2d9d640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f11b4151020 0x7f11b414f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.413+0000 7f11b2d9d640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f11b4151020 0x7f11b414f3b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.414+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f11b4151400 con 0x7f11b4151020 2024-03-16T19:30:09.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.429+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11b40ca400 con 0x7f11b4151020 2024-03-16T19:30:09.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.431+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f11b40ca400 con 0x7f11b4151020 2024-03-16T19:30:09.412 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:09.412 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:09.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.431+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f11b414fe00 0x7f11b4061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.431+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b41501e0 con 0x7f11b414fe00 2024-03-16T19:30:09.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.431+0000 7f11b259c640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f11b414fe00 0x7f11b4061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.432+0000 7f11b259c640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f11b414fe00 0x7f11b4061930 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.432+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f11b41501e0 con 0x7f11b414fe00 2024-03-16T19:30:09.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.446+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11b4157710 con 0x7f11b414fe00 2024-03-16T19:30:09.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.448+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f11b4157710 con 0x7f11b414fe00 2024-03-16T19:30:09.429 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:09.430 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:09.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.448+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f11b4062380 0x7f11b40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.448+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b40627e0 con 0x7f11b4062380 2024-03-16T19:30:09.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.449+0000 7f11b359e640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f11b4062380 0x7f11b40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.449+0000 7f11b359e640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f11b4062380 0x7f11b40cada0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.449+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f11b40627e0 con 0x7f11b4062380 2024-03-16T19:30:09.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.464+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11b41f2060 con 0x7f11b4062380 2024-03-16T19:30:09.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.466+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f11b41f2060 con 0x7f11b4062380 2024-03-16T19:30:09.446 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:09.447 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:09.466+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:09.447 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:09.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.466+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f11b40cb7f0 0x7f11b402bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.466+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b41f2230 con 0x7f11b40cb7f0 2024-03-16T19:30:09.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.466+0000 7f11b2d9d640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f11b40cb7f0 0x7f11b402bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.467+0000 7f11b2d9d640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f11b40cb7f0 0x7f11b402bde0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.467+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f11b41f2230 con 0x7f11b40cb7f0 2024-03-16T19:30:09.453 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:09.453 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:09.453 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:09.453 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:09.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:09.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.482+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11b41f2230 con 0x7f11b40cb7f0 2024-03-16T19:30:09.464 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.484+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f11b41f2230 con 0x7f11b40cb7f0 2024-03-16T19:30:09.464 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:09.465 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:09.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.484+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f11b402c830 0x7f11b402cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.484+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b402d1d0 con 0x7f11b402c830 2024-03-16T19:30:09.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.484+0000 7f11b259c640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f11b402c830 0x7f11b402cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.485+0000 7f11b259c640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f11b402c830 0x7f11b402cc90 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.485+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f11b402d1d0 con 0x7f11b402c830 2024-03-16T19:30:09.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:30:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.500+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11b402d1d0 con 0x7f11b402c830 2024-03-16T19:30:09.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.502+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f11b402d1d0 con 0x7f11b402c830 2024-03-16T19:30:09.483 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:09.484 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:09.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.502+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f11b4205460 0x7f11b4207850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.502+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b402d340 con 0x7f11b4205460 2024-03-16T19:30:09.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.502+0000 7f11b359e640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f11b4205460 0x7f11b4207850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.503+0000 7f11b359e640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f11b4205460 0x7f11b4207850 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.503+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f11ac061b00 con 0x7f11b4205460 2024-03-16T19:30:09.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.519+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11b40cbc10 con 0x7f11b4205460 2024-03-16T19:30:09.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.521+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f11b40cbc10 con 0x7f11b4205460 2024-03-16T19:30:09.502 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:09.502 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:09.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.521+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f11b42082a0 0x7f11b420a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.521+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b420d4e0 con 0x7f11b42082a0 2024-03-16T19:30:09.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.521+0000 7f11b2d9d640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f11b42082a0 0x7f11b420a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.522+0000 7f11b2d9d640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f11b42082a0 0x7f11b420a690 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.522+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f11b420d4e0 con 0x7f11b42082a0 2024-03-16T19:30:09.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.537+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11b420aea0 con 0x7f11b42082a0 2024-03-16T19:30:09.519 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.538+0000 7f119b7fe640 1 -- 172.21.15.172:0/866670092 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f11b420aea0 con 0x7f11b42082a0 2024-03-16T19:30:09.519 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f11b4152f90 msgr2=0x7f11b41505d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f11b4152f90 0x7f11b41505d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f11b4151020 msgr2=0x7f11b414f3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f11b4151020 0x7f11b414f3b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f11b414fe00 msgr2=0x7f11b4061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f11b414fe00 0x7f11b4061930 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f11b4062380 msgr2=0x7f11b40cada0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f11b4062380 0x7f11b40cada0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.521 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f11b40cb7f0 msgr2=0x7f11b402bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f11b40cb7f0 0x7f11b402bde0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f11b402c830 msgr2=0x7f11b402cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f11b402c830 0x7f11b402cc90 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f11b4205460 msgr2=0x7f11b4207850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f11b4205460 0x7f11b4207850 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f11b42082a0 msgr2=0x7f11b420a690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f11b42082a0 0x7f11b420a690 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b359e640 1 -- 172.21.15.172:0/866670092 reap_dead start 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b359e640 1 -- 172.21.15.172:0/866670092 reap_dead start 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b359e640 1 -- 172.21.15.172:0/866670092 reap_dead start 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b359e640 1 -- 172.21.15.172:0/866670092 reap_dead start 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b359e640 1 -- 172.21.15.172:0/866670092 reap_dead start 2024-03-16T19:30:09.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f11a006b3e0 msgr2=0x7f11a006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f11a006b3e0 0x7f11a006d8a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f11a4005880 tx=0x7f11a4009040 comp rx=0 tx=0).stop 2024-03-16T19:30:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11b4152910 msgr2=0x7f11b41ee910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11b4152910 0x7f11b41ee910 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009eb0 tx=0x7f11ac00c490 comp rx=0 tx=0).stop 2024-03-16T19:30:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 shutdown_connections 2024-03-16T19:30:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11b4152910 0x7f11b41ee910 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 --2- 172.21.15.172:0/866670092 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f11a006b3e0 0x7f11a006d8a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 >> 172.21.15.172:0/866670092 conn(0x7f11b4066c20 msgr2=0x7f11b4071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 shutdown_connections 2024-03-16T19:30:09.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.539+0000 7f11b91f4640 1 -- 172.21.15.172:0/866670092 wait complete. 2024-03-16T19:30:09.575 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:09.575 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:09.575 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:09.575 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:09.575 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:09.576 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:09.595+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:09.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:09.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.610+0000 7f3707b6c640 1 Processor -- start 2024-03-16T19:30:09.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.610+0000 7f3707b6c640 1 -- start start 2024-03-16T19:30:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3707b6c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3700154b90 0x7f37000cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3707b6c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f370006bc60 0x7f370006c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3707b6c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37000cb360 con 0x7f370015e8a0 2024-03-16T19:30:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3707b6c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370015a950 con 0x7f370006bc60 2024-03-16T19:30:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3707b6c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370006c5f0 con 0x7f3700154b90 2024-03-16T19:30:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3706109640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f370006bc60 0x7f370006c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3706109640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f370006bc60 0x7f370006c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39622/0 (socket says 172.21.15.172:39622) 2024-03-16T19:30:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3706109640 1 -- 172.21.15.172:0/2704661335 learned_addr learned my addr 172.21.15.172:0/2704661335 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:09.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3705908640 1 --2- 172.21.15.172:0/2704661335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3700154b90 0x7f37000cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3706109640 1 -- 172.21.15.172:0/2704661335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3700154b90 msgr2=0x7f37000cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:09.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3706109640 1 --2- 172.21.15.172:0/2704661335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3700154b90 0x7f37000cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3706109640 1 -- 172.21.15.172:0/2704661335 >> v1:172.21.15.174:6789/0 conn(0x7f370015e8a0 legacy=0x7f37000cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.611+0000 7f3706109640 1 -- 172.21.15.172:0/2704661335 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370015abf0 con 0x7f370006bc60 2024-03-16T19:30:09.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.612+0000 7f3706109640 1 --2- 172.21.15.172:0/2704661335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370006bc60 0x7f370006c0b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009e50 tx=0x7f36f802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1ae60fa182cd50 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.612+0000 7f3704906640 1 -- 172.21.15.172:0/2704661335 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f802cb00 con 0x7f370006bc60 2024-03-16T19:30:09.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.612+0000 7f3704906640 1 -- 172.21.15.172:0/2704661335 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f802cc60 con 0x7f370006bc60 2024-03-16T19:30:09.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.612+0000 7f3704906640 1 -- 172.21.15.172:0/2704661335 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f80343f0 con 0x7f370006bc60 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.613+0000 7f3707b6c640 1 -- 172.21.15.172:0/2704661335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370006bc60 msgr2=0x7f370006c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.613+0000 7f3707b6c640 1 --2- 172.21.15.172:0/2704661335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370006bc60 0x7f370006c0b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009e50 tx=0x7f36f802bf60 comp rx=0 tx=0).stop 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.613+0000 7f3707b6c640 1 -- 172.21.15.172:0/2704661335 shutdown_connections 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.613+0000 7f3707b6c640 1 --2- 172.21.15.172:0/2704661335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370006bc60 0x7f370006c0b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.613+0000 7f3707b6c640 1 --2- 172.21.15.172:0/2704661335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3700154b90 0x7f37000cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.613+0000 7f3707b6c640 1 -- 172.21.15.172:0/2704661335 >> 172.21.15.172:0/2704661335 conn(0x7f3700066e00 msgr2=0x7f3700067210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.613+0000 7f3707b6c640 1 -- 172.21.15.172:0/2704661335 shutdown_connections 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.613+0000 7f3707b6c640 1 -- 172.21.15.172:0/2704661335 wait complete. 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.614+0000 7f3707b6c640 1 Processor -- start 2024-03-16T19:30:09.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.614+0000 7f3707b6c640 1 -- start start 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.614+0000 7f3707b6c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f370006bc60 0x7f37001f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3707b6c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3700154b90 0x7f37001f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3707b6c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f370015e8a0 0x7f37001f0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3707b6c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37000cfbc0 con 0x7f3700154b90 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3707b6c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37000ce880 con 0x7f370015e8a0 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3707b6c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37000ce270 con 0x7f370006bc60 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705107640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3700154b90 0x7f37001f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705908640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f370006bc60 0x7f37001f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3706109640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f370015e8a0 0x7f37001f0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705107640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3700154b90 0x7f37001f06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39636/0 (socket says 172.21.15.172:39636) 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705107640 1 -- 172.21.15.172:0/2420308273 learned_addr learned my addr 172.21.15.172:0/2420308273 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705107640 1 -- 172.21.15.172:0/2420308273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f370006bc60 msgr2=0x7f37001f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705107640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f370006bc60 0x7f37001f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705107640 1 -- 172.21.15.172:0/2420308273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f370015e8a0 msgr2=0x7f37001f0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705107640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f370015e8a0 0x7f37001f0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705107640 1 -- 172.21.15.172:0/2420308273 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f8009b30 con 0x7f3700154b90 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3705107640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3700154b90 0x7f37001f06d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f36f000c3c0 tx=0x7f36f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f36f67fc640 1 -- 172.21.15.172:0/2420308273 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f00052d0 con 0x7f3700154b90 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f36f67fc640 1 -- 172.21.15.172:0/2420308273 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f0005450 con 0x7f3700154b90 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37000cfdd0 con 0x7f3700154b90 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f36f67fc640 1 -- 172.21.15.172:0/2420308273 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0007260 con 0x7f3700154b90 2024-03-16T19:30:09.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.615+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3700154110 con 0x7f3700154b90 2024-03-16T19:30:09.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.617+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3700075a70 con 0x7f3700154b90 2024-03-16T19:30:09.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.617+0000 7f36f67fc640 1 -- 172.21.15.172:0/2420308273 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f36f00073c0 con 0x7f3700154b90 2024-03-16T19:30:09.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.617+0000 7f36f67fc640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e406b3b0 0x7f36e406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:09.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.617+0000 7f36f67fc640 1 -- 172.21.15.172:0/2420308273 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f36f0088f80 con 0x7f3700154b90 2024-03-16T19:30:09.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.617+0000 7f3705908640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e406b3b0 0x7f36e406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:09.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.618+0000 7f3705908640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e406b3b0 0x7f36e406d870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009b40 tx=0x7f36fc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:09.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.622+0000 7f36f67fc640 1 -- 172.21.15.172:0/2420308273 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36f00bce10 con 0x7f3700154b90 2024-03-16T19:30:09.648 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:09.666+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:09.700 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.362355229999999, 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.362340756999998, 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.705 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:30:09.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.361836267999998, 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:30:09.709 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:30:09.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.713 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.361820872000003, 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.717 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.361362188999998, 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.360816018999998, 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.722 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.360258928, 2024-03-16T19:30:09.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:30:09.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.359057180000001, 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:30:09.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:30:09.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:30:09.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.358478470999998, 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:30:09.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.358428944000003, 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.357468300999997, 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.742 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:30:09.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:30:09.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.353754103, 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:30:09.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:30:09.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:30:09.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.913353190999999, 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:30:09.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:30:09.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.347800188000001, 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:30:09.754 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.840201463, 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.762 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:09.763 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:09.763 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:09.764 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:09.764 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:09.782+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:09.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:09.810+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f37000cbb70 con 0x7f3700154b90 2024-03-16T19:30:09.865 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:09.883+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:09.871 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:09.871 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:09.871 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:09.872 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:09.880 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:09.965 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:09.985+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:10.027 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:10.027 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:10.027 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:10.027 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:10.027 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:10.036 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:10.079 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:10.098+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.805942576, 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:10.144 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:10.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:10.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:10.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.804715154, 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:10.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.804312447999997, 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:10.153 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:10.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.157 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.803703697000003, 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:10.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:10.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.803259828000002, 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:10.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:10.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.803117051999997, 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:10.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.803102510999999, 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.173 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.800224815999997, 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.178 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.178 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.178 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:10.179 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:10.180 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:10.197+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:10.280 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:10.300+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:10.327 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:10.327 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:10.327 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:10.327 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:10.335 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:10.381 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:10.399+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:10.460 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:10.460 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:10.460 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:10.460 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:10.460 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:10.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.482+0000 7f36f67fc640 1 -- 172.21.15.172:0/2420308273 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f36f0053bf0 con 0x7f3700154b90 2024-03-16T19:30:10.463 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:30:10.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.484+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e406b3b0 msgr2=0x7f36e406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:10.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.484+0000 7f3707b6c640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e406b3b0 0x7f36e406d870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009b40 tx=0x7f36fc009190 comp rx=0 tx=0).stop 2024-03-16T19:30:10.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.484+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3700154b90 msgr2=0x7f37001f06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:10.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.484+0000 7f3707b6c640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3700154b90 0x7f37001f06d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f36f000c3c0 tx=0x7f36f000c880 comp rx=0 tx=0).stop 2024-03-16T19:30:10.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.485+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 shutdown_connections 2024-03-16T19:30:10.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.485+0000 7f3707b6c640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f370015e8a0 0x7f37001f0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:10.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.485+0000 7f3707b6c640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3700154b90 0x7f37001f06d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:10.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.485+0000 7f3707b6c640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f36e406b3b0 0x7f36e406d870 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:10.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.485+0000 7f3707b6c640 1 --2- 172.21.15.172:0/2420308273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f370006bc60 0x7f37001f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:10.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.485+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 >> 172.21.15.172:0/2420308273 conn(0x7f3700066e00 msgr2=0x7f37000c5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:10.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.485+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 shutdown_connections 2024-03-16T19:30:10.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.485+0000 7f3707b6c640 1 -- 172.21.15.172:0/2420308273 wait complete. 2024-03-16T19:30:10.471 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:10.519 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:10.539+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:10.583 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:10.601+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:10.618 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:10.618 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:10.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:10.618 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:10.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.618 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.279795090999997, 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:30:10.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:30:10.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.279715973000002, 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.279101849, 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.627 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:30:10.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:30:10.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:10.631 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.27866178, 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:30:10.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:30:10.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.278420189999999, 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:10.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:30:10.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:30:10.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:30:10.640 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.640 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.640 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.640 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:10.640 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:10.640 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:10.641 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:10.683 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:10.703+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:10.696 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:30:10.696 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:30:10.696 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-16T19:30:10.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-03-16T19:30:10.784 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:10.804+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:10.788 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:10.789 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:10.789 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:10.789 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:10.797 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:10.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.819+0000 7f65515ac640 1 Processor -- start 2024-03-16T19:30:10.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.819+0000 7f65515ac640 1 -- start start 2024-03-16T19:30:10.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.819+0000 7f65515ac640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f654c06bb10 0x7f654c0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:10.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.819+0000 7f65515ac640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c0cc720 con 0x7f654c06bef0 2024-03-16T19:30:10.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.819+0000 7f65515ac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c0cc9c0 con 0x7f654c0cc260 2024-03-16T19:30:10.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.819+0000 7f65515ac640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654c15de30 con 0x7f654c06bb10 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.820+0000 7f654affd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f654c06bb10 0x7f654c0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.820+0000 7f654a7fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f654c06bef0 0x7f654c154660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:57222/0 (socket says 172.21.15.172:57222) 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.820+0000 7f654a7fc640 1 -- 172.21.15.172:0/3411476408 learned_addr learned my addr 172.21.15.172:0/3411476408 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.820+0000 7f654affd640 1 -- 172.21.15.172:0/3411476408 >> v1:172.21.15.174:6789/0 conn(0x7f654c0cc260 legacy=0x7f654c1651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.820+0000 7f654affd640 1 -- 172.21.15.172:0/3411476408 >> v1:172.21.15.172:6789/0 conn(0x7f654c06bef0 legacy=0x7f654c154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.820+0000 7f6549ffb640 1 -- 172.21.15.172:0/3411476408 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215555161 0 0) 0x7f654c0cc720 con 0x7f654c06bef0 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.820+0000 7f654affd640 1 -- 172.21.15.172:0/3411476408 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c15e110 con 0x7f654c06bb10 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f654affd640 1 --2- 172.21.15.172:0/3411476408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bb10 0x7f654c0cae50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f653c0097a0 tx=0x7f653c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e266ebaa1c5bd782 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f6549ffb640 1 -- 172.21.15.172:0/3411476408 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c02f810 con 0x7f654c06bb10 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f6549ffb640 1 -- 172.21.15.172:0/3411476408 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c02f970 con 0x7f654c06bb10 2024-03-16T19:30:10.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f6549ffb640 1 -- 172.21.15.172:0/3411476408 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c02fd00 con 0x7f654c06bb10 2024-03-16T19:30:10.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f65515ac640 1 -- 172.21.15.172:0/3411476408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bb10 msgr2=0x7f654c0cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:10.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f65515ac640 1 --2- 172.21.15.172:0/3411476408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bb10 0x7f654c0cae50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f653c0097a0 tx=0x7f653c02b9f0 comp rx=0 tx=0).stop 2024-03-16T19:30:10.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f65515ac640 1 -- 172.21.15.172:0/3411476408 shutdown_connections 2024-03-16T19:30:10.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f65515ac640 1 --2- 172.21.15.172:0/3411476408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bb10 0x7f654c0cae50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:10.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f65515ac640 1 -- 172.21.15.172:0/3411476408 >> 172.21.15.172:0/3411476408 conn(0x7f654c067250 msgr2=0x7f654c0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:10.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f65515ac640 1 -- 172.21.15.172:0/3411476408 shutdown_connections 2024-03-16T19:30:10.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.821+0000 7f65515ac640 1 -- 172.21.15.172:0/3411476408 wait complete. 2024-03-16T19:30:10.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.823+0000 7f65515ac640 1 Processor -- start 2024-03-16T19:30:10.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.823+0000 7f65515ac640 1 -- start start 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.823+0000 7f65515ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f654c06bb10 0x7f654c1f1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.823+0000 7f65515ac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bef0 0x7f654c1483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.823+0000 7f65515ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f654c0cc260 0x7f654c148920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.823+0000 7f65515ac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654c15c6e0 con 0x7f654c06bef0 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.823+0000 7f65515ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654c1f0780 con 0x7f654c0cc260 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.823+0000 7f65515ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654c15be40 con 0x7f654c06bb10 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.823+0000 7f654a7fc640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bef0 0x7f654c1483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654a7fc640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bef0 0x7f654c1483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39654/0 (socket says 172.21.15.172:39654) 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654a7fc640 1 -- 172.21.15.172:0/200821509 learned_addr learned my addr 172.21.15.172:0/200821509 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654affd640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f654c06bb10 0x7f654c1f1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654b7fe640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f654c0cc260 0x7f654c148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654a7fc640 1 -- 172.21.15.172:0/200821509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f654c06bb10 msgr2=0x7f654c1f1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654a7fc640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f654c06bb10 0x7f654c1f1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:10.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654a7fc640 1 -- 172.21.15.172:0/200821509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f654c0cc260 msgr2=0x7f654c148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:10.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654a7fc640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f654c0cc260 0x7f654c148920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:10.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654a7fc640 1 -- 172.21.15.172:0/200821509 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c009400 con 0x7f654c06bef0 2024-03-16T19:30:10.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f654a7fc640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bef0 0x7f654c1483e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f654000c460 tx=0x7f654000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:10.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f65337fe640 1 -- 172.21.15.172:0/200821509 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400074e0 con 0x7f654c06bef0 2024-03-16T19:30:10.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f65337fe640 1 -- 172.21.15.172:0/200821509 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6540007680 con 0x7f654c06bef0 2024-03-16T19:30:10.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654c15c120 con 0x7f654c06bef0 2024-03-16T19:30:10.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f65337fe640 1 -- 172.21.15.172:0/200821509 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540004a00 con 0x7f654c06bef0 2024-03-16T19:30:10.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.824+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654c1f0a40 con 0x7f654c06bef0 2024-03-16T19:30:10.808 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.825+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654c0681c0 con 0x7f654c06bef0 2024-03-16T19:30:10.808 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.825+0000 7f65337fe640 1 -- 172.21.15.172:0/200821509 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6540004b60 con 0x7f654c06bef0 2024-03-16T19:30:10.808 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.826+0000 7f65337fe640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f653806b3e0 0x7f653806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:10.808 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.826+0000 7f65337fe640 1 -- 172.21.15.172:0/200821509 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f6540089130 con 0x7f654c06bef0 2024-03-16T19:30:10.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.826+0000 7f654affd640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f653806b3e0 0x7f653806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:10.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.826+0000 7f654affd640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f653806b3e0 0x7f653806d8a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f653c036000 tx=0x7f653c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:10.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:10.830+0000 7f65337fe640 1 -- 172.21.15.172:0/200821509 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6540053e30 con 0x7f654c06bef0 2024-03-16T19:30:10.885 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:10.904+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:10.906 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:10.907 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:10.907 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:10.907 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:10.907 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:10.915 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:10.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.004+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f654c0c6cf0 con 0x7f654c06bef0 2024-03-16T19:30:10.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.006+0000 7f65337fe640 1 -- 172.21.15.172:0/200821509 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v33) v1 ==== 81+0+35230 (secure 0 0 0) 0x7f65400574f0 con 0x7f654c06bef0 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 33, 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout: "fsid": "6c00e600-a879-4eb6-acee-0fdf0f76922a", 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout: "created": "2024-03-16T19:29:02.010581+0000", 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout: "modified": "2024-03-16T19:30:10.477447+0000", 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout: "last_up_change": "2024-03-16T19:29:13.180924+0000", 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout: "last_in_change": "2024-03-16T19:29:04.178546+0000", 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-16T19:30:10.987 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_num": 5806080, 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_set": [ 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "nodeep-scrub", 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "noscrub", 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "pglog_hardlimit", 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "purged_snapdirs", 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_deletes", 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "sortbitwise" 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_version": 4, 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "full_ratio": 0.94999998807907104, 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_snapshot": "", 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_max": 3, 2024-03-16T19:30:10.988 INFO:teuthology.orchestra.run.smithi172.stdout: "max_osd": 8, 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "require_min_compat_client": "luminous", 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "min_compat_client": "luminous", 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "require_osd_release": "squid", 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "allow_crimson": false, 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "pools": [ 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "pool": 1, 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_name": ".mgr", 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "create_time": "2024-03-16T19:29:14.242742+0000", 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": 1, 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_names": "hashpspool", 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "type": 1, 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 2, 2024-03-16T19:30:10.989 INFO:teuthology.orchestra.run.smithi172.stdout: "min_size": 1, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_rule": 0, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_count": 0, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_target": 0, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_barrier": 0, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "object_hash": 2, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_mode": "off", 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num": 1, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num": 1, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num_target": 1, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_target": 1, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_pending": 1, 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "last_pg_merge_meta": { 2024-03-16T19:30:10.990 INFO:teuthology.orchestra.run.smithi172.stdout: "source_pgid": "0.0", 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "ready_epoch": 0, 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_started": 0, 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_clean": 0, 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "source_version": "0'0", 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "target_version": "0'0" 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "last_change": "17", 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend": "0", 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_preluminous": "0", 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "auid": 0, 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_mode": "selfmanaged", 2024-03-16T19:30:10.991 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_seq": 0, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_epoch": 0, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_snaps": [], 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps": "[]", 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_bytes": 0, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_objects": 0, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "tiers": [], 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_of": -1, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "read_tier": -1, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "write_tier": -1, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_mode": "none", 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_bytes": 0, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_objects": 0, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-16T19:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_flush_age": 0, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_evict_age": 0, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profile": "", 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_params": { 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "none" 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_period": 0, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_count": 0, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "use_gmt_hitset": true, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "min_read_recency_for_promote": 0, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "min_write_recency_for_promote": 0, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_grade_decay_rate": 0, 2024-03-16T19:30:10.993 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_search_last_n": 0, 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "grade_table": [], 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "stripe_width": 0, 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "expected_num_objects": 0, 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "fast_read": false, 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "options": { 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_max": 32, 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_min": 1 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "application_metadata": { 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": {} 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "read_balance": { 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "score_type": "Fair distribution", 2024-03-16T19:30:10.994 INFO:teuthology.orchestra.run.smithi172.stdout: "score_acting": 6.9000000953674316, 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "score_stable": 6.9000000953674316, 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "optimal_score": 0.28999999165534973, 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_acting": 2, 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_stable": 2, 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity_weighted": 0.875, 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity": 1, 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "pool": 2, 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_name": "cephfs_metadata", 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "create_time": "2024-03-16T19:29:24.121132+0000", 2024-03-16T19:30:10.995 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": 1, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_names": "hashpspool", 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "type": 1, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 2, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "min_size": 1, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_rule": 0, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_count": 0, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_target": 0, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_barrier": 0, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "object_hash": 2, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_mode": "off", 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num": 32, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num": 32, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num_target": 32, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_target": 32, 2024-03-16T19:30:10.996 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_pending": 32, 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "last_pg_merge_meta": { 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "source_pgid": "0.0", 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "ready_epoch": 0, 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_started": 0, 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_clean": 0, 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "source_version": "0'0", 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "target_version": "0'0" 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "last_change": "20", 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend": "0", 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_preluminous": "0", 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "auid": 0, 2024-03-16T19:30:10.997 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_mode": "selfmanaged", 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_seq": 0, 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_epoch": 0, 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_snaps": [], 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps": "[]", 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_bytes": 0, 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_objects": 0, 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "tiers": [], 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_of": -1, 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "read_tier": -1, 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "write_tier": -1, 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_mode": "none", 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_bytes": 0, 2024-03-16T19:30:10.998 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_objects": 0, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_flush_age": 0, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_evict_age": 0, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profile": "", 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_params": { 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "none" 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_period": 0, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_count": 0, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "use_gmt_hitset": true, 2024-03-16T19:30:10.999 INFO:teuthology.orchestra.run.smithi172.stdout: "min_read_recency_for_promote": 0, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "min_write_recency_for_promote": 0, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_grade_decay_rate": 0, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_search_last_n": 0, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "grade_table": [], 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "stripe_width": 0, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "expected_num_objects": 0, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "fast_read": false, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "options": { 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_bias": 4, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_min": 16, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_priority": 5 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.000 INFO:teuthology.orchestra.run.smithi172.stdout: "application_metadata": { 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs": { 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "metadata": "cephfs" 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "read_balance": { 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "score_type": "Fair distribution", 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "score_acting": 1.5399999618530273, 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "score_stable": 1.5399999618530273, 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "optimal_score": 1.1399999856948853, 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_acting": 1.75, 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_stable": 1.75, 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity_weighted": 0.875, 2024-03-16T19:30:11.001 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity": 1, 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "pool": 3, 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_name": "cephfs_data", 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "create_time": "2024-03-16T19:29:24.651099+0000", 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "flags": 1, 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "flags_names": "hashpspool", 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "type": 1, 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 2, 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "min_size": 1, 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_rule": 0, 2024-03-16T19:30:11.002 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_count": 0, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_target": 0, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_barrier": 0, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "object_hash": 2, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_autoscale_mode": "off", 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num": 64, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num": 64, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_placement_num_target": 64, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_target": 64, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_pending": 64, 2024-03-16T19:30:11.003 INFO:teuthology.orchestra.run.smithi172.stdout: "last_pg_merge_meta": { 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "source_pgid": "0.0", 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "ready_epoch": 0, 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_started": 0, 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "last_epoch_clean": 0, 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "source_version": "0'0", 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "target_version": "0'0" 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "last_change": "21", 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend": "0", 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "last_force_op_resend_preluminous": "0", 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "auid": 0, 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_mode": "selfmanaged", 2024-03-16T19:30:11.004 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_seq": 0, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "snap_epoch": 0, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "pool_snaps": [], 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps": "[]", 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_bytes": 0, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "quota_max_objects": 0, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "tiers": [], 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_of": -1, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "read_tier": -1, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "write_tier": -1, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_mode": "none", 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_bytes": 0, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "target_max_objects": 0, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-16T19:30:11.005 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_flush_age": 0, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "cache_min_evict_age": 0, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profile": "", 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_params": { 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "none" 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_period": 0, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_count": 0, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "use_gmt_hitset": true, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "min_read_recency_for_promote": 0, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "min_write_recency_for_promote": 0, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_grade_decay_rate": 0, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "hit_set_search_last_n": 0, 2024-03-16T19:30:11.006 INFO:teuthology.orchestra.run.smithi172.stdout: "grade_table": [], 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "stripe_width": 0, 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "expected_num_objects": 0, 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "fast_read": false, 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "options": { 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_num_min": 64, 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "target_size_ratio": 0.89999997615814209 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "application_metadata": { 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "cephfs": { 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "data": "cephfs" 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "read_balance": { 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "score_type": "Fair distribution", 2024-03-16T19:30:11.007 INFO:teuthology.orchestra.run.smithi172.stdout: "score_acting": 1.3200000524520874, 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "score_stable": 1.3200000524520874, 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "optimal_score": 1.1399999856948853, 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_acting": 1.5, 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "raw_score_stable": 1.5, 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity_weighted": 0.875, 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity": 1, 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "osds": [ 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 0, 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "460ed9e4-eacf-4544-9199-23ca7208b663", 2024-03-16T19:30:11.008 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 0, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 19, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6800", 2024-03-16T19:30:11.009 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6801", 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.010 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6802", 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6803", 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.011 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6806", 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6807", 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6804", 2024-03-16T19:30:11.012 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6805", 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 491754535 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6801/491754535", 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6803/491754535", 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6807/491754535", 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6805/491754535", 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:30:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 1, 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "b84ebb50-079d-460f-bf05-71040f17500a", 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:30:11.014 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 31, 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6808", 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6809", 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.015 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6810", 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6811", 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6814", 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6815", 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.017 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6812", 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6813", 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3098410089 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6809/3098410089", 2024-03-16T19:30:11.018 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6811/3098410089", 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6815/3098410089", 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6813/3098410089", 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 2, 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "6774c42c-5373-4543-8f8c-ed7b82c12057", 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:30:11.019 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 31, 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6823", 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.020 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6825", 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6826", 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6827", 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6830", 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6831", 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6828", 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.023 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6829", 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1599824697 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6825/1599824697", 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6827/1599824697", 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6831/1599824697", 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6829/1599824697", 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 3, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "b96838ef-d991-43ab-b2d7-16deab58ca78", 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:30:11.025 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6816", 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6817", 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6818", 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6819", 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6822", 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6824", 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6820", 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.172:6821", 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 1613555619 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.172:6817/1613555619", 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.172:6819/1613555619", 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.172:6824/1613555619", 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.172:6821/1613555619", 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 4, 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "f5c78559-d22e-4847-be3f-2f597d7f6789", 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:30:11.030 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6808", 2024-03-16T19:30:11.031 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6809", 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6810", 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6811", 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6814", 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.033 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6815", 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6812", 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6813", 2024-03-16T19:30:11.034 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 967866068 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6809/967866068", 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6811/967866068", 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6815/967866068", 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6813/967866068", 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 5, 2024-03-16T19:30:11.035 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "e1f4797f-31c3-4292-a4a1-88fa0ab64f9a", 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.036 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6816", 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6817", 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6818", 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:30:11.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6820", 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6827", 2024-03-16T19:30:11.038 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6829", 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6822", 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:30:11.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6825", 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3985312519 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6817/3985312519", 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6820/3985312519", 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6829/3985312519", 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6825/3985312519", 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:30:11.040 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 6, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "ea3e888e-c4c6-444a-8ebe-e26cddb2084b", 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 19, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:30:11.041 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6819", 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6821", 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.042 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6823", 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6824", 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.043 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6830", 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6831", 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:30:11.044 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6826", 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6828", 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 3676139844 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6821/3676139844", 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6824/3676139844", 2024-03-16T19:30:11.045 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6831/3676139844", 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6828/3676139844", 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 7, 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "uuid": "bb0252a5-3aa1-43f0-be38-37413819720b", 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "up": 1, 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "in": 1, 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "weight": 1, 2024-03-16T19:30:11.046 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_affinity": 1, 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_begin": 0, 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "last_clean_end": 0, 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "up_from": 13, 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "up_thru": 22, 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "down_at": 0, 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "lost_at": 0, 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addrs": { 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6800", 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6801", 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addrs": { 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6802", 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6803", 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addrs": { 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6806", 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.049 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6807", 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addrs": { 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: "addrvec": [ 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v2", 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6804", 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "v1", 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "addr": "172.21.15.174:6805", 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "nonce": 339941177 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "public_addr": "172.21.15.174:6801/339941177", 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "cluster_addr": "172.21.15.174:6803/339941177", 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_back_addr": "172.21.15.174:6807/339941177", 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "heartbeat_front_addr": "172.21.15.174:6805/339941177", 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "state": [ 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "exists", 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: "up" 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_xinfo": [ 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 0, 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.304258+0000", 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 1, 2024-03-16T19:30:11.052 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.336924+0000", 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 2, 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:30:11.053 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.359766+0000", 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 3, 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.384687+0000", 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 4, 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.291288+0000", 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 5, 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:30:11.055 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.318479+0000", 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 6, 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.313564+0000", 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:30:11.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": 7, 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "down_stamp": "0.000000", 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_probability": 0, 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "laggy_interval": 0, 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "features": 4540701547738038271, 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "old_weight": 0, 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "last_purged_snaps_scrub": "2024-03-16T19:29:09.232618+0000", 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "dead_epoch": 0 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap": [], 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_items": [ 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.057 INFO:teuthology.orchestra.run.smithi172.stdout: "pgid": "3.8", 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: "mappings": [ 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: "from": 3, 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: "to": 2 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: "pgid": "3.d", 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: "mappings": [ 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: "from": 5, 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: "to": 1 2024-03-16T19:30:11.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: "from": 7, 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: "to": 2 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: "pgid": "3.1f", 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: "mappings": [ 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: "from": 5, 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: "to": 1 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: "pgid": "3.2a", 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: "mappings": [ 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: "from": 5, 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: "to": 2 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_primaries": [], 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_temp": [], 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: "primary_temp": [], 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: "blocklist": {}, 2024-03-16T19:30:11.060 INFO:teuthology.orchestra.run.smithi172.stdout: "range_blocklist": {}, 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "erasure_code_profiles": { 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "default": { 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "crush-failure-domain": "osd", 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "k": "2", 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "m": "1", 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "plugin": "jerasure", 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "technique": "reed_sol_van" 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "removed_snaps_queue": [], 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "new_removed_snaps": [], 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "new_purged_snaps": [], 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "crush_node_flags": {}, 2024-03-16T19:30:11.061 INFO:teuthology.orchestra.run.smithi172.stdout: "device_class_flags": {}, 2024-03-16T19:30:11.062 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_mode": { 2024-03-16T19:30:11.062 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_mode_enabled": false, 2024-03-16T19:30:11.062 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_bucket_count": 0, 2024-03-16T19:30:11.062 INFO:teuthology.orchestra.run.smithi172.stdout: "degraded_stretch_mode": 0, 2024-03-16T19:30:11.062 INFO:teuthology.orchestra.run.smithi172.stdout: "recovering_stretch_mode": 0, 2024-03-16T19:30:11.062 INFO:teuthology.orchestra.run.smithi172.stdout: "stretch_mode_bucket": 0 2024-03-16T19:30:11.062 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:11.062 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:11.062 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:11.018+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f653806b3e0 msgr2=0x7f653806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f653806b3e0 0x7f653806d8a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f653c036000 tx=0x7f653c008030 comp rx=0 tx=0).stop 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bef0 msgr2=0x7f654c1483e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bef0 0x7f654c1483e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f654000c460 tx=0x7f654000c920 comp rx=0 tx=0).stop 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 shutdown_connections 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f654c0cc260 0x7f654c148920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f654c06bef0 0x7f654c1483e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f653806b3e0 0x7f653806d8a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 --2- 172.21.15.172:0/200821509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f654c06bb10 0x7f654c1f1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 >> 172.21.15.172:0/200821509 conn(0x7f654c067250 msgr2=0x7f654c14eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 shutdown_connections 2024-03-16T19:30:11.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.008+0000 7f65515ac640 1 -- 172.21.15.172:0/200821509 wait complete. 2024-03-16T19:30:11.065 INFO:tasks.thrashosds.thrasher:j is {'epoch': 33, 'fsid': '6c00e600-a879-4eb6-acee-0fdf0f76922a', 'created': '2024-03-16T19:29:02.010581+0000', 'modified': '2024-03-16T19:30:10.477447+0000', 'last_up_change': '2024-03-16T19:29:13.180924+0000', 'last_in_change': '2024-03-16T19:29:04.178546+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:29:14.242742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:29:24.121132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:29:24.651099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '460ed9e4-eacf-4544-9199-23ca7208b663', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 491754535}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 491754535}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 491754535}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 491754535}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 491754535}]}, 'public_addr': '172.21.15.172:6801/491754535', 'cluster_addr': '172.21.15.172:6803/491754535', 'heartbeat_back_addr': '172.21.15.172:6807/491754535', 'heartbeat_front_addr': '172.21.15.172:6805/491754535', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b84ebb50-079d-460f-bf05-71040f17500a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6809', 'nonce': 3098410089}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6810', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 3098410089}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6814', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 3098410089}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 3098410089}, {'type': 'v1', 'addr': '172.21.15.172:6813', 'nonce': 3098410089}]}, 'public_addr': '172.21.15.172:6809/3098410089', 'cluster_addr': '172.21.15.172:6811/3098410089', 'heartbeat_back_addr': '172.21.15.172:6815/3098410089', 'heartbeat_front_addr': '172.21.15.172:6813/3098410089', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6774c42c-5373-4543-8f8c-ed7b82c12057', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6823', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 1599824697}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 1599824697}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 1599824697}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 1599824697}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 1599824697}]}, 'public_addr': '172.21.15.172:6825/1599824697', 'cluster_addr': '172.21.15.172:6827/1599824697', 'heartbeat_back_addr': '172.21.15.172:6831/1599824697', 'heartbeat_front_addr': '172.21.15.172:6829/1599824697', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'b96838ef-d991-43ab-b2d7-16deab58ca78', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6817', 'nonce': 1613555619}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6818', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6819', 'nonce': 1613555619}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6824', 'nonce': 1613555619}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6820', 'nonce': 1613555619}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 1613555619}]}, 'public_addr': '172.21.15.172:6817/1613555619', 'cluster_addr': '172.21.15.172:6819/1613555619', 'heartbeat_back_addr': '172.21.15.172:6824/1613555619', 'heartbeat_front_addr': '172.21.15.172:6821/1613555619', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f5c78559-d22e-4847-be3f-2f597d7f6789', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6808', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6809', 'nonce': 967866068}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6810', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6811', 'nonce': 967866068}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6814', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6815', 'nonce': 967866068}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6812', 'nonce': 967866068}, {'type': 'v1', 'addr': '172.21.15.174:6813', 'nonce': 967866068}]}, 'public_addr': '172.21.15.174:6809/967866068', 'cluster_addr': '172.21.15.174:6811/967866068', 'heartbeat_back_addr': '172.21.15.174:6815/967866068', 'heartbeat_front_addr': '172.21.15.174:6813/967866068', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'e1f4797f-31c3-4292-a4a1-88fa0ab64f9a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6816', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6817', 'nonce': 3985312519}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6818', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6820', 'nonce': 3985312519}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6827', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6829', 'nonce': 3985312519}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6822', 'nonce': 3985312519}, {'type': 'v1', 'addr': '172.21.15.174:6825', 'nonce': 3985312519}]}, 'public_addr': '172.21.15.174:6817/3985312519', 'cluster_addr': '172.21.15.174:6820/3985312519', 'heartbeat_back_addr': '172.21.15.174:6829/3985312519', 'heartbeat_front_addr': '172.21.15.174:6825/3985312519', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ea3e888e-c4c6-444a-8ebe-e26cddb2084b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6819', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6821', 'nonce': 3676139844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6823', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6824', 'nonce': 3676139844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6830', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6831', 'nonce': 3676139844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6826', 'nonce': 3676139844}, {'type': 'v1', 'addr': '172.21.15.174:6828', 'nonce': 3676139844}]}, 'public_addr': '172.21.15.174:6821/3676139844', 'cluster_addr': '172.21.15.174:6824/3676139844', 'heartbeat_back_addr': '172.21.15.174:6831/3676139844', 'heartbeat_front_addr': '172.21.15.174:6828/3676139844', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bb0252a5-3aa1-43f0-be38-37413819720b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6800', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6801', 'nonce': 339941177}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6802', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6803', 'nonce': 339941177}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6806', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6807', 'nonce': 339941177}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6804', 'nonce': 339941177}, {'type': 'v1', 'addr': '172.21.15.174:6805', 'nonce': 339941177}]}, 'public_addr': '172.21.15.174:6801/339941177', 'cluster_addr': '172.21.15.174:6803/339941177', 'heartbeat_back_addr': '172.21.15.174:6807/339941177', 'heartbeat_front_addr': '172.21.15.174:6805/339941177', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.304258+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.336924+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.359766+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.384687+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.291288+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.318479+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.313564+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-16T19:29:09.232618+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.8', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.d', 'mappings': [{'from': 5, 'to': 1}, {'from': 7, 'to': 2}]}, {'pgid': '3.1f', 'mappings': [{'from': 5, 'to': 1}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-16T19:30:11.065 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.d 2024-03-16T19:30:11.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 3.d 2024-03-16T19:30:11.066 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.725292476999996, 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:30:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:30:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.725268303999997, 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:30:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:30:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:30:11.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:30:11.076 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.725262477000001, 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:30:11.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.725258564999997, 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.082 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:11.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:11.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:30:11.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.725252583, 2024-03-16T19:30:11.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:11.092 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.725248917000002, 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:11.097 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.725238257000001, 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:11.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:30:11.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.725234368000002, 2024-03-16T19:30:11.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:30:11.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:30:11.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.721717677000001, 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:30:11.105 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:30:11.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.721647242000003, 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.109 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:30:11.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:30:11.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.721264894999997, 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:30:11.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.719498612000002, 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:30:11.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.717685690000003, 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:30:11.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:25 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.717447909999997, 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189085, 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2099999999999996e-07 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365068+0000", 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9700000000000001e-06 2024-03-16T19:30:11.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0020999999999999e-05 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7449999999999998e-06 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365136+0000", 2024-03-16T19:30:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0973999999999997e-05 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365145+0000", 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9560000000000003e-06 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365253+0000", 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107798 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.716829941, 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:30:11.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:30:11.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.710969175000002, 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.135 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:30:11.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:30:11.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:30:11.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 31.278417966999999, 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:30:11.139 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:40 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:11.144 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 31.275137454999999, 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019577200000000001, 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1600000000000001e-07 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807377+0000", 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7839999999999999e-06 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807384+0000", 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8410000000000002e-06 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807388+0000", 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0550000000000001e-06 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807431+0000", 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3271e-05 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807441+0000", 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2320000000000007e-06 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807570+0000", 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000129073 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.710924260999999, 2024-03-16T19:30:11.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:11.153 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.203312010000001, 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.158 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:11.160 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:11.160 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:11.178+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:11.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.279+0000 7fa1e773e640 1 Processor -- start 2024-03-16T19:30:11.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.279+0000 7fa1e773e640 1 -- start start 2024-03-16T19:30:11.260 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:11.279+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:11.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e773e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa1e01580b0 0x7fa1e006b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e773e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e00c1eb0 con 0x7fa1e006baf0 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e773e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e00c1350 con 0x7fa1e006aed0 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e773e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e00c1150 con 0x7fa1e01580b0 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e4cd9640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa1e006baf0 0x7fa1e015ffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:57234/0 (socket says 172.21.15.172:57234) 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e4cd9640 1 -- 172.21.15.172:0/2818630513 learned_addr learned my addr 172.21.15.172:0/2818630513 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e5cdb640 1 --2- 172.21.15.172:0/2818630513 >> v2:172.21.15.174:3300/0 conn(0x7fa1e01580b0 0x7fa1e006b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1d7fff640 1 -- 172.21.15.172:0/2818630513 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976174474 0 0) 0x7fa1e00c1eb0 con 0x7fa1e006baf0 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1d7fff640 1 -- 172.21.15.172:0/2818630513 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1cc003520 con 0x7fa1e006baf0 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e5cdb640 1 -- 172.21.15.172:0/2818630513 >> v1:172.21.15.174:6789/0 conn(0x7fa1e006aed0 legacy=0x7fa1e01579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e5cdb640 1 -- 172.21.15.172:0/2818630513 >> v1:172.21.15.172:6789/0 conn(0x7fa1e006baf0 legacy=0x7fa1e015ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1d7fff640 1 -- 172.21.15.172:0/2818630513 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871660180 0 0) 0x7fa1e00c1350 con 0x7fa1e006aed0 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.280+0000 7fa1e5cdb640 1 -- 172.21.15.172:0/2818630513 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e00c1610 con 0x7fa1e01580b0 2024-03-16T19:30:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.281+0000 7fa1e5cdb640 1 --2- 172.21.15.172:0/2818630513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e01580b0 0x7fa1e006b2b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8009d90 tx=0x7fa1d802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21028080527b4a28 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.281+0000 7fa1d7fff640 1 -- 172.21.15.172:0/2818630513 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d802cd00 con 0x7fa1e01580b0 2024-03-16T19:30:11.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.281+0000 7fa1d7fff640 1 -- 172.21.15.172:0/2818630513 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d802ce60 con 0x7fa1e01580b0 2024-03-16T19:30:11.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.281+0000 7fa1d7fff640 1 -- 172.21.15.172:0/2818630513 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8034660 con 0x7fa1e01580b0 2024-03-16T19:30:11.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.282+0000 7fa1e773e640 1 -- 172.21.15.172:0/2818630513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e01580b0 msgr2=0x7fa1e006b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.282+0000 7fa1e773e640 1 --2- 172.21.15.172:0/2818630513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e01580b0 0x7fa1e006b2b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8009d90 tx=0x7fa1d802bf50 comp rx=0 tx=0).stop 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.282+0000 7fa1e773e640 1 -- 172.21.15.172:0/2818630513 shutdown_connections 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.282+0000 7fa1e773e640 1 --2- 172.21.15.172:0/2818630513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e01580b0 0x7fa1e006b2b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.282+0000 7fa1e773e640 1 -- 172.21.15.172:0/2818630513 >> 172.21.15.172:0/2818630513 conn(0x7fa1e00cc030 msgr2=0x7fa1e00cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.282+0000 7fa1e773e640 1 -- 172.21.15.172:0/2818630513 shutdown_connections 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.282+0000 7fa1e773e640 1 -- 172.21.15.172:0/2818630513 wait complete. 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.283+0000 7fa1e773e640 1 Processor -- start 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.283+0000 7fa1e773e640 1 -- start start 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.283+0000 7fa1e773e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e006aed0 0x7fa1e01ee0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.283+0000 7fa1e773e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1e006baf0 0x7fa1e0145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.283+0000 7fa1e773e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e01580b0 0x7fa1e014d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.283+0000 7fa1e773e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e00c4aa0 con 0x7fa1e006aed0 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e773e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e00c4d80 con 0x7fa1e01580b0 2024-03-16T19:30:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e773e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e00c5060 con 0x7fa1e006baf0 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e54da640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e006aed0 0x7fa1e01ee0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e54da640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e006aed0 0x7fa1e01ee0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39664/0 (socket says 172.21.15.172:39664) 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e54da640 1 -- 172.21.15.172:0/3462727729 learned_addr learned my addr 172.21.15.172:0/3462727729 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e4cd9640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1e006baf0 0x7fa1e0145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e5cdb640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e01580b0 0x7fa1e014d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e54da640 1 -- 172.21.15.172:0/3462727729 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1e006baf0 msgr2=0x7fa1e0145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e54da640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1e006baf0 0x7fa1e0145c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e54da640 1 -- 172.21.15.172:0/3462727729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e01580b0 msgr2=0x7fa1e014d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e54da640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e01580b0 0x7fa1e014d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e54da640 1 -- 172.21.15.172:0/3462727729 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d8009a70 con 0x7fa1e006aed0 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e54da640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e006aed0 0x7fa1e01ee0b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c4b0 tx=0x7fa1dc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1d5ffb640 1 -- 172.21.15.172:0/3462727729 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc005480 con 0x7fa1e006aed0 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1d5ffb640 1 -- 172.21.15.172:0/3462727729 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc0035f0 con 0x7fa1e006aed0 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e00c5340 con 0x7fa1e006aed0 2024-03-16T19:30:11.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1d5ffb640 1 -- 172.21.15.172:0/3462727729 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0039e0 con 0x7fa1e006aed0 2024-03-16T19:30:11.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.284+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e00c5900 con 0x7fa1e006aed0 2024-03-16T19:30:11.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.285+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e00c5b20 con 0x7fa1e006aed0 2024-03-16T19:30:11.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.286+0000 7fa1d5ffb640 1 -- 172.21.15.172:0/3462727729 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa1dc003b40 con 0x7fa1e006aed0 2024-03-16T19:30:11.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.286+0000 7fa1d5ffb640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1c406b3b0 0x7fa1c406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.286+0000 7fa1e4cd9640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1c406b3b0 0x7fa1c406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.286+0000 7fa1d5ffb640 1 -- 172.21.15.172:0/3462727729 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fa1dc0892c0 con 0x7fa1e006aed0 2024-03-16T19:30:11.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.287+0000 7fa1e4cd9640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1c406b3b0 0x7fa1c406d870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00c8720 tx=0x7fa1d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.291+0000 7fa1d5ffb640 1 -- 172.21.15.172:0/3462727729 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1e00c5b20 con 0x7fa1e006aed0 2024-03-16T19:30:11.337 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:11.337 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:11.338 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:11.338 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:11.349 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:11.392 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:11.411+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:11.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.473+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "3.d"} v 0) v1 -- 0x7fa1e00c5b20 con 0x7fa1e006aed0 2024-03-16T19:30:11.455 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:11.456 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:11.461 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:11.480+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:11.464 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:11.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.484+0000 7fa1d5ffb640 1 -- 172.21.15.172:0/3462727729 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "3.d"}]=0 clear 3.d pg_upmap_items mapping v34) v1 ==== 116+0+0 (secure 0 0 0) 0x7fa1e00c5b20 con 0x7fa1e006aed0 2024-03-16T19:30:11.465 INFO:teuthology.orchestra.run.smithi172.stderr:clear 3.d pg_upmap_items mapping 2024-03-16T19:30:11.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1c406b3b0 msgr2=0x7fa1c406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1c406b3b0 0x7fa1c406d870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00c8720 tx=0x7fa1d0009040 comp rx=0 tx=0).stop 2024-03-16T19:30:11.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e006aed0 msgr2=0x7fa1e01ee0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e006aed0 0x7fa1e01ee0b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c4b0 tx=0x7fa1dc00c970 comp rx=0 tx=0).stop 2024-03-16T19:30:11.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 shutdown_connections 2024-03-16T19:30:11.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e01580b0 0x7fa1e014d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1c406b3b0 0x7fa1c406d870 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1e006baf0 0x7fa1e0145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 --2- 172.21.15.172:0/3462727729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1e006aed0 0x7fa1e01ee0b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 >> 172.21.15.172:0/3462727729 conn(0x7fa1e00cc030 msgr2=0x7fa1e00c2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:11.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 shutdown_connections 2024-03-16T19:30:11.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.486+0000 7fa1e773e640 1 -- 172.21.15.172:0/3462727729 wait complete. 2024-03-16T19:30:11.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:11.580 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:11.600+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:11.613 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:11.613 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 55.402196707000002, 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.614 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:30:11.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.619 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 55.025501159999997, 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:30:11.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.623 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.249524716000003, 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.624 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.249440303, 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:30:11.629 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:30:11.634 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.249432947000003, 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:30:11.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:11.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:30:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:30:11.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:30:11.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.056893754999997, 2024-03-16T19:30:11.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:30:11.645 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.056887948000004, 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.646 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:30:11.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.056769815000003, 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:30:11.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:30:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.056674618000002, 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.657 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.662 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.056670607000001, 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:30:11.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:30:11.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:30:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:30:11.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.637738732999999, 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.668 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:30:11.673 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.637689364000003, 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:30:11.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:11.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:30:11.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.679 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.637610531999997, 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:11.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:30:11.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.637607320999997, 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.685 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:11.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:30:11.690 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.637604367999998, 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:30:11.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:11.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:30:11.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:11.696 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.637489393999999, 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:30:11.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:30:11.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:30:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.637486041999999, 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.702 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:30:11.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:30:11.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.637482925999997, 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:11.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:30:11.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:30:11.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:30:11.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.637422457, 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.713 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:11.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:30:11.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:30:11.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:30:11.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.028742407000003, 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:11.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:11.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:11.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:11.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:30:11.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:30:11.724 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.724 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.724 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.724 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:11.724 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:11.724 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:11.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:11.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.654+0000 7f031652c640 1 Processor -- start 2024-03-16T19:30:11.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.654+0000 7f031652c640 1 -- start start 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.654+0000 7f031652c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f031006d860 0x7f031006cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.654+0000 7f031652c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d180 0x7f031015c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.654+0000 7f031652c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0310070f10 con 0x7f0310152920 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.654+0000 7f031652c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03100713c0 con 0x7f031006d860 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.654+0000 7f031652c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03100c89b0 con 0x7f031006d180 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f030f7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f031006d860 0x7f031006cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f030f7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f031006d860 0x7f031006cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:55952/0 (socket says 172.21.15.172:55952) 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f030f7fe640 1 -- 172.21.15.172:0/4061900461 learned_addr learned my addr 172.21.15.172:0/4061900461 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f030ffff640 1 --1- 172.21.15.172:0/4061900461 >> v1:172.21.15.174:6789/0 conn(0x7f0310152920 0x7f03101591c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:52482/0 (socket says 172.21.15.172:52482) 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f0314ac9640 1 --2- 172.21.15.172:0/4061900461 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d180 0x7f031015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f030f7fe640 1 -- 172.21.15.172:0/4061900461 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d180 msgr2=0x7f031015c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f030f7fe640 1 --2- 172.21.15.172:0/4061900461 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d180 0x7f031015c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f030f7fe640 1 -- 172.21.15.172:0/4061900461 >> v1:172.21.15.174:6789/0 conn(0x7f0310152920 legacy=0x7f03101591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f030effd640 1 -- 172.21.15.172:0/4061900461 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485487520 0 0) 0x7f0310070f10 con 0x7f0310152920 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.655+0000 7f030f7fe640 1 -- 172.21.15.172:0/4061900461 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031015beb0 con 0x7f031006d860 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f030f7fe640 1 --2- 172.21.15.172:0/4061900461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f031006d860 0x7f031006cc40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f03000098a0 tx=0x7f030002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cadf154ecc6f2b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f030effd640 1 -- 172.21.15.172:0/4061900461 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300030840 con 0x7f031006d860 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f030effd640 1 -- 172.21.15.172:0/4061900461 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03000309a0 con 0x7f031006d860 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f030effd640 1 -- 172.21.15.172:0/4061900461 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300030d30 con 0x7f031006d860 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f031652c640 1 -- 172.21.15.172:0/4061900461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f031006d860 msgr2=0x7f031006cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f031652c640 1 --2- 172.21.15.172:0/4061900461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f031006d860 0x7f031006cc40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f03000098a0 tx=0x7f030002b980 comp rx=0 tx=0).stop 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f031652c640 1 -- 172.21.15.172:0/4061900461 shutdown_connections 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f031652c640 1 --2- 172.21.15.172:0/4061900461 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d180 0x7f031015c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f031652c640 1 --2- 172.21.15.172:0/4061900461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f031006d860 0x7f031006cc40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.656+0000 7f031652c640 1 -- 172.21.15.172:0/4061900461 >> 172.21.15.172:0/4061900461 conn(0x7f03100c8b30 msgr2=0x7f03100c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.657+0000 7f031652c640 1 -- 172.21.15.172:0/4061900461 shutdown_connections 2024-03-16T19:30:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.657+0000 7f031652c640 1 -- 172.21.15.172:0/4061900461 wait complete. 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.658+0000 7f031652c640 1 Processor -- start 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.658+0000 7f031652c640 1 -- start start 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.658+0000 7f031652c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f031006d180 0x7f03100cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.658+0000 7f031652c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d860 0x7f03101eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.658+0000 7f031652c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0310152920 0x7f031014eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.658+0000 7f031652c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03100cbc60 con 0x7f031006d180 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.658+0000 7f031652c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031014dbb0 con 0x7f0310152920 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.658+0000 7f031652c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03101ef010 con 0x7f031006d860 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f031006d180 0x7f03100cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d860 0x7f03101eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f0314ac9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0310152920 0x7f031014eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f031006d180 0x7f03100cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39678/0 (socket says 172.21.15.172:39678) 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030ffff640 1 -- 172.21.15.172:0/529140955 learned_addr learned my addr 172.21.15.172:0/529140955 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:11.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030f7fe640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0310152920 msgr2=0x7f031014eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030f7fe640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0310152920 0x7f031014eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030f7fe640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f031006d180 msgr2=0x7f03100cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030f7fe640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f031006d180 0x7f03100cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030f7fe640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03000094d0 con 0x7f031006d860 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f0314ac9640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0310152920 0x7f031014eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030f7fe640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d860 0x7f03101eead0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0300033000 tx=0x7f0300004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300039c90 con 0x7f031006d860 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0300039df0 con 0x7f031006d860 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300043620 con 0x7f031006d860 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03101e2880 con 0x7f031006d860 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.659+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031014e600 con 0x7f031006d860 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.660+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0310072b50 con 0x7f031006d860 2024-03-16T19:30:11.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.661+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0300043780 con 0x7f031006d860 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.661+0000 7f030cff9640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f02fc06b3b0 0x7f02fc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.661+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f030003e070 con 0x7f031006d860 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.661+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f02fc06b3b0 0x7f02fc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.661+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v34) v1 ==== 54+0+16 (secure 0 0 0) 0x7f03000b1240 con 0x7f031006d860 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.661+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f031014dd90 0x7f0310150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.661+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03100c86d0 con 0x7f031014dd90 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.662+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f02fc06b3b0 0x7f02fc06d870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0304000f60 tx=0x7f0304007ea0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.662+0000 7f0314ac9640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f031014dd90 0x7f0310150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.662+0000 7f0314ac9640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f031014dd90 0x7f0310150c30 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.662+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f030800f040 con 0x7f031014dd90 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.677+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f03100681c0 con 0x7f031014dd90 2024-03-16T19:30:11.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.679+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f03100681c0 con 0x7f031014dd90 2024-03-16T19:30:11.736 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:11.736 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:11.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.679+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 0x7f03101e2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.679+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03101e3f50 con 0x7f03101e2440 2024-03-16T19:30:11.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.679+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 0x7f03101e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.679+0000 7f030ffff640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 msgr2=0x7f03101e2d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 22 2024-03-16T19:30:11.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.679+0000 7f030ffff640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 msgr2=0x7f03101e2d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:30:11.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.679+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 0x7f03101e2d10 crc :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:30:11.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.679+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 0x7f03101e2d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:30:11.737 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:11.745+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:11.826 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:11.846+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:11.844 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:11.844 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:11.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:11.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.880+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 0x7f03101e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.880+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 0x7f03101e2d10 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.881+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f03101e3f50 con 0x7f03101e2440 2024-03-16T19:30:11.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.896+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f03100c67c0 con 0x7f03101e2440 2024-03-16T19:30:11.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.898+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f03100c67c0 con 0x7f03101e2440 2024-03-16T19:30:11.878 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:11.879 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:11.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.898+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f03101e33c0 0x7f03101f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.898+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03101f2010 con 0x7f03101e33c0 2024-03-16T19:30:11.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.898+0000 7f030f7fe640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f03101e33c0 0x7f03101f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.898+0000 7f030f7fe640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f03101e33c0 0x7f03101f1ad0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.899+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f03101f2010 con 0x7f03101e33c0 2024-03-16T19:30:11.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.914+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0310061930 con 0x7f03101e33c0 2024-03-16T19:30:11.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.915+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0310061930 con 0x7f03101e33c0 2024-03-16T19:30:11.896 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:11.896 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:11.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.915+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f03100623d0 0x7f0310158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.915+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03101f2460 con 0x7f03100623d0 2024-03-16T19:30:11.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.916+0000 7f0314ac9640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f03100623d0 0x7f0310158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.916+0000 7f0314ac9640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f03100623d0 0x7f0310158060 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.916+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f03101f2460 con 0x7f03100623d0 2024-03-16T19:30:11.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.931+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f03101585a0 con 0x7f03100623d0 2024-03-16T19:30:11.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.932+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f03101585a0 con 0x7f03100623d0 2024-03-16T19:30:11.913 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:11.914 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:11.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.932+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f0310158c80 0x7f031002bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.932+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0310204050 con 0x7f0310158c80 2024-03-16T19:30:11.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.933+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f0310158c80 0x7f031002bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.933+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f0310158c80 0x7f031002bde0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.934+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0310204050 con 0x7f0310158c80 2024-03-16T19:30:11.927 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:11.946+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:11.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.948+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f031002c320 con 0x7f0310158c80 2024-03-16T19:30:11.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.951+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f031002c320 con 0x7f0310158c80 2024-03-16T19:30:11.932 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:11.933 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:11.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.951+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f031002ca00 0x7f031002ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.951+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f031002d380 con 0x7f031002ca00 2024-03-16T19:30:11.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.951+0000 7f030f7fe640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f031002ca00 0x7f031002ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.952+0000 7f030f7fe640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f031002ca00 0x7f031002ce40 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.952+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f031002d380 con 0x7f031002ca00 2024-03-16T19:30:11.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.982+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f031002d380 con 0x7f031002ca00 2024-03-16T19:30:11.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.984+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f031002d380 con 0x7f031002ca00 2024-03-16T19:30:11.965 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:11.965 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:11.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.984+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0310204e60 0x7f0310207250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.984+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f031020a0a0 con 0x7f0310204e60 2024-03-16T19:30:11.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.984+0000 7f0314ac9640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0310204e60 0x7f0310207250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.985+0000 7f0314ac9640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0310204e60 0x7f0310207250 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:11.985+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f031020a0a0 con 0x7f0310204e60 2024-03-16T19:30:11.980 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:11.980 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:11.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.000+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0310207810 con 0x7f0310204e60 2024-03-16T19:30:11.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.002+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0310207810 con 0x7f0310204e60 2024-03-16T19:30:11.982 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:11.983 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:11.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.002+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f0310207e90 0x7f03102082d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:11.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.002+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f031020d450 con 0x7f0310207e90 2024-03-16T19:30:11.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.002+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f0310207e90 0x7f03102082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:11.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.002+0000 7f030ffff640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f0310207e90 0x7f03102082d0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:11.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.003+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f031020d450 con 0x7f0310207e90 2024-03-16T19:30:11.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:11.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.017+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0310208920 con 0x7f0310207e90 2024-03-16T19:30:12.000 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:12.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f030cff9640 1 -- 172.21.15.172:0/529140955 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f0310208920 con 0x7f0310207e90 2024-03-16T19:30:12.001 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f031014dd90 msgr2=0x7f0310150c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f031014dd90 0x7f0310150c30 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 msgr2=0x7f03101e2d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f03101e2440 0x7f03101e2d10 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f03101e33c0 msgr2=0x7f03101f1ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f03101e33c0 0x7f03101f1ad0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f03100623d0 msgr2=0x7f0310158060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f03100623d0 0x7f0310158060 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f0310158c80 msgr2=0x7f031002bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f0310158c80 0x7f031002bde0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f031002ca00 msgr2=0x7f031002ce40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f0314ac9640 1 -- 172.21.15.172:0/529140955 reap_dead start 2024-03-16T19:30:12.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f031002ca00 0x7f031002ce40 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f0314ac9640 1 -- 172.21.15.172:0/529140955 reap_dead start 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f0314ac9640 1 -- 172.21.15.172:0/529140955 reap_dead start 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0310204e60 msgr2=0x7f0310207250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0310204e60 0x7f0310207250 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f0310207e90 msgr2=0x7f03102082d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.019+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f0310207e90 0x7f03102082d0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f02fc06b3b0 msgr2=0x7f02fc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f02fc06b3b0 0x7f02fc06d870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0304000f60 tx=0x7f0304007ea0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d860 msgr2=0x7f03101eead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d860 0x7f03101eead0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0300033000 tx=0x7f0300004730 comp rx=0 tx=0).stop 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 shutdown_connections 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f0310207e90 0x7f03102082d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f0310204e60 0x7f0310207250 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f02fc06b3b0 0x7f02fc06d870 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 --2- 172.21.15.172:0/529140955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f031006d860 0x7f03101eead0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 >> 172.21.15.172:0/529140955 conn(0x7f03100c8b30 msgr2=0x7f0310157130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:12.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 shutdown_connections 2024-03-16T19:30:12.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.020+0000 7f031652c640 1 -- 172.21.15.172:0/529140955 wait complete. 2024-03-16T19:30:12.052 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:12.072+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:12.104 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:12.104 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:12.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:12.153 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:12.173+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:12.230 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:12.248+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:12.260 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:12.260 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:12.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:12.331 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:12.351+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:12.376 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:12.376 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:12.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:12.469 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:12.489+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:12.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:30:12.533 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:12.552+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:12.536 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:12.536 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:12.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:12.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.604+0000 7f06ffe47640 1 Processor -- start 2024-03-16T19:30:12.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.604+0000 7f06ffe47640 1 -- start start 2024-03-16T19:30:12.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.605+0000 7f06ffe47640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f06f8154e50 0x7f06f80cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:12.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.605+0000 7f06ffe47640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f806b110 con 0x7f06f80caf40 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.605+0000 7f06ffe47640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f815b8d0 con 0x7f06f80b5f10 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.605+0000 7f06ffe47640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f815ad50 con 0x7f06f8154e50 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.605+0000 7f06fe3e4640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f06f80caf40 0x7f06f8165260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:57236/0 (socket says 172.21.15.172:57236) 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.605+0000 7f06fe3e4640 1 -- 172.21.15.172:0/1647535296 learned_addr learned my addr 172.21.15.172:0/1647535296 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.605+0000 7f06fdbe3640 1 --2- 172.21.15.172:0/1647535296 >> v2:172.21.15.174:3300/0 conn(0x7f06f8154e50 0x7f06f80cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.606+0000 7f06fcbe1640 1 -- 172.21.15.172:0/1647535296 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055722659 0 0) 0x7f06f806b110 con 0x7f06f80caf40 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.606+0000 7f06fcbe1640 1 -- 172.21.15.172:0/1647535296 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e0003520 con 0x7f06f80caf40 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.606+0000 7f06fdbe3640 1 -- 172.21.15.172:0/1647535296 >> v1:172.21.15.174:6789/0 conn(0x7f06f80b5f10 legacy=0x7f06f806b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.606+0000 7f06fcbe1640 1 -- 172.21.15.172:0/1647535296 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177152488 0 0) 0x7f06f815b8d0 con 0x7f06f80b5f10 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.606+0000 7f06fdbe3640 1 -- 172.21.15.172:0/1647535296 >> v1:172.21.15.172:6789/0 conn(0x7f06f80caf40 legacy=0x7f06f8165260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.606+0000 7f06fdbe3640 1 -- 172.21.15.172:0/1647535296 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f815b030 con 0x7f06f8154e50 2024-03-16T19:30:12.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.607+0000 7f06fdbe3640 1 --2- 172.21.15.172:0/1647535296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f8154e50 0x7f06f80cbc30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f06f00097a0 tx=0x7f06f002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64e3bb07b28a9666 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.607+0000 7f06fcbe1640 1 -- 172.21.15.172:0/1647535296 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f002f810 con 0x7f06f8154e50 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.607+0000 7f06fcbe1640 1 -- 172.21.15.172:0/1647535296 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f002f970 con 0x7f06f8154e50 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.607+0000 7f06fcbe1640 1 -- 172.21.15.172:0/1647535296 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f002fd00 con 0x7f06f8154e50 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.607+0000 7f06ffe47640 1 -- 172.21.15.172:0/1647535296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f8154e50 msgr2=0x7f06f80cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.607+0000 7f06ffe47640 1 --2- 172.21.15.172:0/1647535296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f8154e50 0x7f06f80cbc30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f06f00097a0 tx=0x7f06f002b9f0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.607+0000 7f06ffe47640 1 -- 172.21.15.172:0/1647535296 shutdown_connections 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.607+0000 7f06ffe47640 1 --2- 172.21.15.172:0/1647535296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f8154e50 0x7f06f80cbc30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.607+0000 7f06ffe47640 1 -- 172.21.15.172:0/1647535296 >> 172.21.15.172:0/1647535296 conn(0x7f06f8066c20 msgr2=0x7f06f8067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.608+0000 7f06ffe47640 1 -- 172.21.15.172:0/1647535296 shutdown_connections 2024-03-16T19:30:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.608+0000 7f06ffe47640 1 -- 172.21.15.172:0/1647535296 wait complete. 2024-03-16T19:30:12.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06ffe47640 1 Processor -- start 2024-03-16T19:30:12.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06ffe47640 1 -- start start 2024-03-16T19:30:12.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06ffe47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f80b5f10 0x7f06f81f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:12.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06ffe47640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f06f80caf40 0x7f06f8159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:12.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06ffe47640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06f8154e50 0x7f06f80c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:12.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06ffe47640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80c58c0 con 0x7f06f8154e50 2024-03-16T19:30:12.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06ffe47640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f81589d0 con 0x7f06f80b5f10 2024-03-16T19:30:12.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06ffe47640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8158cb0 con 0x7f06f80caf40 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06fe3e4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06f8154e50 0x7f06f80c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06fdbe3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f80b5f10 0x7f06f81f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06fe3e4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06f8154e50 0x7f06f80c5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39680/0 (socket says 172.21.15.172:39680) 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.609+0000 7f06fe3e4640 1 -- 172.21.15.172:0/3145884700 learned_addr learned my addr 172.21.15.172:0/3145884700 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06fd3e2640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f06f80caf40 0x7f06f8159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06fe3e4640 1 -- 172.21.15.172:0/3145884700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f06f80caf40 msgr2=0x7f06f8159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06fe3e4640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f06f80caf40 0x7f06f8159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06fe3e4640 1 -- 172.21.15.172:0/3145884700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f80b5f10 msgr2=0x7f06f81f1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06fe3e4640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f80b5f10 0x7f06f81f1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06fe3e4640 1 -- 172.21.15.172:0/3145884700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f0009400 con 0x7f06f8154e50 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06fdbe3640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f80b5f10 0x7f06f81f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:12.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06fe3e4640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06f8154e50 0x7f06f80c5e10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f06e800c430 tx=0x7f06e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:12.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06f67fc640 1 -- 172.21.15.172:0/3145884700 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8005ce0 con 0x7f06f8154e50 2024-03-16T19:30:12.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06f67fc640 1 -- 172.21.15.172:0/3145884700 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e8005e40 con 0x7f06f8154e50 2024-03-16T19:30:12.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06f67fc640 1 -- 172.21.15.172:0/3145884700 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8007310 con 0x7f06f8154e50 2024-03-16T19:30:12.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f815ab20 con 0x7f06f8154e50 2024-03-16T19:30:12.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.610+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f8159270 con 0x7f06f8154e50 2024-03-16T19:30:12.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.611+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f80c9ab0 con 0x7f06f8154e50 2024-03-16T19:30:12.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.612+0000 7f06f67fc640 1 -- 172.21.15.172:0/3145884700 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f06e8007470 con 0x7f06f8154e50 2024-03-16T19:30:12.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.612+0000 7f06f67fc640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f06e406b3b0 0x7f06e406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:12.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.612+0000 7f06f67fc640 1 -- 172.21.15.172:0/3145884700 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f06e8089340 con 0x7f06f8154e50 2024-03-16T19:30:12.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.612+0000 7f06fdbe3640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f06e406b3b0 0x7f06e406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:12.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.613+0000 7f06fdbe3640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f06e406b3b0 0x7f06e406d870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f06f0036000 tx=0x7f06f0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:12.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.617+0000 7f06f67fc640 1 -- 172.21.15.172:0/3145884700 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06e8053f40 con 0x7f06f8154e50 2024-03-16T19:30:12.634 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:12.654+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:12.656 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:12.656 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:12.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:12.735 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:12.754+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:12.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:12.805+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f06f815a960 con 0x7f06f8154e50 2024-03-16T19:30:12.812 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:12.812 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:12.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:12.836 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:12.855+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:12.926 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:12.926 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:12.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:12.937 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:12.956+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:13.037 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:13.056+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:13.084 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:13.084 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:13.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:13.138 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:13.158+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:13.200 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:13.200 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:13.209 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:13.250 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:13.270+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:13.340 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:13.359+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:13.358 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:13.358 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:13.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:13.441 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:13.461+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:13.472 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:13.472 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:13.481 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:13.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.506+0000 7f06f67fc640 1 -- 172.21.15.172:0/3145884700 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v35) v1 ==== 91+0+0 (secure 0 0 0) 0x7f06e8057600 con 0x7f06f8154e50 2024-03-16T19:30:13.487 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:30:13.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f06e406b3b0 msgr2=0x7f06e406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:13.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f06e406b3b0 0x7f06e406d870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f06f0036000 tx=0x7f06f0008030 comp rx=0 tx=0).stop 2024-03-16T19:30:13.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06f8154e50 msgr2=0x7f06f80c5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:13.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06f8154e50 0x7f06f80c5e10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f06e800c430 tx=0x7f06e800c8f0 comp rx=0 tx=0).stop 2024-03-16T19:30:13.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 shutdown_connections 2024-03-16T19:30:13.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06f8154e50 0x7f06f80c5e10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:13.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f06e406b3b0 0x7f06e406d870 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:13.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f06f80caf40 0x7f06f8159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:13.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 --2- 172.21.15.172:0/3145884700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06f80b5f10 0x7f06f81f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:13.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 >> 172.21.15.172:0/3145884700 conn(0x7f06f8066c20 msgr2=0x7f06f80ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:13.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 shutdown_connections 2024-03-16T19:30:13.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:13.509+0000 7f06ffe47640 1 -- 172.21.15.172:0/3145884700 wait complete. 2024-03-16T19:30:13.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:13.561+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:13.590 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:13.590 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:13.598 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:13.644 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:13.663+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:13.744 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:13.763+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:13.745 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:13.745 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:13.754 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:13.845 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:13.864+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:13.901 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:13.901 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:13.910 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:13.946 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:13.966+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:14.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:14.056 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:14.076+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:14.057 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:14.057 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:14.066 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:14.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.131+0000 7fcbf80c3640 1 Processor -- start 2024-03-16T19:30:14.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.131+0000 7fcbf80c3640 1 -- start start 2024-03-16T19:30:14.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf80c3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcbf00ce4a0 0x7fcbf00ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf80c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 0x7fcbf00c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf80c3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf00c32c0 con 0x7fcbf006f420 2024-03-16T19:30:14.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf80c3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf00c9cb0 con 0x7fcbf00ce4a0 2024-03-16T19:30:14.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf80c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01593c0 con 0x7fcbf00ce880 2024-03-16T19:30:14.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf565e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 0x7fcbf00c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf5e5f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcbf00ce4a0 0x7fcbf00ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf5e5f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcbf00ce4a0 0x7fcbf00ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:55988/0 (socket says 172.21.15.172:55988) 2024-03-16T19:30:14.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf565e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 0x7fcbf00c3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:43946/0 (socket says 172.21.15.172:43946) 2024-03-16T19:30:14.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf5e5f640 1 -- 172.21.15.172:0/1938787648 learned_addr learned my addr 172.21.15.172:0/1938787648 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:14.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.132+0000 7fcbf5e5f640 1 -- 172.21.15.172:0/1938787648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 msgr2=0x7fcbf00c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:14.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.133+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/1938787648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 0x7fcbf00c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.133+0000 7fcbf5e5f640 1 -- 172.21.15.172:0/1938787648 >> v1:172.21.15.174:6789/0 conn(0x7fcbf006f420 legacy=0x7fcbf01588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.133+0000 7fcbf5e5f640 1 -- 172.21.15.172:0/1938787648 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf01596a0 con 0x7fcbf00ce4a0 2024-03-16T19:30:14.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.133+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/1938787648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbf00ce4a0 0x7fcbf00ca910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80098a0 tx=0x7fcbe802ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd41c409dc57e08 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.133+0000 7fcbf4e5d640 1 -- 172.21.15.172:0/1938787648 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe802f830 con 0x7fcbf00ce4a0 2024-03-16T19:30:14.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.133+0000 7fcbf4e5d640 1 -- 172.21.15.172:0/1938787648 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe802f990 con 0x7fcbf00ce4a0 2024-03-16T19:30:14.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.133+0000 7fcbf4e5d640 1 -- 172.21.15.172:0/1938787648 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe802fd20 con 0x7fcbf00ce4a0 2024-03-16T19:30:14.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.134+0000 7fcbf80c3640 1 -- 172.21.15.172:0/1938787648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbf00ce4a0 msgr2=0x7fcbf00ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.134+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/1938787648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbf00ce4a0 0x7fcbf00ca910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80098a0 tx=0x7fcbe802ba10 comp rx=0 tx=0).stop 2024-03-16T19:30:14.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.134+0000 7fcbf80c3640 1 -- 172.21.15.172:0/1938787648 shutdown_connections 2024-03-16T19:30:14.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.134+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/1938787648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 0x7fcbf00c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.134+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/1938787648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbf00ce4a0 0x7fcbf00ca910 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.134+0000 7fcbf80c3640 1 -- 172.21.15.172:0/1938787648 >> 172.21.15.172:0/1938787648 conn(0x7fcbf0066c20 msgr2=0x7fcbf0067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:14.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.134+0000 7fcbf80c3640 1 -- 172.21.15.172:0/1938787648 shutdown_connections 2024-03-16T19:30:14.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.134+0000 7fcbf80c3640 1 -- 172.21.15.172:0/1938787648 wait complete. 2024-03-16T19:30:14.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.135+0000 7fcbf80c3640 1 Processor -- start 2024-03-16T19:30:14.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.135+0000 7fcbf80c3640 1 -- start start 2024-03-16T19:30:14.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf80c3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf006f420 0x7fcbf015b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf80c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbf00ce4a0 0x7fcbf00c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf80c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 0x7fcbf0159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf80c3640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf015c6a0 con 0x7fcbf006f420 2024-03-16T19:30:14.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf80c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00c7850 con 0x7fcbf00ce4a0 2024-03-16T19:30:14.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf80c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf015a5a0 con 0x7fcbf00ce880 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf5e5f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf006f420 0x7fcbf015b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf6660640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 0x7fcbf0159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf5e5f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf006f420 0x7fcbf015b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39696/0 (socket says 172.21.15.172:39696) 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf5e5f640 1 -- 172.21.15.172:0/4082707304 learned_addr learned my addr 172.21.15.172:0/4082707304 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf565e640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbf00ce4a0 0x7fcbf00c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf5e5f640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 msgr2=0x7fcbf0159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf00ce880 0x7fcbf0159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf5e5f640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbf00ce4a0 msgr2=0x7fcbf00c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbf00ce4a0 0x7fcbf00c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf5e5f640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80094d0 con 0x7fcbf006f420 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.136+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf006f420 0x7fcbf015b640 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8000c00 tx=0x7fcbe8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.137+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8037c40 con 0x7fcbf006f420 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.137+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe8037da0 con 0x7fcbf006f420 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.137+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8041640 con 0x7fcbf006f420 2024-03-16T19:30:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.137+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf015a880 con 0x7fcbf006f420 2024-03-16T19:30:14.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.137+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf015ae40 con 0x7fcbf006f420 2024-03-16T19:30:14.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.137+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fcbf00681c0 con 0x7fcbf006f420 2024-03-16T19:30:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.138+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fcbe80417a0 con 0x7fcbf006f420 2024-03-16T19:30:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.138+0000 7fcbee7fc640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbdc06b3e0 0x7fcbdc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.139+0000 7fcbf565e640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbdc06b3e0 0x7fcbdc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.139+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fcbe803c070 con 0x7fcbf006f420 2024-03-16T19:30:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.139+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v35) v1 ==== 54+0+16 (secure 0 0 0) 0x7fcbe80b3070 con 0x7fcbf006f420 2024-03-16T19:30:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.139+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fcbf015b060 0x7fcbf0140bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.139+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbf00c37f0 con 0x7fcbf015b060 2024-03-16T19:30:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.139+0000 7fcbf565e640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbdc06b3e0 0x7fcbdc06d8a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcbf015ce20 tx=0x7fcbe4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.139+0000 7fcbf6660640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fcbf015b060 0x7fcbf0140bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.139+0000 7fcbf6660640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fcbf015b060 0x7fcbf0140bc0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.140+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcbe0001a80 con 0x7fcbf015b060 2024-03-16T19:30:14.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.154+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbf0158fb0 con 0x7fcbf015b060 2024-03-16T19:30:14.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.156+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcbe0001f10 con 0x7fcbf015b060 2024-03-16T19:30:14.137 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:14.137 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:14.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.156+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fcbf0157d20 0x7fcbf01f10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.156+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbf0141610 con 0x7fcbf0157d20 2024-03-16T19:30:14.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.156+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fcbf0157d20 0x7fcbf01f10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.156+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fcbf0157d20 0x7fcbf01f10f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.157+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcbf0141610 con 0x7fcbf0157d20 2024-03-16T19:30:14.148 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:14.167+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:14.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.171+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbf00c9570 con 0x7fcbf0157d20 2024-03-16T19:30:14.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.173+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcbf00c9570 con 0x7fcbf0157d20 2024-03-16T19:30:14.154 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:14.155 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:14.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.173+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fcbf01f1b40 0x7fcbf0061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.173+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbf01f1f20 con 0x7fcbf01f1b40 2024-03-16T19:30:14.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.173+0000 7fcbf565e640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fcbf01f1b40 0x7fcbf0061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.174+0000 7fcbf565e640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fcbf01f1b40 0x7fcbf0061930 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.174+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcbf01f1f20 con 0x7fcbf01f1b40 2024-03-16T19:30:14.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.188+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbf0061e70 con 0x7fcbf01f1b40 2024-03-16T19:30:14.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.190+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcbf0061e70 con 0x7fcbf01f1b40 2024-03-16T19:30:14.170 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:14.171 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:14.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.190+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fcbf00b1810 0x7fcbf00b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.190+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbf0062910 con 0x7fcbf00b1810 2024-03-16T19:30:14.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.190+0000 7fcbf6660640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fcbf00b1810 0x7fcbf00b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.190+0000 7fcbf6660640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fcbf00b1810 0x7fcbf00b1c30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.191+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcbf0062910 con 0x7fcbf00b1810 2024-03-16T19:30:14.173 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:14.173 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:14.182 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:14.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.204+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbf00b23c0 con 0x7fcbf00b1810 2024-03-16T19:30:14.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.206+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcbf00b23c0 con 0x7fcbf00b1810 2024-03-16T19:30:14.187 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:14.187 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:14.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.206+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fcbf00b2590 0x7fcbf002c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.206+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbf002c830 con 0x7fcbf00b2590 2024-03-16T19:30:14.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.206+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fcbf00b2590 0x7fcbf002c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.207+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fcbf00b2590 0x7fcbf002c2f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.207+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcbf002c830 con 0x7fcbf00b2590 2024-03-16T19:30:14.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.221+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbf002c830 con 0x7fcbf00b2590 2024-03-16T19:30:14.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.223+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcbf002c830 con 0x7fcbf00b2590 2024-03-16T19:30:14.203 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.223+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fcbf002ceb0 0x7fcbf0204480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.223+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbf002d310 con 0x7fcbf002ceb0 2024-03-16T19:30:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.223+0000 7fcbf565e640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fcbf002ceb0 0x7fcbf0204480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.223+0000 7fcbf565e640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fcbf002ceb0 0x7fcbf0204480 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.224+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcbf002d310 con 0x7fcbf002ceb0 2024-03-16T19:30:14.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.237+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbf002d310 con 0x7fcbf002ceb0 2024-03-16T19:30:14.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.239+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcbf002d310 con 0x7fcbf002ceb0 2024-03-16T19:30:14.220 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:14.220 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:14.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.240+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fcbf0204ed0 0x7fcbf0205310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.240+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbf020a430 con 0x7fcbf0204ed0 2024-03-16T19:30:14.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.240+0000 7fcbf6660640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fcbf0204ed0 0x7fcbf0205310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.240+0000 7fcbf6660640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fcbf0204ed0 0x7fcbf0205310 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.241+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcbf020a430 con 0x7fcbf0204ed0 2024-03-16T19:30:14.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.254+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbf0205960 con 0x7fcbf0204ed0 2024-03-16T19:30:14.236 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.256+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcbf0205960 con 0x7fcbf0204ed0 2024-03-16T19:30:14.237 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:14.238 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:14.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.256+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fcbf02061b0 0x7fcbf0206610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:14.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.256+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbf0206bd0 con 0x7fcbf02061b0 2024-03-16T19:30:14.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.257+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fcbf02061b0 0x7fcbf0206610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:14.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.257+0000 7fcbf5e5f640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fcbf02061b0 0x7fcbf0206610 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:14.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.258+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcbf0206bd0 con 0x7fcbf02061b0 2024-03-16T19:30:14.249 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:14.268+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:14.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.271+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbf0206e50 con 0x7fcbf02061b0 2024-03-16T19:30:14.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbee7fc640 1 -- 172.21.15.172:0/4082707304 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcbf0206e50 con 0x7fcbf02061b0 2024-03-16T19:30:14.254 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:14.255 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:14.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fcbf015b060 msgr2=0x7fcbf0140bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fcbf015b060 0x7fcbf0140bc0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fcbf0157d20 msgr2=0x7fcbf01f10f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fcbf0157d20 0x7fcbf01f10f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fcbf01f1b40 msgr2=0x7fcbf0061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fcbf01f1b40 0x7fcbf0061930 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fcbf00b1810 msgr2=0x7fcbf00b1c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fcbf00b1810 0x7fcbf00b1c30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fcbf00b2590 msgr2=0x7fcbf002c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fcbf00b2590 0x7fcbf002c2f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fcbf002ceb0 msgr2=0x7fcbf0204480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fcbf002ceb0 0x7fcbf0204480 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fcbf0204ed0 msgr2=0x7fcbf0205310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fcbf0204ed0 0x7fcbf0205310 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fcbf02061b0 msgr2=0x7fcbf0206610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf6660640 1 -- 172.21.15.172:0/4082707304 reap_dead start 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fcbf02061b0 0x7fcbf0206610 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.273+0000 7fcbf6660640 1 -- 172.21.15.172:0/4082707304 reap_dead start 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf6660640 1 -- 172.21.15.172:0/4082707304 reap_dead start 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf6660640 1 -- 172.21.15.172:0/4082707304 reap_dead start 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbdc06b3e0 msgr2=0x7fcbdc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbdc06b3e0 0x7fcbdc06d8a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcbf015ce20 tx=0x7fcbe4009190 comp rx=0 tx=0).stop 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf006f420 msgr2=0x7fcbf015b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf006f420 0x7fcbf015b640 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8000c00 tx=0x7fcbe8004730 comp rx=0 tx=0).stop 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 shutdown_connections 2024-03-16T19:30:14.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbdc06b3e0 0x7fcbdc06d8a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 --2- 172.21.15.172:0/4082707304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbf006f420 0x7fcbf015b640 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:14.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 >> 172.21.15.172:0/4082707304 conn(0x7fcbf0066c20 msgr2=0x7fcbf0067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:14.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 shutdown_connections 2024-03-16T19:30:14.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:14.274+0000 7fcbf80c3640 1 -- 172.21.15.172:0/4082707304 wait complete. 2024-03-16T19:30:14.330 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:14.330 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:14.330 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:14.330 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:14.339 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:14.350 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:14.370+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:14.446 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:14.446 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:14.446 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:14.447 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:14.447 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:14.451 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:14.471+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:14.455 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:14.552 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:14.571+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:14.563 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.224925484000003, 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:30:14.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:14.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:30:14.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:14.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:30:14.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:30:14.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.224901310999996, 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.570 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:30:14.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:30:14.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.575 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.224895484000001, 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:30:14.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.224891571999997, 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:30:14.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.22488559, 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.586 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.591 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.224881924000002, 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:30:14.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:30:14.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:30:14.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:30:14.596 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.224871264000001, 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:30:14.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:30:14.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.600 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.224867375000002, 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:14.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:14.604 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.221350684000001, 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:14.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:30:14.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:14.608 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.221280249000003, 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:14.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.612 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.220897901999997, 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:30:14.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:30:14.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:30:14.616 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.219131619000002, 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:14.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:30:14.620 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.217318697000003, 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:30:14.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:30:14.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:25 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.217080916999997, 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189085, 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365064+0000", 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2099999999999996e-07 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365068+0000", 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9700000000000001e-06 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0020999999999999e-05 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7449999999999998e-06 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365136+0000", 2024-03-16T19:30:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0973999999999997e-05 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365145+0000", 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9560000000000003e-06 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365253+0000", 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107798 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.216462948, 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:30:14.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:30:14.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:30:14.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.210602182000002, 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.634 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:30:14.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:30:14.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:14.638 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.778050974000003, 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:30:14.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:30:14.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.643 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:40 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.774770461999999, 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019577200000000001, 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:14.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807374+0000", 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1600000000000001e-07 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807377+0000", 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7839999999999999e-06 2024-03-16T19:30:14.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807384+0000", 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8410000000000002e-06 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807388+0000", 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0550000000000001e-06 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807431+0000", 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3271e-05 2024-03-16T19:30:14.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807441+0000", 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2320000000000007e-06 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807570+0000", 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000129073 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.647 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.210557268000002, 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:14.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:30:14.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.652 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 31.702945017000001, 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:14.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:14.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.657 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:14.658 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:14.658 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:14.659 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:14.660 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:14.678+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:14.760 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:14.778+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:14.817 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:14.817 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:14.817 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:14.817 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:14.828 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:14.861 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:14.881+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:14.962 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:14.980+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:14.978 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:14.978 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:14.978 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:14.978 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:14.979 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:14.987 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:15.071 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:15.090+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:15.134 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.923377881, 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:30:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.546682334000003, 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:30:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.770705890000002, 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:30:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:30:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.770621476999999, 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.770614121000001, 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:30:15.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.161 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.578074929000003, 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:30:15.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:30:15.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:30:15.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.167 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.578069122000002, 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:30:15.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:30:15.173 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.577950989000001, 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:30:15.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.577855792000001, 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:30:15.180 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:30:15.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.577851781, 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:30:15.186 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:15.192 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.158919906999998, 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:30:15.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.158870538000002, 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.198 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:30:15.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:30:15.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.158791706000002, 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.204 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:30:15.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:30:15.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.158788495000003, 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.209 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:15.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:30:15.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:30:15.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.158785541999997, 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.215 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:15.220 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.158670567999998, 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:30:15.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:30:15.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:30:15.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:30:15.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.158667215999998, 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:15.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:15.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:30:15.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:30:15.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.158664100000003, 2024-03-16T19:30:15.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:30:15.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:30:15.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:30:15.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:30:15.238 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.158603630999998, 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.239 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:30:15.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:30:15.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:30:15.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:30:15.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 56.549923581000002, 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:15.244 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:15.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:15.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:30:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.250 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.251 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.251 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:15.251 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:15.251 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:15.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:15.253 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:15.272+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:15.354 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:15.373+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:15.369 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:15.369 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:15.369 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:15.369 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:15.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:15.455 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:15.473+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:15.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:30:15.528 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:15.529 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:15.529 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:15.529 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:15.529 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:15.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:15.580 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:15.600+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:15.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.626+0000 7f9085f1e640 1 Processor -- start 2024-03-16T19:30:15.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.627+0000 7f9085f1e640 1 -- start start 2024-03-16T19:30:15.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.627+0000 7f9085f1e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9080157e50 0x7f90800ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.627+0000 7f9085f1e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9080158230 0x7f90800cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.627+0000 7f9085f1e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90800c9530 0x7f908006b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.627+0000 7f9085f1e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908006aed0 con 0x7f9080157e50 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.627+0000 7f9085f1e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90800719e0 con 0x7f9080158230 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.627+0000 7f9085f1e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90800c7b50 con 0x7f90800c9530 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.627+0000 7f907f7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9080157e50 0x7f90800ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.628+0000 7f907f7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9080157e50 0x7f90800ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39700/0 (socket says 172.21.15.172:39700) 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.628+0000 7f907f7fe640 1 -- 172.21.15.172:0/2870114633 learned_addr learned my addr 172.21.15.172:0/2870114633 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.628+0000 7f907ffff640 1 --2- 172.21.15.172:0/2870114633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90800c9530 0x7f908006b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.628+0000 7f907effd640 1 --2- 172.21.15.172:0/2870114633 >> v2:172.21.15.174:3300/0 conn(0x7f9080158230 0x7f90800cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.628+0000 7f907f7fe640 1 -- 172.21.15.172:0/2870114633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90800c9530 msgr2=0x7f908006b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.628+0000 7f907f7fe640 1 --2- 172.21.15.172:0/2870114633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90800c9530 0x7f908006b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.628+0000 7f907f7fe640 1 -- 172.21.15.172:0/2870114633 >> v2:172.21.15.174:3300/0 conn(0x7f9080158230 msgr2=0x7f90800cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.628+0000 7f907f7fe640 1 --2- 172.21.15.172:0/2870114633 >> v2:172.21.15.174:3300/0 conn(0x7f9080158230 0x7f90800cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:15.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.628+0000 7f907f7fe640 1 -- 172.21.15.172:0/2870114633 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90800c7e10 con 0x7f9080157e50 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f907f7fe640 1 --2- 172.21.15.172:0/2870114633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9080157e50 0x7f90800ccf60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f908006e000 tx=0x7f907002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560bfe9e82c4142d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f907e7fc640 1 -- 172.21.15.172:0/2870114633 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070031870 con 0x7f9080157e50 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f907e7fc640 1 -- 172.21.15.172:0/2870114633 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90700319d0 con 0x7f9080157e50 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f907e7fc640 1 -- 172.21.15.172:0/2870114633 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070031d60 con 0x7f9080157e50 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f9085f1e640 1 -- 172.21.15.172:0/2870114633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9080157e50 msgr2=0x7f90800ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f9085f1e640 1 --2- 172.21.15.172:0/2870114633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9080157e50 0x7f90800ccf60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f908006e000 tx=0x7f907002b910 comp rx=0 tx=0).stop 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f9085f1e640 1 -- 172.21.15.172:0/2870114633 shutdown_connections 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f9085f1e640 1 --2- 172.21.15.172:0/2870114633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90800c9530 0x7f908006b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f9085f1e640 1 --2- 172.21.15.172:0/2870114633 >> v2:172.21.15.174:3300/0 conn(0x7f9080158230 0x7f90800cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f9085f1e640 1 --2- 172.21.15.172:0/2870114633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9080157e50 0x7f90800ccf60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.629+0000 7f9085f1e640 1 -- 172.21.15.172:0/2870114633 >> 172.21.15.172:0/2870114633 conn(0x7f908006db70 msgr2=0x7f908006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.630+0000 7f9085f1e640 1 -- 172.21.15.172:0/2870114633 shutdown_connections 2024-03-16T19:30:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.630+0000 7f9085f1e640 1 -- 172.21.15.172:0/2870114633 wait complete. 2024-03-16T19:30:15.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.631+0000 7f9085f1e640 1 Processor -- start 2024-03-16T19:30:15.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.631+0000 7f9085f1e640 1 -- start start 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.631+0000 7f9085f1e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90800c9530 0x7f90801475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.631+0000 7f9085f1e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9080157e50 0x7f90801f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.631+0000 7f9085f1e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9080158230 0x7f9080153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.631+0000 7f9085f1e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90801f4450 con 0x7f90800c9530 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.631+0000 7f9085f1e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90800cce30 con 0x7f9080157e50 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.631+0000 7f9085f1e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908014dfa0 con 0x7f9080158230 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90800c9530 0x7f90801475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90800c9530 0x7f90801475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39704/0 (socket says 172.21.15.172:39704) 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907f7fe640 1 -- 172.21.15.172:0/470754880 learned_addr learned my addr 172.21.15.172:0/470754880 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907effd640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9080157e50 0x7f90801f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907ffff640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9080158230 0x7f9080153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907effd640 1 -- 172.21.15.172:0/470754880 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9080158230 msgr2=0x7f9080153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907effd640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9080158230 0x7f9080153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907effd640 1 -- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90800c9530 msgr2=0x7f90801475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907effd640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90800c9530 0x7f90801475d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907effd640 1 -- 172.21.15.172:0/470754880 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700094d0 con 0x7f9080157e50 2024-03-16T19:30:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907f7fe640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90800c9530 0x7f90801475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f907effd640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9080157e50 0x7f90801f0680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f907400e880 tx=0x7f907400ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f9067fff640 1 -- 172.21.15.172:0/470754880 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907400c940 con 0x7f9080157e50 2024-03-16T19:30:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f9067fff640 1 -- 172.21.15.172:0/470754880 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907400caa0 con 0x7f9080157e50 2024-03-16T19:30:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90801513a0 con 0x7f9080157e50 2024-03-16T19:30:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f9067fff640 1 -- 172.21.15.172:0/470754880 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907400ce50 con 0x7f9080157e50 2024-03-16T19:30:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.632+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908014d9e0 con 0x7f9080157e50 2024-03-16T19:30:15.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.634+0000 7f9067fff640 1 -- 172.21.15.172:0/470754880 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9074013470 con 0x7f9080157e50 2024-03-16T19:30:15.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.634+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90800c7530 con 0x7f9080157e50 2024-03-16T19:30:15.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.634+0000 7f9067fff640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f906c06b3b0 0x7f906c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:15.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.634+0000 7f9067fff640 1 -- 172.21.15.172:0/470754880 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f9074089d50 con 0x7f9080157e50 2024-03-16T19:30:15.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.634+0000 7f907f7fe640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f906c06b3b0 0x7f906c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:15.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.634+0000 7f907f7fe640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f906c06b3b0 0x7f906c06d870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f90700090b0 tx=0x7f907003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:15.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.639+0000 7f9067fff640 1 -- 172.21.15.172:0/470754880 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9074054a50 con 0x7f9080157e50 2024-03-16T19:30:15.656 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:15.676+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:15.702 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:15.702 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 59.127813244999999, 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:30:15.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:30:15.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.706 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.337587073999998, 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:30:15.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:30:15.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.336826838999997, 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.336012883999999, 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:30:15.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:30:15.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.335372958999997, 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:15.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.334714861999998, 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.334182286999997, 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.728 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:30:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:30:15.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.732 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.276010384000003, 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.27540819, 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:30:15.736 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:30:15.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.274765244999998, 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:30:15.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:30:15.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.274261283999998, 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.747 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.116115018000002, 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:15.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:30:15.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.115584882, 2024-03-16T19:30:15.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.363615623000001, 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:30:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:30:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.363591087000003, 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:30:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:30:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.363586556999998, 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:30:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.363576745000003, 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:30:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.363572892000001, 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.363569576000003, 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:30:15.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.362373992999999, 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:15.783 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:15.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:15.787 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:15.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:15.789 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:15.807+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:15.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:15.838+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f9080075a70 con 0x7f9080157e50 2024-03-16T19:30:15.889 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:15.908+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:15.926 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:15.926 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:15.926 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:15.926 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:15.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:15.989 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:16.008+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:16.082 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:16.082 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:16.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:16.091 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:16.109+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:16.191 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:16.210+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:16.200 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.861705618000002, 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:30:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:30:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:30:16.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.861688504, 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:16.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.211 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.861681224000002, 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:16.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.861667357000002, 2024-03-16T19:30:16.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:30:16.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:30:16.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.860994521999999, 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:30:16.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:16.229 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.858696272000003, 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:30:16.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:30:16.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:30:16.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.858651170999998, 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:30:16.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.858623115999997, 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:30:16.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:30:16.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.858601702000001, 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.242 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:30:16.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:30:16.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.858579351000003, 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.246 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:30:16.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.858558952000003, 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.250 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:30:16.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:30:16.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.858452960000001, 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.254 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.856661248999998, 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.856613748000001, 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.856374496999997, 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:30:16.265 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:30:16.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:30 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.856218918000003, 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126666, 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362824+0000", 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7999999999999995e-07 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362828+0000", 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0879999999999997e-06 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362837+0000", 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.704e-06 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362843+0000", 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9390000000000002e-06 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362877+0000", 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4779e-05 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362883+0000", 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9009999999999999e-06 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362950+0000", 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6674999999999994e-05 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.847876352, 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:30:16.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:30:16.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:30:16.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.846536475999997, 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:16.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:30:16.281 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.282 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:16.282 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:16.282 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:16.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:16.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:16.292 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:16.310+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:16.392 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:16.411+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:16.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.444+0000 7f3416e5c640 1 Processor -- start 2024-03-16T19:30:16.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.444+0000 7f3416e5c640 1 -- start start 2024-03-16T19:30:16.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.444+0000 7f3416e5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34100cbe60 0x7f34100cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.444+0000 7f3416e5c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34100cb110 con 0x7f34100cb5a0 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.444+0000 7f3416e5c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341015e6f0 con 0x7f341006bcc0 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.444+0000 7f3416e5c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34100c97c0 con 0x7f34100cbe60 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f34153f9640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f34100cb5a0 0x7f34101651d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:57242/0 (socket says 172.21.15.172:57242) 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f34153f9640 1 -- 172.21.15.172:0/269333028 learned_addr learned my addr 172.21.15.172:0/269333028 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f3414bf8640 1 --2- 172.21.15.172:0/269333028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34100cbe60 0x7f34100cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f340f7fe640 1 -- 172.21.15.172:0/269333028 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150772806 0 0) 0x7f34100cb110 con 0x7f34100cb5a0 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f340f7fe640 1 -- 172.21.15.172:0/269333028 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3400003520 con 0x7f34100cb5a0 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f340f7fe640 1 -- 172.21.15.172:0/269333028 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783191885 0 0) 0x7f341015e6f0 con 0x7f341006bcc0 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f340f7fe640 1 -- 172.21.15.172:0/269333028 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34100cb110 con 0x7f341006bcc0 2024-03-16T19:30:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f3414bf8640 1 -- 172.21.15.172:0/269333028 >> v1:172.21.15.174:6789/0 conn(0x7f341006bcc0 legacy=0x7f341015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f3414bf8640 1 -- 172.21.15.172:0/269333028 >> v1:172.21.15.172:6789/0 conn(0x7f34100cb5a0 legacy=0x7f34101651d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.445+0000 7f3414bf8640 1 -- 172.21.15.172:0/269333028 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341015e9d0 con 0x7f34100cbe60 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.446+0000 7f3414bf8640 1 --2- 172.21.15.172:0/269333028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34100cbe60 0x7f34100cc240 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34040097d0 tx=0x7f340402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4cb5e65941aa49a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.446+0000 7f340f7fe640 1 -- 172.21.15.172:0/269333028 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404030880 con 0x7f34100cbe60 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.446+0000 7f340f7fe640 1 -- 172.21.15.172:0/269333028 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34040309e0 con 0x7f34100cbe60 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.446+0000 7f340f7fe640 1 -- 172.21.15.172:0/269333028 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404030d70 con 0x7f34100cbe60 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.446+0000 7f3416e5c640 1 -- 172.21.15.172:0/269333028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34100cbe60 msgr2=0x7f34100cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.446+0000 7f3416e5c640 1 --2- 172.21.15.172:0/269333028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34100cbe60 0x7f34100cc240 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34040097d0 tx=0x7f340402b9c0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.446+0000 7f3416e5c640 1 -- 172.21.15.172:0/269333028 shutdown_connections 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.446+0000 7f3416e5c640 1 --2- 172.21.15.172:0/269333028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34100cbe60 0x7f34100cc240 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.446+0000 7f3416e5c640 1 -- 172.21.15.172:0/269333028 >> 172.21.15.172:0/269333028 conn(0x7f34100d0140 msgr2=0x7f34100d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.447+0000 7f3416e5c640 1 -- 172.21.15.172:0/269333028 shutdown_connections 2024-03-16T19:30:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.447+0000 7f3416e5c640 1 -- 172.21.15.172:0/269333028 wait complete. 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.448+0000 7f3416e5c640 1 Processor -- start 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.448+0000 7f3416e5c640 1 -- start start 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3416e5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f341006bcc0 0x7f34101f11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3416e5c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34100cb5a0 0x7f34101502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3416e5c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34100cbe60 0x7f34100c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3416e5c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34101f16f0 con 0x7f34100cbe60 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3416e5c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341014f870 con 0x7f34100cb5a0 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3416e5c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341015acc0 con 0x7f341006bcc0 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f34153f9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34100cbe60 0x7f34100c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3414bf8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f341006bcc0 0x7f34101f11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3414bf8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f341006bcc0 0x7f34101f11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:43980/0 (socket says 172.21.15.172:43980) 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3414bf8640 1 -- 172.21.15.172:0/3434376131 learned_addr learned my addr 172.21.15.172:0/3434376131 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f34153f9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34100cbe60 0x7f34100c6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39714/0 (socket says 172.21.15.172:39714) 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f340ffff640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34100cb5a0 0x7f34101502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3414bf8640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34100cb5a0 msgr2=0x7f34101502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3414bf8640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34100cb5a0 0x7f34101502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3414bf8640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34100cbe60 msgr2=0x7f34100c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3414bf8640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34100cbe60 0x7f34100c6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3414bf8640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3404009400 con 0x7f341006bcc0 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34100cbe60 0x7f34100c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3414bf8640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f341006bcc0 0x7f34101f11b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f34040098f0 tx=0x7f34040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340403e070 con 0x7f341006bcc0 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3404039a10 con 0x7f341006bcc0 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341015b6a0 con 0x7f341006bcc0 2024-03-16T19:30:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404039dc0 con 0x7f341006bcc0 2024-03-16T19:30:16.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.449+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341015b240 con 0x7f341006bcc0 2024-03-16T19:30:16.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.450+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f341015b430 con 0x7f341006bcc0 2024-03-16T19:30:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.451+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f34040423f0 con 0x7f341006bcc0 2024-03-16T19:30:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.451+0000 7f340d7fa640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f33f406b3b0 0x7f33f406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.451+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f34040abcc0 con 0x7f341006bcc0 2024-03-16T19:30:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.451+0000 7f340ffff640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f33f406b3b0 0x7f33f406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.451+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v35) v1 ==== 54+0+16 (secure 0 0 0) 0x7f34040ae1c0 con 0x7f341006bcc0 2024-03-16T19:30:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.452+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 0x7f34101e43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.452+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34100cada0 con 0x7f34101533c0 2024-03-16T19:30:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.452+0000 7f340ffff640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f33f406b3b0 0x7f33f406d870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f33fc007fd0 tx=0x7f33fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.452+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 0x7f34101e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.452+0000 7f34153f9640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 msgr2=0x7f34101e43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-03-16T19:30:16.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.452+0000 7f34153f9640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 msgr2=0x7f34101e43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:30:16.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.452+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 0x7f34101e43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:30:16.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.452+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 0x7f34101e43b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:30:16.435 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:16.435 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:16.435 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:16.435 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:16.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:16.480 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:30:16.480 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:30:16.480 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-16T19:30:16.480 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-03-16T19:30:16.494 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:16.512+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:16.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.520+0000 7f9067fff640 1 -- 172.21.15.172:0/470754880 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v36) v1 ==== 101+0+0 (secure 0 0 0) 0x7f9074017090 con 0x7f9080157e50 2024-03-16T19:30:16.500 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:30:16.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f906c06b3b0 msgr2=0x7f906c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f906c06b3b0 0x7f906c06d870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f90700090b0 tx=0x7f907003c040 comp rx=0 tx=0).stop 2024-03-16T19:30:16.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9080157e50 msgr2=0x7f90801f0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9080157e50 0x7f90801f0680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f907400e880 tx=0x7f907400ed40 comp rx=0 tx=0).stop 2024-03-16T19:30:16.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 shutdown_connections 2024-03-16T19:30:16.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f906c06b3b0 0x7f906c06d870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9080158230 0x7f9080153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9080157e50 0x7f90801f0680 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 --2- 172.21.15.172:0/470754880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90800c9530 0x7f90801475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 >> 172.21.15.172:0/470754880 conn(0x7f908006db70 msgr2=0x7f908006f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:16.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.522+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 shutdown_connections 2024-03-16T19:30:16.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.523+0000 7f9085f1e640 1 -- 172.21.15.172:0/470754880 wait complete. 2024-03-16T19:30:16.554 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:16.554 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:16.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:16.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.611+0000 7f6546415640 1 Processor -- start 2024-03-16T19:30:16.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.611+0000 7f6546415640 1 -- start start 2024-03-16T19:30:16.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f6546415640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f654015ab50 0x7f65400cb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f6546415640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400b3340 0x7f65400b3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f6546415640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654015afe0 con 0x7f65400ca450 2024-03-16T19:30:16.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f6546415640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654015bb40 con 0x7f654015ab50 2024-03-16T19:30:16.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f6546415640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654015b2a0 con 0x7f65400b3340 2024-03-16T19:30:16.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f653f7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f654015ab50 0x7f65400cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f65449b2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400b3340 0x7f65400b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f653ffff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f65400ca450 0x7f654015e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:52526/0 (socket says 172.21.15.172:52526) 2024-03-16T19:30:16.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f653ffff640 1 -- 172.21.15.172:0/1118743515 learned_addr learned my addr 172.21.15.172:0/1118743515 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:16.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f653f7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f654015ab50 0x7f65400cb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:56030/0 (socket says 172.21.15.172:56030) 2024-03-16T19:30:16.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.612+0000 7f65449b2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400b3340 0x7f65400b3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:43992/0 (socket says 172.21.15.172:43992) 2024-03-16T19:30:16.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.613+0000 7f653f7fe640 1 -- 172.21.15.172:0/1118743515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400b3340 msgr2=0x7f65400b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:16.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.613+0000 7f653f7fe640 1 --2- 172.21.15.172:0/1118743515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400b3340 0x7f65400b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.613+0000 7f653f7fe640 1 -- 172.21.15.172:0/1118743515 >> v1:172.21.15.174:6789/0 conn(0x7f65400ca450 legacy=0x7f654015e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.613+0000 7f653effd640 1 -- 172.21.15.172:0/1118743515 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890340041 0 0) 0x7f654015afe0 con 0x7f65400ca450 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.613+0000 7f653f7fe640 1 -- 172.21.15.172:0/1118743515 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654015b580 con 0x7f654015ab50 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.613+0000 7f65449b2640 1 --2- 172.21.15.172:0/1118743515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400b3340 0x7f65400b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.613+0000 7f653f7fe640 1 --2- 172.21.15.172:0/1118743515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f654015ab50 0x7f65400cb330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f65340097a0 tx=0x7f653402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bcf28254b6e9295 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.614+0000 7f653effd640 1 -- 172.21.15.172:0/1118743515 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6534030610 con 0x7f654015ab50 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.614+0000 7f653effd640 1 -- 172.21.15.172:0/1118743515 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6534030770 con 0x7f654015ab50 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.614+0000 7f653effd640 1 -- 172.21.15.172:0/1118743515 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6534030b00 con 0x7f654015ab50 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.614+0000 7f6546415640 1 -- 172.21.15.172:0/1118743515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f654015ab50 msgr2=0x7f65400cb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.614+0000 7f6546415640 1 --2- 172.21.15.172:0/1118743515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f654015ab50 0x7f65400cb330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f65340097a0 tx=0x7f653402b960 comp rx=0 tx=0).stop 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.614+0000 7f6546415640 1 -- 172.21.15.172:0/1118743515 shutdown_connections 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.614+0000 7f6546415640 1 --2- 172.21.15.172:0/1118743515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400b3340 0x7f65400b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.614+0000 7f6546415640 1 --2- 172.21.15.172:0/1118743515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f654015ab50 0x7f65400cb330 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.615+0000 7f6546415640 1 -- 172.21.15.172:0/1118743515 >> 172.21.15.172:0/1118743515 conn(0x7f654006db50 msgr2=0x7f654006b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:16.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.615+0000 7f6546415640 1 -- 172.21.15.172:0/1118743515 shutdown_connections 2024-03-16T19:30:16.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.615+0000 7f6546415640 1 -- 172.21.15.172:0/1118743515 wait complete. 2024-03-16T19:30:16.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.616+0000 7f6546415640 1 Processor -- start 2024-03-16T19:30:16.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.616+0000 7f6546415640 1 -- start start 2024-03-16T19:30:16.598 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:16.613+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:16.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f6546415640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65400b3340 0x7f65400d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f6546415640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400ca450 0x7f6540152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f6546415640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65401f0740 0x7f65400cf360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f6546415640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401f1750 con 0x7f65401f0740 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f6546415640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b4270 con 0x7f65400b3340 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f6546415640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65401f0bd0 con 0x7f65400ca450 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f65449b2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65401f0740 0x7f65400cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f65449b2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65401f0740 0x7f65400cf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39730/0 (socket says 172.21.15.172:39730) 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f65449b2640 1 -- 172.21.15.172:0/702554394 learned_addr learned my addr 172.21.15.172:0/702554394 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f653ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65400b3340 0x7f65400d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f653f7fe640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400ca450 0x7f6540152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f65449b2640 1 -- 172.21.15.172:0/702554394 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400ca450 msgr2=0x7f6540152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f65449b2640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65400ca450 0x7f6540152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f65449b2640 1 -- 172.21.15.172:0/702554394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65400b3340 msgr2=0x7f65400d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f65449b2640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65400b3340 0x7f65400d01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f65449b2640 1 -- 172.21.15.172:0/702554394 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6538009a70 con 0x7f65401f0740 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f653ffff640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65400b3340 0x7f65400d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.617+0000 7f65449b2640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65401f0740 0x7f65400cf360 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f653800efc0 tx=0x7f6538009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.618+0000 7f653cff9640 1 -- 172.21.15.172:0/702554394 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538009d70 con 0x7f65401f0740 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.618+0000 7f653cff9640 1 -- 172.21.15.172:0/702554394 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653800f040 con 0x7f65401f0740 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.618+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6534009400 con 0x7f65401f0740 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.618+0000 7f653cff9640 1 -- 172.21.15.172:0/702554394 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538003990 con 0x7f65401f0740 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.618+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65401f1190 con 0x7f65401f0740 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.618+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6540158990 con 0x7f65401f0740 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.619+0000 7f653cff9640 1 -- 172.21.15.172:0/702554394 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6538014480 con 0x7f65401f0740 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.620+0000 7f653cff9640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f652c06b3b0 0x7f652c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.620+0000 7f653cff9640 1 -- 172.21.15.172:0/702554394 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f653808b350 con 0x7f65401f0740 2024-03-16T19:30:16.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.620+0000 7f653ffff640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f652c06b3b0 0x7f652c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.620+0000 7f653cff9640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f652c073730 0x7f652c075bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.620+0000 7f653cff9640 1 -- 172.21.15.172:0/702554394 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f652c076280 con 0x7f652c073730 2024-03-16T19:30:16.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.620+0000 7f653cff9640 1 -- 172.21.15.172:0/702554394 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f653808b7a0 con 0x7f65401f0740 2024-03-16T19:30:16.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.620+0000 7f653f7fe640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f652c073730 0x7f652c075bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.620+0000 7f653ffff640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f652c06b3b0 0x7f652c06d870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6530000f60 tx=0x7f6530009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.620+0000 7f653f7fe640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f652c073730 0x7f652c075bb0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.621+0000 7f653cff9640 1 -- 172.21.15.172:0/702554394 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f652c076280 con 0x7f652c073730 2024-03-16T19:30:16.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.638+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f654015e280 con 0x7f652c073730 2024-03-16T19:30:16.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.638+0000 7f653cff9640 1 -- 172.21.15.172:0/702554394 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f653403f040 con 0x7f652c073730 2024-03-16T19:30:16.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f652c073730 msgr2=0x7f652c075bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f652c073730 0x7f652c075bb0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f652c06b3b0 msgr2=0x7f652c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f652c06b3b0 0x7f652c06d870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6530000f60 tx=0x7f6530009040 comp rx=0 tx=0).stop 2024-03-16T19:30:16.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65401f0740 msgr2=0x7f65400cf360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 --2- 172.21.15.172:0/702554394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65401f0740 0x7f65400cf360 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f653800efc0 tx=0x7f6538009f90 comp rx=0 tx=0).stop 2024-03-16T19:30:16.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f65449b2640 1 -- 172.21.15.172:0/702554394 reap_dead start 2024-03-16T19:30:16.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 shutdown_connections 2024-03-16T19:30:16.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 >> 172.21.15.172:0/702554394 conn(0x7f654006db50 msgr2=0x7f65400c7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:16.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 shutdown_connections 2024-03-16T19:30:16.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.639+0000 7f6546415640 1 -- 172.21.15.172:0/702554394 wait complete. 2024-03-16T19:30:16.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-03-16T19:30:16.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.653+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 0x7f34101e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.653+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 0x7f34101e43b0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.654+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f34080020b0 con 0x7f34101533c0 2024-03-16T19:30:16.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.685+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f34100caac0 con 0x7f34101533c0 2024-03-16T19:30:16.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.687+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3408007330 con 0x7f34101533c0 2024-03-16T19:30:16.667 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:16.668 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:16.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.687+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f341014e940 0x7f34101f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.687+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34100ca140 con 0x7f341014e940 2024-03-16T19:30:16.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.687+0000 7f3414bf8640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f341014e940 0x7f34101f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.687+0000 7f3414bf8640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f341014e940 0x7f34101f3ca0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.688+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f34100ca140 con 0x7f341014e940 2024-03-16T19:30:16.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.703+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f341014ff10 con 0x7f341014e940 2024-03-16T19:30:16.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.705+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f341014ff10 con 0x7f341014e940 2024-03-16T19:30:16.686 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:16.686 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:16.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.705+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f34101f46f0 0x7f3410061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.686 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.705+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34101f4ad0 con 0x7f34101f46f0 2024-03-16T19:30:16.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.705+0000 7f340ffff640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f34101f46f0 0x7f3410061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.706+0000 7f340ffff640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f34101f46f0 0x7f3410061930 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.707+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f34101f4ad0 con 0x7f34101f46f0 2024-03-16T19:30:16.695 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:16.714+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:16.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.721+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3410154160 con 0x7f34101f46f0 2024-03-16T19:30:16.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.724+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3410154160 con 0x7f34101f46f0 2024-03-16T19:30:16.704 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:16.705 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:16.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.724+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f341015bc40 0x7f341015c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.724+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34100c92b0 con 0x7f341015bc40 2024-03-16T19:30:16.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.724+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f341015bc40 0x7f341015c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.725+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f341015bc40 0x7f341015c060 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.725+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f34100c92b0 con 0x7f341015bc40 2024-03-16T19:30:16.716 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:16.716 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:16.720 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.740+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f34100b5d70 con 0x7f341015bc40 2024-03-16T19:30:16.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.741+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f34100b5d70 con 0x7f341015bc40 2024-03-16T19:30:16.722 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:16.723 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:16.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.742+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f341002bde0 0x7f341002c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.742+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34100cfc60 con 0x7f341002bde0 2024-03-16T19:30:16.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.742+0000 7f3414bf8640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f341002bde0 0x7f341002c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.742+0000 7f3414bf8640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f341002bde0 0x7f341002c200 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.743+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f34100cfc60 con 0x7f341002bde0 2024-03-16T19:30:16.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:16.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.758+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f34100cfc60 con 0x7f341002bde0 2024-03-16T19:30:16.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.759+0000 7f5a03a47640 1 Processor -- start 2024-03-16T19:30:16.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.759+0000 7f5a03a47640 1 -- start start 2024-03-16T19:30:16.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f34100cfc60 con 0x7f341002bde0 2024-03-16T19:30:16.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a03a47640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f59fc152a20 0x7f59fc06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a03a47640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f59fc0c93a0 0x7f59fc0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a03a47640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc072610 con 0x7f59fc0c79d0 2024-03-16T19:30:16.742 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:16.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a03a47640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc0c9890 con 0x7f59fc152a20 2024-03-16T19:30:16.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a03a47640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc0729c0 con 0x7f59fc0c93a0 2024-03-16T19:30:16.743 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:16.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a00fe2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f59fc0c93a0 0x7f59fc0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f341002cc50 0x7f3410207030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a017e3640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f59fc152a20 0x7f59fc06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f341014ed20 con 0x7f341002cc50 2024-03-16T19:30:16.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f340ffff640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f341002cc50 0x7f3410207030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a00fe2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f59fc0c93a0 0x7f59fc0ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:56054/0 (socket says 172.21.15.172:56054) 2024-03-16T19:30:16.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a00fe2640 1 -- 172.21.15.172:0/1378743919 learned_addr learned my addr 172.21.15.172:0/1378743919 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:16.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.760+0000 7f5a017e3640 1 --2- 172.21.15.172:0/1378743919 >> v2:172.21.15.172:3300/0 conn(0x7f59fc152a20 0x7f59fc06c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39742/0 (socket says 172.21.15.172:39742) 2024-03-16T19:30:16.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.761+0000 7f5a00fe2640 1 -- 172.21.15.172:0/1378743919 >> v2:172.21.15.172:3300/0 conn(0x7f59fc152a20 msgr2=0x7f59fc06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:16.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.761+0000 7f5a00fe2640 1 --2- 172.21.15.172:0/1378743919 >> v2:172.21.15.172:3300/0 conn(0x7f59fc152a20 0x7f59fc06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.761+0000 7f5a00fe2640 1 -- 172.21.15.172:0/1378743919 >> v1:172.21.15.172:6789/0 conn(0x7f59fc0c79d0 legacy=0x7f59fc06b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.761+0000 7f5a00fe2640 1 -- 172.21.15.172:0/1378743919 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc072ca0 con 0x7f59fc0c93a0 2024-03-16T19:30:16.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.761+0000 7f340ffff640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f341002cc50 0x7f3410207030 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.761+0000 7f5a00fe2640 1 --2- 172.21.15.172:0/1378743919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0c93a0 0x7f59fc0ccf60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009870 tx=0x7f59f802ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77b6edc6b68f5391 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f341014ed20 con 0x7f341002cc50 2024-03-16T19:30:16.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f59f3fff640 1 -- 172.21.15.172:0/1378743919 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f802f840 con 0x7f59fc0c93a0 2024-03-16T19:30:16.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f59f3fff640 1 -- 172.21.15.172:0/1378743919 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f802f9a0 con 0x7f59fc0c93a0 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f59f3fff640 1 -- 172.21.15.172:0/1378743919 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f802fd30 con 0x7f59fc0c93a0 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f5a03a47640 1 -- 172.21.15.172:0/1378743919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0c93a0 msgr2=0x7f59fc0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f5a03a47640 1 --2- 172.21.15.172:0/1378743919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0c93a0 0x7f59fc0ccf60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009870 tx=0x7f59f802ba20 comp rx=0 tx=0).stop 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f5a03a47640 1 -- 172.21.15.172:0/1378743919 shutdown_connections 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f5a03a47640 1 --2- 172.21.15.172:0/1378743919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0c93a0 0x7f59fc0ccf60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f5a03a47640 1 --2- 172.21.15.172:0/1378743919 >> v2:172.21.15.172:3300/0 conn(0x7f59fc152a20 0x7f59fc06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.762+0000 7f5a03a47640 1 -- 172.21.15.172:0/1378743919 >> 172.21.15.172:0/1378743919 conn(0x7f59fc06db70 msgr2=0x7f59fc06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.763+0000 7f5a03a47640 1 -- 172.21.15.172:0/1378743919 shutdown_connections 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.763+0000 7f5a03a47640 1 -- 172.21.15.172:0/1378743919 wait complete. 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.764+0000 7f5a03a47640 1 Processor -- start 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.764+0000 7f5a03a47640 1 -- start start 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.764+0000 7f5a03a47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0c79d0 0x7f59fc0c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.764+0000 7f5a03a47640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59fc0c93a0 0x7f59fc1f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.764+0000 7f5a03a47640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59fc152a20 0x7f59fc1eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.764+0000 7f5a03a47640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc1ee0b0 con 0x7f59fc152a20 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.764+0000 7f5a03a47640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0c64c0 con 0x7f59fc0c79d0 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.764+0000 7f5a03a47640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc1f2570 con 0x7f59fc0c93a0 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a01fe4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59fc152a20 0x7f59fc1eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a01fe4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59fc152a20 0x7f59fc1eeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39752/0 (socket says 172.21.15.172:39752) 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a01fe4640 1 -- 172.21.15.172:0/3927273548 learned_addr learned my addr 172.21.15.172:0/3927273548 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a00fe2640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59fc0c93a0 0x7f59fc1f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a017e3640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0c79d0 0x7f59fc0c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a01fe4640 1 -- 172.21.15.172:0/3927273548 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59fc0c93a0 msgr2=0x7f59fc1f2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a01fe4640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59fc0c93a0 0x7f59fc1f2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a01fe4640 1 -- 172.21.15.172:0/3927273548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0c79d0 msgr2=0x7f59fc0c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a01fe4640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0c79d0 0x7f59fc0c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a01fe4640 1 -- 172.21.15.172:0/3927273548 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f80094d0 con 0x7f59fc152a20 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a00fe2640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59fc0c93a0 0x7f59fc1f2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:16.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a01fe4640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59fc152a20 0x7f59fc1eeaa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f59f400c7a0 tx=0x7f59f400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f59f1ffb640 1 -- 172.21.15.172:0/3927273548 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4003b70 con 0x7f59fc152a20 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f59f1ffb640 1 -- 172.21.15.172:0/3927273548 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f4003cf0 con 0x7f59fc152a20 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f59f1ffb640 1 -- 172.21.15.172:0/3927273548 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f400aa30 con 0x7f59fc152a20 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.765+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc152f80 con 0x7f59fc152a20 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.766+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc1ee610 con 0x7f59fc152a20 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.767+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f59fc152e00 con 0x7f59fc152a20 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.767+0000 7f59f1ffb640 1 -- 172.21.15.172:0/3927273548 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f59f400ab90 con 0x7f59fc152a20 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.767+0000 7f59f1ffb640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f59e006b3b0 0x7f59e006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.767+0000 7f59f1ffb640 1 -- 172.21.15.172:0/3927273548 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f59f40899f0 con 0x7f59fc152a20 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.767+0000 7f5a017e3640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f59e006b3b0 0x7f59e006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.767+0000 7f59f1ffb640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f59e0073730 0x7f59e0075bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.767+0000 7f59f1ffb640 1 -- 172.21.15.172:0/3927273548 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59e0076280 con 0x7f59e0073730 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.767+0000 7f59f1ffb640 1 -- 172.21.15.172:0/3927273548 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59f408e2c0 con 0x7f59fc152a20 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.768+0000 7f5a00fe2640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f59e0073730 0x7f59e0075bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.768+0000 7f5a017e3640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f59e006b3b0 0x7f59e006d870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009aa0 tx=0x7f59ec009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.768+0000 7f5a00fe2640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f59e0073730 0x7f59e0075bb0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.769+0000 7f59f1ffb640 1 -- 172.21.15.172:0/3927273548 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f59e0076280 con 0x7f59e0073730 2024-03-16T19:30:16.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.779+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f341014ed20 con 0x7f341002cc50 2024-03-16T19:30:16.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.780+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f341014ed20 con 0x7f341002cc50 2024-03-16T19:30:16.761 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:16.762 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:16.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.781+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f341002d090 0x7f3410209a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.781+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f341020c860 con 0x7f341002d090 2024-03-16T19:30:16.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.781+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f341002d090 0x7f3410209a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.781+0000 7f34153f9640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f341002d090 0x7f3410209a90 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.782+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f341020c860 con 0x7f341002d090 2024-03-16T19:30:16.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.790+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f59fc073260 con 0x7f59e0073730 2024-03-16T19:30:16.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.790+0000 7f59f1ffb640 1 -- 172.21.15.172:0/3927273548 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f59f802f800 con 0x7f59e0073730 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f59e0073730 msgr2=0x7f59e0075bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f59e0073730 0x7f59e0075bb0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f59e006b3b0 msgr2=0x7f59e006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f59e006b3b0 0x7f59e006d870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009aa0 tx=0x7f59ec009190 comp rx=0 tx=0).stop 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59fc152a20 msgr2=0x7f59fc1eeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 --2- 172.21.15.172:0/3927273548 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59fc152a20 0x7f59fc1eeaa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f59f400c7a0 tx=0x7f59f400cc60 comp rx=0 tx=0).stop 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a01fe4640 1 -- 172.21.15.172:0/3927273548 reap_dead start 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 shutdown_connections 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 >> 172.21.15.172:0/3927273548 conn(0x7f59fc06db70 msgr2=0x7f59fc0719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 shutdown_connections 2024-03-16T19:30:16.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.791+0000 7f5a03a47640 1 -- 172.21.15.172:0/3927273548 wait complete. 2024-03-16T19:30:16.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.800+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3410209fd0 con 0x7f341002d090 2024-03-16T19:30:16.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-03-16T19:30:16.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.802+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3410209fd0 con 0x7f341002d090 2024-03-16T19:30:16.783 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:16.784 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:16.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.802+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f341020a6b0 0x7f341020caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.802+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f341020fa70 con 0x7f341020a6b0 2024-03-16T19:30:16.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.803+0000 7f3414bf8640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f341020a6b0 0x7f341020caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.803+0000 7f3414bf8640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f341020a6b0 0x7f341020caa0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.804+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f341020fa70 con 0x7f341020a6b0 2024-03-16T19:30:16.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.819+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f341020e840 con 0x7f341020a6b0 2024-03-16T19:30:16.800 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:16.820+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:16.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f340d7fa640 1 -- 172.21.15.172:0/3434376131 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f341020e840 con 0x7f341020a6b0 2024-03-16T19:30:16.802 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:16.803 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:16.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 msgr2=0x7f34101e43b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f34101533c0 0x7f34101e43b0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f341014e940 msgr2=0x7f34101f3ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f341014e940 0x7f34101f3ca0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f34101f46f0 msgr2=0x7f3410061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f34101f46f0 0x7f3410061930 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f341015bc40 msgr2=0x7f341015c060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f341015bc40 0x7f341015c060 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f341002bde0 msgr2=0x7f341002c200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f341002bde0 0x7f341002c200 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f341002cc50 msgr2=0x7f3410207030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f341002cc50 0x7f3410207030 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f341002d090 msgr2=0x7f3410209a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f34153f9640 1 -- 172.21.15.172:0/3434376131 reap_dead start 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f341002d090 0x7f3410209a90 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.821+0000 7f34153f9640 1 -- 172.21.15.172:0/3434376131 reap_dead start 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f341020a6b0 msgr2=0x7f341020caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f341020a6b0 0x7f341020caa0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f34153f9640 1 -- 172.21.15.172:0/3434376131 reap_dead start 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f33f406b3b0 msgr2=0x7f33f406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f33f406b3b0 0x7f33f406d870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f33fc007fd0 tx=0x7f33fc009040 comp rx=0 tx=0).stop 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f341006bcc0 msgr2=0x7f34101f11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f341006bcc0 0x7f34101f11b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f34040098f0 tx=0x7f34040046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 shutdown_connections 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f33f406b3b0 0x7f33f406d870 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 --2- 172.21.15.172:0/3434376131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f341006bcc0 0x7f34101f11b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 >> 172.21.15.172:0/3434376131 conn(0x7f34100d0140 msgr2=0x7f34101e3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 shutdown_connections 2024-03-16T19:30:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.822+0000 7f3416e5c640 1 -- 172.21.15.172:0/3434376131 wait complete. 2024-03-16T19:30:16.852 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:16.852 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:16.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:16.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.908+0000 7f600352e640 1 Processor -- start 2024-03-16T19:30:16.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.909+0000 7f600352e640 1 -- start start 2024-03-16T19:30:16.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.909+0000 7f600352e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ffc154e50 0x7f5ffc0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.909+0000 7f600352e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ffc0caf40 0x7f5ffc0cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.909+0000 7f600352e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ffc06b110 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.909+0000 7f600352e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ffc15b8d0 con 0x7f5ffc0caf40 2024-03-16T19:30:16.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.909+0000 7f600352e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc15ad50 con 0x7f5ffc154e50 2024-03-16T19:30:16.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.909+0000 7f6001acb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ffc0caf40 0x7f5ffc0cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.909+0000 7f6001acb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5ffc0caf40 0x7f5ffc0cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39760/0 (socket says 172.21.15.172:39760) 2024-03-16T19:30:16.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.909+0000 7f6001acb640 1 -- 172.21.15.172:0/316444771 learned_addr learned my addr 172.21.15.172:0/316444771 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:16.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.910+0000 7f60012ca640 1 --2- 172.21.15.172:0/316444771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ffc154e50 0x7f5ffc0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.910+0000 7f6001acb640 1 -- 172.21.15.172:0/316444771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ffc154e50 msgr2=0x7f5ffc0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:16.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.910+0000 7f6001acb640 1 --2- 172.21.15.172:0/316444771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ffc154e50 0x7f5ffc0cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.910+0000 7f6001acb640 1 -- 172.21.15.172:0/316444771 >> v1:172.21.15.174:6789/0 conn(0x7f5ffc0b5f10 legacy=0x7f5ffc06b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.910+0000 7f6001acb640 1 -- 172.21.15.172:0/316444771 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc15b030 con 0x7f5ffc0caf40 2024-03-16T19:30:16.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.910+0000 7f5ffbfff640 1 -- 172.21.15.172:0/316444771 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464092418 0 0) 0x7f5ffc06b110 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.910+0000 7f6001acb640 1 --2- 172.21.15.172:0/316444771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0caf40 0x7f5ffc0cc170 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009870 tx=0x7f5fec02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb1be65f5d1a6bb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f5ffbfff640 1 -- 172.21.15.172:0/316444771 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec02f840 con 0x7f5ffc0caf40 2024-03-16T19:30:16.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f5ffbfff640 1 -- 172.21.15.172:0/316444771 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec02f9a0 con 0x7f5ffc0caf40 2024-03-16T19:30:16.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f5ffbfff640 1 -- 172.21.15.172:0/316444771 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec02fd30 con 0x7f5ffc0caf40 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f600352e640 1 -- 172.21.15.172:0/316444771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0caf40 msgr2=0x7f5ffc0cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f600352e640 1 --2- 172.21.15.172:0/316444771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0caf40 0x7f5ffc0cc170 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009870 tx=0x7f5fec02ba20 comp rx=0 tx=0).stop 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f600352e640 1 -- 172.21.15.172:0/316444771 shutdown_connections 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f600352e640 1 --2- 172.21.15.172:0/316444771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0caf40 0x7f5ffc0cc170 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f600352e640 1 --2- 172.21.15.172:0/316444771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ffc154e50 0x7f5ffc0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f600352e640 1 -- 172.21.15.172:0/316444771 >> 172.21.15.172:0/316444771 conn(0x7f5ffc066c20 msgr2=0x7f5ffc067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f600352e640 1 -- 172.21.15.172:0/316444771 shutdown_connections 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.911+0000 7f600352e640 1 -- 172.21.15.172:0/316444771 wait complete. 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f600352e640 1 Processor -- start 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f600352e640 1 -- start start 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f600352e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0b5f10 0x7f5ffc149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f600352e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc0caf40 0x7f5ffc0c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f600352e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ffc154e50 0x7f5ffc0c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f600352e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc151430 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f600352e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc1554a0 con 0x7f5ffc0caf40 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f600352e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc1559d0 con 0x7f5ffc154e50 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f60012ca640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0b5f10 0x7f5ffc149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f60012ca640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0b5f10 0x7f5ffc149e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39764/0 (socket says 172.21.15.172:39764) 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f60012ca640 1 -- 172.21.15.172:0/795346120 learned_addr learned my addr 172.21.15.172:0/795346120 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f6001acb640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ffc154e50 0x7f5ffc0c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.913+0000 7f6000ac9640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc0caf40 0x7f5ffc0c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f60012ca640 1 -- 172.21.15.172:0/795346120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ffc154e50 msgr2=0x7f5ffc0c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f60012ca640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ffc154e50 0x7f5ffc0c50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f60012ca640 1 -- 172.21.15.172:0/795346120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc0caf40 msgr2=0x7f5ffc0c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f60012ca640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ffc0caf40 0x7f5ffc0c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f60012ca640 1 -- 172.21.15.172:0/795346120 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec0094d0 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f60012ca640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0b5f10 0x7f5ffc149e90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5ff400efc0 tx=0x7f5ff400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f5ff9ffb640 1 -- 172.21.15.172:0/795346120 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4009b80 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f5ff9ffb640 1 -- 172.21.15.172:0/795346120 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff400f040 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f600352e640 1 -- 172.21.15.172:0/795346120 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ffc15ab20 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f5ff9ffb640 1 -- 172.21.15.172:0/795346120 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff40146a0 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.914+0000 7f600352e640 1 -- 172.21.15.172:0/795346120 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ffc1556f0 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.915+0000 7f5fe37fe640 1 -- 172.21.15.172:0/795346120 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5ffc0ca9b0 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.916+0000 7f5ff9ffb640 1 -- 172.21.15.172:0/795346120 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5ff4014800 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.916+0000 7f5ff9ffb640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fe806b3e0 0x7f5fe806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.916+0000 7f5ff9ffb640 1 -- 172.21.15.172:0/795346120 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f5ff408a6a0 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.916+0000 7f6000ac9640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fe806b3e0 0x7f5fe806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.916+0000 7f5ff9ffb640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f5fe8073700 0x7f5fe8075b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:16.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.916+0000 7f5ff9ffb640 1 -- 172.21.15.172:0/795346120 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe8076250 con 0x7f5fe8073700 2024-03-16T19:30:16.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.916+0000 7f5ff9ffb640 1 -- 172.21.15.172:0/795346120 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ff408cd60 con 0x7f5ffc0b5f10 2024-03-16T19:30:16.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.917+0000 7f6001acb640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f5fe8073700 0x7f5fe8075b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:16.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.917+0000 7f6000ac9640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fe806b3e0 0x7f5fe806d8a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0c5cf0 tx=0x7f5ff0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.917+0000 7f6001acb640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f5fe8073700 0x7f5fe8075b80 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:16.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.918+0000 7f5ff9ffb640 1 -- 172.21.15.172:0/795346120 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5fe8076250 con 0x7f5fe8073700 2024-03-16T19:30:16.900 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:16.917+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:16.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.935+0000 7f5fe37fe640 1 -- 172.21.15.172:0/795346120 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f5ffc0cb780 con 0x7f5fe8073700 2024-03-16T19:30:16.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.935+0000 7f5ff9ffb640 1 -- 172.21.15.172:0/795346120 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5ffc0cb780 con 0x7f5fe8073700 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 -- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f5fe8073700 msgr2=0x7f5fe8075b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f5fe8073700 0x7f5fe8075b80 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 -- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fe806b3e0 msgr2=0x7f5fe806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fe806b3e0 0x7f5fe806d8a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0c5cf0 tx=0x7f5ff0009040 comp rx=0 tx=0).stop 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 -- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0b5f10 msgr2=0x7f5ffc149e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 --2- 172.21.15.172:0/795346120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ffc0b5f10 0x7f5ffc149e90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5ff400efc0 tx=0x7f5ff400c490 comp rx=0 tx=0).stop 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f6001acb640 1 -- 172.21.15.172:0/795346120 reap_dead start 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 -- 172.21.15.172:0/795346120 shutdown_connections 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 -- 172.21.15.172:0/795346120 >> 172.21.15.172:0/795346120 conn(0x7f5ffc066c20 msgr2=0x7f5ffc0cef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 -- 172.21.15.172:0/795346120 shutdown_connections 2024-03-16T19:30:16.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:16.936+0000 7f5fe37fe640 1 -- 172.21.15.172:0/795346120 wait complete. 2024-03-16T19:30:16.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-03-16T19:30:17.000 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:17.000 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:17.001 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:17.020+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:17.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:17.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.090+0000 7f1e53d1e640 1 Processor -- start 2024-03-16T19:30:17.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.090+0000 7f1e53d1e640 1 -- start start 2024-03-16T19:30:17.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e53d1e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 0x7f1e4c06b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e53d1e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1e4c1580b0 0x7f1e4c161fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e53d1e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e4c0c1eb0 con 0x7f1e4c06baf0 2024-03-16T19:30:17.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e53d1e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4c0c1150 con 0x7f1e4c1580b0 2024-03-16T19:30:17.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e53d1e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c06b7f0 con 0x7f1e4c06aed0 2024-03-16T19:30:17.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e522bb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1e4c1580b0 0x7f1e4c161fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e51aba640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 0x7f1e4c06b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e522bb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1e4c1580b0 0x7f1e4c161fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:56096/0 (socket says 172.21.15.172:56096) 2024-03-16T19:30:17.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e51aba640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 0x7f1e4c06b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44030/0 (socket says 172.21.15.172:44030) 2024-03-16T19:30:17.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e522bb640 1 -- 172.21.15.172:0/962005654 learned_addr learned my addr 172.21.15.172:0/962005654 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:17.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.091+0000 7f1e512b9640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f1e4c06baf0 0x7f1e4c1579a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:52552/0 (socket says 172.21.15.172:52552) 2024-03-16T19:30:17.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.092+0000 7f1e522bb640 1 -- 172.21.15.172:0/962005654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 msgr2=0x7f1e4c06b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:17.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.092+0000 7f1e522bb640 1 --2- 172.21.15.172:0/962005654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 0x7f1e4c06b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.092+0000 7f1e522bb640 1 -- 172.21.15.172:0/962005654 >> v1:172.21.15.174:6789/0 conn(0x7f1e4c06baf0 legacy=0x7f1e4c1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.092+0000 7f1e522bb640 1 -- 172.21.15.172:0/962005654 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c0c1350 con 0x7f1e4c1580b0 2024-03-16T19:30:17.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.092+0000 7f1e522bb640 1 --2- 172.21.15.172:0/962005654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e4c1580b0 0x7f1e4c161fd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e44009e50 tx=0x7f1e4402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf52da1953917d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.092+0000 7f1e50ab8640 1 -- 172.21.15.172:0/962005654 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4402cb00 con 0x7f1e4c1580b0 2024-03-16T19:30:17.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.092+0000 7f1e50ab8640 1 -- 172.21.15.172:0/962005654 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4402cc60 con 0x7f1e4c1580b0 2024-03-16T19:30:17.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.093+0000 7f1e50ab8640 1 -- 172.21.15.172:0/962005654 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e440343f0 con 0x7f1e4c1580b0 2024-03-16T19:30:17.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.093+0000 7f1e53d1e640 1 -- 172.21.15.172:0/962005654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e4c1580b0 msgr2=0x7f1e4c161fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.093+0000 7f1e53d1e640 1 --2- 172.21.15.172:0/962005654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e4c1580b0 0x7f1e4c161fd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e44009e50 tx=0x7f1e4402bf60 comp rx=0 tx=0).stop 2024-03-16T19:30:17.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.093+0000 7f1e53d1e640 1 -- 172.21.15.172:0/962005654 shutdown_connections 2024-03-16T19:30:17.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.093+0000 7f1e53d1e640 1 --2- 172.21.15.172:0/962005654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e4c1580b0 0x7f1e4c161fd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.093+0000 7f1e53d1e640 1 --2- 172.21.15.172:0/962005654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 0x7f1e4c06b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.093+0000 7f1e53d1e640 1 -- 172.21.15.172:0/962005654 >> 172.21.15.172:0/962005654 conn(0x7f1e4c0cc030 msgr2=0x7f1e4c0cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.093+0000 7f1e53d1e640 1 -- 172.21.15.172:0/962005654 shutdown_connections 2024-03-16T19:30:17.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.093+0000 7f1e53d1e640 1 -- 172.21.15.172:0/962005654 wait complete. 2024-03-16T19:30:17.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.094+0000 7f1e53d1e640 1 Processor -- start 2024-03-16T19:30:17.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.094+0000 7f1e53d1e640 1 -- start start 2024-03-16T19:30:17.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e53d1e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 0x7f1e4c14c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e53d1e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e4c06baf0 0x7f1e4c13f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e53d1e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e4c1580b0 0x7f1e4c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e53d1e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c0b1640 con 0x7f1e4c06baf0 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e53d1e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c155ed0 con 0x7f1e4c1580b0 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e53d1e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c14cb20 con 0x7f1e4c06aed0 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e512b9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e4c06baf0 0x7f1e4c13f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e512b9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e4c06baf0 0x7f1e4c13f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39774/0 (socket says 172.21.15.172:39774) 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e51aba640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 0x7f1e4c14c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e512b9640 1 -- 172.21.15.172:0/3716205496 learned_addr learned my addr 172.21.15.172:0/3716205496 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.095+0000 7f1e522bb640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e4c1580b0 0x7f1e4c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e51aba640 1 -- 172.21.15.172:0/3716205496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e4c1580b0 msgr2=0x7f1e4c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e51aba640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e4c1580b0 0x7f1e4c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e51aba640 1 -- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e4c06baf0 msgr2=0x7f1e4c13f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e51aba640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e4c06baf0 0x7f1e4c13f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e51aba640 1 -- 172.21.15.172:0/3716205496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e44009b30 con 0x7f1e4c06aed0 2024-03-16T19:30:17.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e51aba640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 0x7f1e4c14c5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00e880 tx=0x7f1e3c00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e427fc640 1 -- 172.21.15.172:0/3716205496 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c00c940 con 0x7f1e4c06aed0 2024-03-16T19:30:17.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e427fc640 1 -- 172.21.15.172:0/3716205496 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c00caa0 con 0x7f1e4c06aed0 2024-03-16T19:30:17.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4c1570e0 con 0x7f1e4c06aed0 2024-03-16T19:30:17.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e427fc640 1 -- 172.21.15.172:0/3716205496 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c00ce50 con 0x7f1e4c06aed0 2024-03-16T19:30:17.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.096+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4c1f4ab0 con 0x7f1e4c06aed0 2024-03-16T19:30:17.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.097+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e4c0c1bd0 con 0x7f1e4c06aed0 2024-03-16T19:30:17.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.097+0000 7f1e427fc640 1 -- 172.21.15.172:0/3716205496 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1e3c013470 con 0x7f1e4c06aed0 2024-03-16T19:30:17.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.098+0000 7f1e427fc640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e3006b3b0 0x7f1e3006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.098+0000 7f1e427fc640 1 -- 172.21.15.172:0/3716205496 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f1e3c089c40 con 0x7f1e4c06aed0 2024-03-16T19:30:17.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.098+0000 7f1e512b9640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e3006b3b0 0x7f1e3006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.098+0000 7f1e427fc640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1e300736d0 0x7f1e30075b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.098+0000 7f1e427fc640 1 -- 172.21.15.172:0/3716205496 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e30076220 con 0x7f1e300736d0 2024-03-16T19:30:17.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.098+0000 7f1e427fc640 1 -- 172.21.15.172:0/3716205496 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e3c08c300 con 0x7f1e4c06aed0 2024-03-16T19:30:17.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.098+0000 7f1e522bb640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1e300736d0 0x7f1e30075b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.098+0000 7f1e512b9640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e3006b3b0 0x7f1e3006d870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c157ea0 tx=0x7f1e48007f10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.099+0000 7f1e522bb640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1e300736d0 0x7f1e30075b50 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.099+0000 7f1e427fc640 1 -- 172.21.15.172:0/3716205496 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1e30076220 con 0x7f1e300736d0 2024-03-16T19:30:17.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.116+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f1e4c0c18f0 con 0x7f1e300736d0 2024-03-16T19:30:17.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.116+0000 7f1e427fc640 1 -- 172.21.15.172:0/3716205496 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1e4402caf0 con 0x7f1e300736d0 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.116+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1e300736d0 msgr2=0x7f1e30075b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.116+0000 7f1e53d1e640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1e300736d0 0x7f1e30075b50 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.117+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e3006b3b0 msgr2=0x7f1e3006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.117+0000 7f1e53d1e640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e3006b3b0 0x7f1e3006d870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c157ea0 tx=0x7f1e48007f10 comp rx=0 tx=0).stop 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.117+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 msgr2=0x7f1e4c14c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.117+0000 7f1e53d1e640 1 --2- 172.21.15.172:0/3716205496 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e4c06aed0 0x7f1e4c14c5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00e880 tx=0x7f1e3c00ed40 comp rx=0 tx=0).stop 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.117+0000 7f1e522bb640 1 -- 172.21.15.172:0/3716205496 reap_dead start 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.117+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 shutdown_connections 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.117+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 >> 172.21.15.172:0/3716205496 conn(0x7f1e4c0cc030 msgr2=0x7f1e4c147b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.117+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 shutdown_connections 2024-03-16T19:30:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.117+0000 7f1e53d1e640 1 -- 172.21.15.172:0/3716205496 wait complete. 2024-03-16T19:30:17.100 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:17.119+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:17.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-03-16T19:30:17.160 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:17.161 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:17.170 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:17.211 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:17.231+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:17.251 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.270+0000 7fb2d79c7640 1 Processor -- start 2024-03-16T19:30:17.251 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.270+0000 7fb2d79c7640 1 -- start start 2024-03-16T19:30:17.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.271+0000 7fb2d79c7640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb2d0154be0 0x7fb2d015afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.271+0000 7fb2d79c7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb2d006eb40 0x7fb2d00c9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.271+0000 7fb2d79c7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d015ab30 con 0x7fb2d015b500 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.271+0000 7fb2d79c7640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d00c7e00 con 0x7fb2d0154be0 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.271+0000 7fb2d79c7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d015a890 con 0x7fb2d006eb40 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.271+0000 7fb2d5763640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb2d0154be0 0x7fb2d015afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.271+0000 7fb2d5763640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb2d0154be0 0x7fb2d015afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39780/0 (socket says 172.21.15.172:39780) 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.271+0000 7fb2d5763640 1 -- 172.21.15.172:0/3036517967 learned_addr learned my addr 172.21.15.172:0/3036517967 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.272+0000 7fb2d4f62640 1 --2- 172.21.15.172:0/3036517967 >> v2:172.21.15.174:3300/0 conn(0x7fb2d006eb40 0x7fb2d00c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.272+0000 7fb2d5763640 1 -- 172.21.15.172:0/3036517967 >> v2:172.21.15.174:3300/0 conn(0x7fb2d006eb40 msgr2=0x7fb2d00c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.272+0000 7fb2d5763640 1 --2- 172.21.15.172:0/3036517967 >> v2:172.21.15.174:3300/0 conn(0x7fb2d006eb40 0x7fb2d00c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.272+0000 7fb2d5763640 1 -- 172.21.15.172:0/3036517967 >> v1:172.21.15.172:6789/0 conn(0x7fb2d015b500 legacy=0x7fb2d00c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.272+0000 7fb2d5763640 1 -- 172.21.15.172:0/3036517967 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0154fc0 con 0x7fb2d0154be0 2024-03-16T19:30:17.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.272+0000 7fb2cffff640 1 -- 172.21.15.172:0/3036517967 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991472919 0 0) 0x7fb2d015ab30 con 0x7fb2d015b500 2024-03-16T19:30:17.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.272+0000 7fb2d5763640 1 --2- 172.21.15.172:0/3036517967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d0154be0 0x7fb2d015afc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00ca5d0 tx=0x7fb2c802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c7c311f893f69e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2cffff640 1 -- 172.21.15.172:0/3036517967 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c802ca80 con 0x7fb2d0154be0 2024-03-16T19:30:17.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2cffff640 1 -- 172.21.15.172:0/3036517967 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c802cbe0 con 0x7fb2d0154be0 2024-03-16T19:30:17.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2cffff640 1 -- 172.21.15.172:0/3036517967 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c80343f0 con 0x7fb2d0154be0 2024-03-16T19:30:17.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2d79c7640 1 -- 172.21.15.172:0/3036517967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d0154be0 msgr2=0x7fb2d015afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2d79c7640 1 --2- 172.21.15.172:0/3036517967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d0154be0 0x7fb2d015afc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00ca5d0 tx=0x7fb2c802bee0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2d79c7640 1 -- 172.21.15.172:0/3036517967 shutdown_connections 2024-03-16T19:30:17.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2d79c7640 1 --2- 172.21.15.172:0/3036517967 >> v2:172.21.15.174:3300/0 conn(0x7fb2d006eb40 0x7fb2d00c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2d79c7640 1 --2- 172.21.15.172:0/3036517967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d0154be0 0x7fb2d015afc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2d79c7640 1 -- 172.21.15.172:0/3036517967 >> 172.21.15.172:0/3036517967 conn(0x7fb2d00cc720 msgr2=0x7fb2d00ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2d79c7640 1 -- 172.21.15.172:0/3036517967 shutdown_connections 2024-03-16T19:30:17.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.273+0000 7fb2d79c7640 1 -- 172.21.15.172:0/3036517967 wait complete. 2024-03-16T19:30:17.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.274+0000 7fb2d79c7640 1 Processor -- start 2024-03-16T19:30:17.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.274+0000 7fb2d79c7640 1 -- start start 2024-03-16T19:30:17.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d79c7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d006eb40 0x7fb2d01f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d79c7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d0154be0 0x7fb2d00c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d79c7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d015b500 0x7fb2d01f0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d79c7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d01f43d0 con 0x7fb2d015b500 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d79c7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d01f02f0 con 0x7fb2d006eb40 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d79c7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0154300 con 0x7fb2d0154be0 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d5f64640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d015b500 0x7fb2d01f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d5f64640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d015b500 0x7fb2d01f0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39782/0 (socket says 172.21.15.172:39782) 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d5f64640 1 -- 172.21.15.172:0/2780629749 learned_addr learned my addr 172.21.15.172:0/2780629749 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d5763640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d006eb40 0x7fb2d01f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.275+0000 7fb2d4f62640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d0154be0 0x7fb2d00c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2d5f64640 1 -- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d0154be0 msgr2=0x7fb2d00c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2d5f64640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d0154be0 0x7fb2d00c5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2d5f64640 1 -- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d006eb40 msgr2=0x7fb2d01f3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2d5f64640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d006eb40 0x7fb2d01f3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2d5f64640 1 -- 172.21.15.172:0/2780629749 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c8009b30 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2d5f64640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d015b500 0x7fb2d01f0780 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000c430 tx=0x7fb2c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2cdffb640 1 -- 172.21.15.172:0/2780629749 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c00035f0 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2cdffb640 1 -- 172.21.15.172:0/2780629749 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c0003770 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2cdffb640 1 -- 172.21.15.172:0/2780629749 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0003b20 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d0153270 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.276+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d0153830 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.277+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2d015e800 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.277+0000 7fb2cdffb640 1 -- 172.21.15.172:0/2780629749 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb2c0003c80 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.278+0000 7fb2cdffb640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2bc06b3b0 0x7fb2bc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.278+0000 7fb2cdffb640 1 -- 172.21.15.172:0/2780629749 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fb2c0089310 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.278+0000 7fb2d5763640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2bc06b3b0 0x7fb2bc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.278+0000 7fb2cdffb640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb2bc0736d0 0x7fb2bc075b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.278+0000 7fb2cdffb640 1 -- 172.21.15.172:0/2780629749 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2bc076220 con 0x7fb2bc0736d0 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.278+0000 7fb2cdffb640 1 -- 172.21.15.172:0/2780629749 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2c008b9d0 con 0x7fb2d015b500 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.278+0000 7fb2d4f62640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb2bc0736d0 0x7fb2bc075b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.278+0000 7fb2d5763640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2bc06b3b0 0x7fb2bc06d870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8035000 tx=0x7fb2c8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.278+0000 7fb2d4f62640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb2bc0736d0 0x7fb2bc075b50 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.279+0000 7fb2cdffb640 1 -- 172.21.15.172:0/2780629749 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2bc076220 con 0x7fb2bc0736d0 2024-03-16T19:30:17.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7fb2d015e520 con 0x7fb2bc0736d0 2024-03-16T19:30:17.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2cdffb640 1 -- 172.21.15.172:0/2780629749 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb2c4009bd0 con 0x7fb2bc0736d0 2024-03-16T19:30:17.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb2bc0736d0 msgr2=0x7fb2bc075b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d79c7640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fb2bc0736d0 0x7fb2bc075b50 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2bc06b3b0 msgr2=0x7fb2bc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d79c7640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2bc06b3b0 0x7fb2bc06d870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8035000 tx=0x7fb2c8030000 comp rx=0 tx=0).stop 2024-03-16T19:30:17.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d015b500 msgr2=0x7fb2d01f0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d79c7640 1 --2- 172.21.15.172:0/2780629749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d015b500 0x7fb2d01f0780 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000c430 tx=0x7fb2c000c8f0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d5f64640 1 -- 172.21.15.172:0/2780629749 reap_dead start 2024-03-16T19:30:17.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 shutdown_connections 2024-03-16T19:30:17.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.296+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 >> 172.21.15.172:0/2780629749 conn(0x7fb2d00cc720 msgr2=0x7fb2d0158870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.297+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 shutdown_connections 2024-03-16T19:30:17.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.297+0000 7fb2d79c7640 1 -- 172.21.15.172:0/2780629749 wait complete. 2024-03-16T19:30:17.278 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:17.278 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:17.287 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:17.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-03-16T19:30:17.331 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:17.350+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:17.397 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:17.397 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:17.403 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:17.423+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:17.406 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:17.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.450+0000 7f6f9e8cd640 1 Processor -- start 2024-03-16T19:30:17.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.450+0000 7f6f9e8cd640 1 -- start start 2024-03-16T19:30:17.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f9e8cd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f9806baf0 0x7f6f9806b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f9e8cd640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f980c1eb0 con 0x7f6f9806aed0 2024-03-16T19:30:17.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f9e8cd640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f980c1350 con 0x7f6f981580b0 2024-03-16T19:30:17.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f9e8cd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980c1150 con 0x7f6f9806baf0 2024-03-16T19:30:17.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f97fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6f9806aed0 0x7f6f981579a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:57258/0 (socket says 172.21.15.172:57258) 2024-03-16T19:30:17.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f97fff640 1 -- 172.21.15.172:0/2468968876 learned_addr learned my addr 172.21.15.172:0/2468968876 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:17.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f977fe640 1 --2- 172.21.15.172:0/2468968876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f9806baf0 0x7f6f9806b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f96ffd640 1 -- 172.21.15.172:0/2468968876 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998632707 0 0) 0x7f6f980c1eb0 con 0x7f6f9806aed0 2024-03-16T19:30:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f96ffd640 1 -- 172.21.15.172:0/2468968876 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f80003520 con 0x7f6f9806aed0 2024-03-16T19:30:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f96ffd640 1 -- 172.21.15.172:0/2468968876 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068618648 0 0) 0x7f6f980c1350 con 0x7f6f981580b0 2024-03-16T19:30:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f96ffd640 1 -- 172.21.15.172:0/2468968876 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f980c1eb0 con 0x7f6f981580b0 2024-03-16T19:30:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f977fe640 1 -- 172.21.15.172:0/2468968876 >> v1:172.21.15.174:6789/0 conn(0x7f6f981580b0 legacy=0x7f6f98161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f977fe640 1 -- 172.21.15.172:0/2468968876 >> v1:172.21.15.172:6789/0 conn(0x7f6f9806aed0 legacy=0x7f6f981579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.451+0000 7f6f977fe640 1 -- 172.21.15.172:0/2468968876 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980c1610 con 0x7f6f9806baf0 2024-03-16T19:30:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.452+0000 7f6f977fe640 1 --2- 172.21.15.172:0/2468968876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f9806baf0 0x7f6f9806b2b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c0097d0 tx=0x7f6f8c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbbc7f6d79b55009 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.452+0000 7f6f96ffd640 1 -- 172.21.15.172:0/2468968876 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c031890 con 0x7f6f9806baf0 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.452+0000 7f6f96ffd640 1 -- 172.21.15.172:0/2468968876 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c0319f0 con 0x7f6f9806baf0 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.452+0000 7f6f96ffd640 1 -- 172.21.15.172:0/2468968876 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c031d80 con 0x7f6f9806baf0 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.453+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/2468968876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f9806baf0 msgr2=0x7f6f9806b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.453+0000 7f6f9e8cd640 1 --2- 172.21.15.172:0/2468968876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f9806baf0 0x7f6f9806b2b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c0097d0 tx=0x7f6f8c02b930 comp rx=0 tx=0).stop 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.453+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/2468968876 shutdown_connections 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.453+0000 7f6f9e8cd640 1 --2- 172.21.15.172:0/2468968876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f9806baf0 0x7f6f9806b2b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.453+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/2468968876 >> 172.21.15.172:0/2468968876 conn(0x7f6f980cc030 msgr2=0x7f6f980cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.453+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/2468968876 shutdown_connections 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.453+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/2468968876 wait complete. 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.454+0000 7f6f9e8cd640 1 Processor -- start 2024-03-16T19:30:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.454+0000 7f6f9e8cd640 1 -- start start 2024-03-16T19:30:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f9e8cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f9806aed0 0x7f6f98158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f9e8cd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f9806baf0 0x7f6f98152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f9e8cd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f981580b0 0x7f6f981f0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f9e8cd640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f981f0480 con 0x7f6f9806baf0 2024-03-16T19:30:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f9e8cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980b0b90 con 0x7f6f9806aed0 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f9e8cd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f981540f0 con 0x7f6f981580b0 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f977fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f9806baf0 0x7f6f98152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f977fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f9806baf0 0x7f6f98152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39784/0 (socket says 172.21.15.172:39784) 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f977fe640 1 -- 172.21.15.172:0/3710487738 learned_addr learned my addr 172.21.15.172:0/3710487738 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f97fff640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f9806aed0 0x7f6f98158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f9ce6a640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f981580b0 0x7f6f981f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f977fe640 1 -- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f981580b0 msgr2=0x7f6f981f0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f977fe640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f981580b0 0x7f6f981f0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f977fe640 1 -- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f9806aed0 msgr2=0x7f6f98158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f977fe640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f9806aed0 0x7f6f98158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.455+0000 7f6f977fe640 1 -- 172.21.15.172:0/3710487738 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c009400 con 0x7f6f9806baf0 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.456+0000 7f6f977fe640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f9806baf0 0x7f6f98152cb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c0097d0 tx=0x7f6f8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.456+0000 7f6f94ff9640 1 -- 172.21.15.172:0/3710487738 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c03bbf0 con 0x7f6f9806baf0 2024-03-16T19:30:17.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.456+0000 7f6f94ff9640 1 -- 172.21.15.172:0/3710487738 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c03bd50 con 0x7f6f9806baf0 2024-03-16T19:30:17.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.456+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f981543d0 con 0x7f6f9806baf0 2024-03-16T19:30:17.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.456+0000 7f6f94ff9640 1 -- 172.21.15.172:0/3710487738 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c045630 con 0x7f6f9806baf0 2024-03-16T19:30:17.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.456+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f98154990 con 0x7f6f9806baf0 2024-03-16T19:30:17.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.457+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6f980c6c50 con 0x7f6f9806baf0 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.457+0000 7f6f94ff9640 1 -- 172.21.15.172:0/3710487738 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6f8c04e450 con 0x7f6f9806baf0 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.457+0000 7f6f94ff9640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6f8406b3b0 0x7f6f8406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.458+0000 7f6f97fff640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6f8406b3b0 0x7f6f8406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.458+0000 7f6f94ff9640 1 -- 172.21.15.172:0/3710487738 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f6f8c0ac210 con 0x7f6f9806baf0 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.458+0000 7f6f94ff9640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6f840736d0 0x7f6f84075b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.458+0000 7f6f94ff9640 1 -- 172.21.15.172:0/3710487738 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f84076220 con 0x7f6f840736d0 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.458+0000 7f6f94ff9640 1 -- 172.21.15.172:0/3710487738 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f8c0ae8d0 con 0x7f6f9806baf0 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.458+0000 7f6f97fff640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6f8406b3b0 0x7f6f8406d870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6f88000f60 tx=0x7f6f88007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.458+0000 7f6f9ce6a640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6f840736d0 0x7f6f84075b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.458+0000 7f6f9ce6a640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6f840736d0 0x7f6f84075b50 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.459+0000 7f6f94ff9640 1 -- 172.21.15.172:0/3710487738 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6f84076220 con 0x7f6f840736d0 2024-03-16T19:30:17.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f6f9806bfc0 con 0x7f6f840736d0 2024-03-16T19:30:17.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f94ff9640 1 -- 172.21.15.172:0/3710487738 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6f90001b30 con 0x7f6f840736d0 2024-03-16T19:30:17.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6f840736d0 msgr2=0x7f6f84075b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9e8cd640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6f840736d0 0x7f6f84075b50 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6f8406b3b0 msgr2=0x7f6f8406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9e8cd640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6f8406b3b0 0x7f6f8406d870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6f88000f60 tx=0x7f6f88007d70 comp rx=0 tx=0).stop 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f9806baf0 msgr2=0x7f6f98152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9e8cd640 1 --2- 172.21.15.172:0/3710487738 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f9806baf0 0x7f6f98152cb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c0097d0 tx=0x7f6f8c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9ce6a640 1 -- 172.21.15.172:0/3710487738 reap_dead start 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 shutdown_connections 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.476+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 >> 172.21.15.172:0/3710487738 conn(0x7f6f980cc030 msgr2=0x7f6f980c6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.477+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 shutdown_connections 2024-03-16T19:30:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.477+0000 7f6f9e8cd640 1 -- 172.21.15.172:0/3710487738 wait complete. 2024-03-16T19:30:17.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-03-16T19:30:17.509 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:17.529+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:17.553 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:17.553 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:17.561 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:17.605 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:17.624+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:17.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.630+0000 7fb7b9bb5640 1 Processor -- start 2024-03-16T19:30:17.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.630+0000 7fb7b9bb5640 1 -- start start 2024-03-16T19:30:17.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b9bb5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb7b40c4a20 0x7fb7b415b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b9bb5640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb7b415ded0 0x7fb7b415e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b9bb5640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b415b920 con 0x7fb7b406fe70 2024-03-16T19:30:17.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b9bb5640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7b406db40 con 0x7fb7b415ded0 2024-03-16T19:30:17.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b9bb5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7b40c4e00 con 0x7fb7b40c4a20 2024-03-16T19:30:17.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b3fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb7b415ded0 0x7fb7b415e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b2ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb7b406fe70 0x7fb7b40ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:57272/0 (socket says 172.21.15.172:57272) 2024-03-16T19:30:17.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b2ffd640 1 -- 172.21.15.172:0/539967229 learned_addr learned my addr 172.21.15.172:0/539967229 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:17.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b37fe640 1 --2- 172.21.15.172:0/539967229 >> v2:172.21.15.174:3300/0 conn(0x7fb7b40c4a20 0x7fb7b415b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b3fff640 1 -- 172.21.15.172:0/539967229 >> v2:172.21.15.174:3300/0 conn(0x7fb7b40c4a20 msgr2=0x7fb7b415b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:17.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b3fff640 1 --2- 172.21.15.172:0/539967229 >> v2:172.21.15.174:3300/0 conn(0x7fb7b40c4a20 0x7fb7b415b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b3fff640 1 -- 172.21.15.172:0/539967229 >> v1:172.21.15.172:6789/0 conn(0x7fb7b406fe70 legacy=0x7fb7b40ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b3fff640 1 -- 172.21.15.172:0/539967229 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b406bfe0 con 0x7fb7b415ded0 2024-03-16T19:30:17.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.631+0000 7fb7b27fc640 1 -- 172.21.15.172:0/539967229 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961447069 0 0) 0x7fb7a8001110 con 0x7fb7b406fe70 2024-03-16T19:30:17.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.632+0000 7fb7b3fff640 1 --2- 172.21.15.172:0/539967229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b415ded0 0x7fb7b415e320 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40cf130 tx=0x7fb7ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e1b7624217717a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.632+0000 7fb7b27fc640 1 -- 172.21.15.172:0/539967229 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac02cc80 con 0x7fb7b415ded0 2024-03-16T19:30:17.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.632+0000 7fb7b27fc640 1 -- 172.21.15.172:0/539967229 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ac02cde0 con 0x7fb7b415ded0 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.632+0000 7fb7b27fc640 1 -- 172.21.15.172:0/539967229 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac034660 con 0x7fb7b415ded0 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.633+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/539967229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b415ded0 msgr2=0x7fb7b415e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.633+0000 7fb7b9bb5640 1 --2- 172.21.15.172:0/539967229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b415ded0 0x7fb7b415e320 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40cf130 tx=0x7fb7ac02bed0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.633+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/539967229 shutdown_connections 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.633+0000 7fb7b9bb5640 1 --2- 172.21.15.172:0/539967229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b415ded0 0x7fb7b415e320 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.633+0000 7fb7b9bb5640 1 --2- 172.21.15.172:0/539967229 >> v2:172.21.15.174:3300/0 conn(0x7fb7b40c4a20 0x7fb7b415b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.633+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/539967229 >> 172.21.15.172:0/539967229 conn(0x7fb7b4066c20 msgr2=0x7fb7b4067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.633+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/539967229 shutdown_connections 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.633+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/539967229 wait complete. 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.634+0000 7fb7b9bb5640 1 Processor -- start 2024-03-16T19:30:17.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.634+0000 7fb7b9bb5640 1 -- start start 2024-03-16T19:30:17.617 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b9bb5640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b406fe70 0x7fb7b4152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.617 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b9bb5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7b40c4a20 0x7fb7b40af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.617 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b9bb5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b415ded0 0x7fb7b41f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b9bb5640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b415c8d0 con 0x7fb7b406fe70 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b9bb5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b4150b70 con 0x7fb7b40c4a20 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b9bb5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b415bac0 con 0x7fb7b415ded0 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b406fe70 0x7fb7b4152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b406fe70 0x7fb7b4152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39802/0 (socket says 172.21.15.172:39802) 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b37fe640 1 -- 172.21.15.172:0/2717163131 learned_addr learned my addr 172.21.15.172:0/2717163131 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b2ffd640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7b40c4a20 0x7fb7b40af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b3fff640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b415ded0 0x7fb7b41f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b37fe640 1 -- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b415ded0 msgr2=0x7fb7b41f47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b37fe640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b415ded0 0x7fb7b41f47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b37fe640 1 -- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7b40c4a20 msgr2=0x7fb7b40af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b37fe640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7b40c4a20 0x7fb7b40af1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b37fe640 1 -- 172.21.15.172:0/2717163131 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac009a70 con 0x7fb7b406fe70 2024-03-16T19:30:17.618 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b37fe640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b406fe70 0x7fb7b4152180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb7a400e9b0 tx=0x7fb7a400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb79bfff640 1 -- 172.21.15.172:0/2717163131 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a4009670 con 0x7fb7b406fe70 2024-03-16T19:30:17.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb79bfff640 1 -- 172.21.15.172:0/2717163131 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a400c9a0 con 0x7fb7b406fe70 2024-03-16T19:30:17.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b415bd80 con 0x7fb7b406fe70 2024-03-16T19:30:17.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.635+0000 7fb79bfff640 1 -- 172.21.15.172:0/2717163131 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a400cd50 con 0x7fb7b406fe70 2024-03-16T19:30:17.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.636+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b41f3d10 con 0x7fb7b406fe70 2024-03-16T19:30:17.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.636+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb7b415e610 con 0x7fb7b406fe70 2024-03-16T19:30:17.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.637+0000 7fb79bfff640 1 -- 172.21.15.172:0/2717163131 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb7a4013470 con 0x7fb7b406fe70 2024-03-16T19:30:17.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.637+0000 7fb79bfff640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb7a006b3e0 0x7fb7a006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.637+0000 7fb7b2ffd640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb7a006b3e0 0x7fb7a006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.637+0000 7fb79bfff640 1 -- 172.21.15.172:0/2717163131 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fb7a408a1d0 con 0x7fb7b406fe70 2024-03-16T19:30:17.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.638+0000 7fb79bfff640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb7a0073760 0x7fb7a0075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.638+0000 7fb79bfff640 1 -- 172.21.15.172:0/2717163131 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7a00762b0 con 0x7fb7a0073760 2024-03-16T19:30:17.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.638+0000 7fb79bfff640 1 -- 172.21.15.172:0/2717163131 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7a408a620 con 0x7fb7b406fe70 2024-03-16T19:30:17.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.638+0000 7fb7b3fff640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb7a0073760 0x7fb7a0075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.638+0000 7fb7b2ffd640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb7a006b3e0 0x7fb7a006d8a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4067c00 tx=0x7fb7a8007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.638+0000 7fb7b3fff640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb7a0073760 0x7fb7a0075be0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.639+0000 7fb79bfff640 1 -- 172.21.15.172:0/2717163131 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb7a00762b0 con 0x7fb7a0073760 2024-03-16T19:30:17.636 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.655+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7fb7b406bb10 con 0x7fb7a0073760 2024-03-16T19:30:17.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.656+0000 7fb79bfff640 1 -- 172.21.15.172:0/2717163131 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb7b406bb10 con 0x7fb7a0073760 2024-03-16T19:30:17.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.656+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb7a0073760 msgr2=0x7fb7a0075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.656+0000 7fb7b9bb5640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fb7a0073760 0x7fb7a0075be0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.656+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb7a006b3e0 msgr2=0x7fb7a006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.656+0000 7fb7b9bb5640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb7a006b3e0 0x7fb7a006d8a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4067c00 tx=0x7fb7a8007de0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.656+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b406fe70 msgr2=0x7fb7b4152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.656+0000 7fb7b9bb5640 1 --2- 172.21.15.172:0/2717163131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb7b406fe70 0x7fb7b4152180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb7a400e9b0 tx=0x7fb7a400ee70 comp rx=0 tx=0).stop 2024-03-16T19:30:17.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.656+0000 7fb7b3fff640 1 -- 172.21.15.172:0/2717163131 reap_dead start 2024-03-16T19:30:17.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.657+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 shutdown_connections 2024-03-16T19:30:17.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.657+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 >> 172.21.15.172:0/2717163131 conn(0x7fb7b4066c20 msgr2=0x7fb7b4071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.657+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 shutdown_connections 2024-03-16T19:30:17.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.657+0000 7fb7b9bb5640 1 -- 172.21.15.172:0/2717163131 wait complete. 2024-03-16T19:30:17.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-03-16T19:30:17.685 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:17.685 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:17.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:17.738 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:17.757+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:17.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.810+0000 7f8e039e2640 1 Processor -- start 2024-03-16T19:30:17.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.810+0000 7f8e039e2640 1 -- start start 2024-03-16T19:30:17.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e039e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc152a20 0x7f8dfc0c8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e039e2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8dfc0c8090 0x7f8dfc0c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e039e2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc158ac0 con 0x7f8dfc158f50 2024-03-16T19:30:17.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e039e2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dfc06b9f0 con 0x7f8dfc0c8090 2024-03-16T19:30:17.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e039e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc06bcb0 con 0x7f8dfc152a20 2024-03-16T19:30:17.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e0177e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc152a20 0x7f8dfc0c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e00f7d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8dfc0c8090 0x7f8dfc0c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e0177e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc152a20 0x7f8dfc0c8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44074/0 (socket says 172.21.15.172:44074) 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e00f7d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8dfc0c8090 0x7f8dfc0c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:56164/0 (socket says 172.21.15.172:56164) 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e01f7f640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f8dfc158f50 0x7f8dfc0c6a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:52556/0 (socket says 172.21.15.172:52556) 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e0177e640 1 -- 172.21.15.172:0/3637126864 learned_addr learned my addr 172.21.15.172:0/3637126864 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e0177e640 1 -- 172.21.15.172:0/3637126864 >> v2:172.21.15.174:3300/0 conn(0x7f8dfc0c8090 msgr2=0x7f8dfc0c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e0177e640 1 --2- 172.21.15.172:0/3637126864 >> v2:172.21.15.174:3300/0 conn(0x7f8dfc0c8090 0x7f8dfc0c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e0177e640 1 -- 172.21.15.172:0/3637126864 >> v1:172.21.15.174:6789/0 conn(0x7f8dfc158f50 legacy=0x7f8dfc0c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e0177e640 1 -- 172.21.15.172:0/3637126864 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc152e00 con 0x7f8dfc152a20 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8df3fff640 1 -- 172.21.15.172:0/3637126864 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818947670 0 0) 0x7f8dfc158ac0 con 0x7f8dfc158f50 2024-03-16T19:30:17.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.811+0000 7f8e00f7d640 1 --2- 172.21.15.172:0/3637126864 >> v2:172.21.15.174:3300/0 conn(0x7f8dfc0c8090 0x7f8dfc0c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:17.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.812+0000 7f8e0177e640 1 --2- 172.21.15.172:0/3637126864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc152a20 0x7f8dfc0c8550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009e50 tx=0x7f8df802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d2db778141b7afb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.812+0000 7f8df3fff640 1 -- 172.21.15.172:0/3637126864 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df802cb90 con 0x7f8dfc152a20 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.812+0000 7f8df3fff640 1 -- 172.21.15.172:0/3637126864 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df802ccf0 con 0x7f8dfc152a20 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.812+0000 7f8df3fff640 1 -- 172.21.15.172:0/3637126864 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8034610 con 0x7f8dfc152a20 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.813+0000 7f8e039e2640 1 -- 172.21.15.172:0/3637126864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc152a20 msgr2=0x7f8dfc0c8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.813+0000 7f8e039e2640 1 --2- 172.21.15.172:0/3637126864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc152a20 0x7f8dfc0c8550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009e50 tx=0x7f8df802bff0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.813+0000 7f8e039e2640 1 -- 172.21.15.172:0/3637126864 shutdown_connections 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.813+0000 7f8e039e2640 1 --2- 172.21.15.172:0/3637126864 >> v2:172.21.15.174:3300/0 conn(0x7f8dfc0c8090 0x7f8dfc0c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.813+0000 7f8e039e2640 1 --2- 172.21.15.172:0/3637126864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc152a20 0x7f8dfc0c8550 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.813+0000 7f8e039e2640 1 -- 172.21.15.172:0/3637126864 >> 172.21.15.172:0/3637126864 conn(0x7f8dfc0cc030 msgr2=0x7f8dfc0cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.813+0000 7f8e039e2640 1 -- 172.21.15.172:0/3637126864 shutdown_connections 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.813+0000 7f8e039e2640 1 -- 172.21.15.172:0/3637126864 wait complete. 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.814+0000 7f8e039e2640 1 Processor -- start 2024-03-16T19:30:17.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.814+0000 7f8e039e2640 1 -- start start 2024-03-16T19:30:17.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e039e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc0c8090 0x7f8dfc14d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e039e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dfc152a20 0x7f8dfc14d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e039e2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dfc158f50 0x7f8dfc15baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e039e2640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc14ef30 con 0x7f8dfc158f50 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e039e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc152160 con 0x7f8dfc152a20 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e039e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc14ccc0 con 0x7f8dfc0c8090 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e01f7f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dfc158f50 0x7f8dfc15baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e0177e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc0c8090 0x7f8dfc14d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e01f7f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dfc158f50 0x7f8dfc15baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39804/0 (socket says 172.21.15.172:39804) 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e01f7f640 1 -- 172.21.15.172:0/3442096248 learned_addr learned my addr 172.21.15.172:0/3442096248 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e00f7d640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dfc152a20 0x7f8dfc14d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e0177e640 1 -- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dfc152a20 msgr2=0x7f8dfc14d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e0177e640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dfc152a20 0x7f8dfc14d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e0177e640 1 -- 172.21.15.172:0/3442096248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dfc158f50 msgr2=0x7f8dfc15baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e0177e640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dfc158f50 0x7f8dfc15baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e0177e640 1 -- 172.21.15.172:0/3442096248 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec009400 con 0x7f8dfc0c8090 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e00f7d640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dfc152a20 0x7f8dfc14d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.815+0000 7f8e01f7f640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dfc158f50 0x7f8dfc15baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.816+0000 7f8e0177e640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc0c8090 0x7f8dfc14d3f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc06bdf0 tx=0x7f8df8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.816+0000 7f8df1ffb640 1 -- 172.21.15.172:0/3442096248 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8009510 con 0x7f8dfc0c8090 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.816+0000 7f8df1ffb640 1 -- 172.21.15.172:0/3442096248 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df8003990 con 0x7f8dfc0c8090 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.816+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df8009b30 con 0x7f8dfc0c8090 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.816+0000 7f8df1ffb640 1 -- 172.21.15.172:0/3442096248 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8003d40 con 0x7f8dfc0c8090 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.816+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc151340 con 0x7f8dfc0c8090 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.817+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8dfc06cc60 con 0x7f8dfc0c8090 2024-03-16T19:30:17.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.817+0000 7f8df1ffb640 1 -- 172.21.15.172:0/3442096248 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8df8004030 con 0x7f8dfc0c8090 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.818+0000 7f8df1ffb640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8de006b3e0 0x7f8de006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.818+0000 7f8df1ffb640 1 -- 172.21.15.172:0/3442096248 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f8df80acff0 con 0x7f8dfc0c8090 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.818+0000 7f8e00f7d640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8de006b3e0 0x7f8de006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.818+0000 7f8df1ffb640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f8de0073700 0x7f8de0075b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.818+0000 7f8df1ffb640 1 -- 172.21.15.172:0/3442096248 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8de0076250 con 0x7f8de0073700 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.818+0000 7f8df1ffb640 1 -- 172.21.15.172:0/3442096248 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8df80ad400 con 0x7f8dfc0c8090 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.818+0000 7f8e01f7f640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f8de0073700 0x7f8de0075b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.818+0000 7f8e00f7d640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8de006b3e0 0x7f8de006d8a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc145630 tx=0x7f8dec009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.818+0000 7f8e01f7f640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f8de0073700 0x7f8de0075b80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:17.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.819+0000 7f8df1ffb640 1 -- 172.21.15.172:0/3442096248 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8de0076250 con 0x7f8de0073700 2024-03-16T19:30:17.802 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:17.802 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:17.807 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:17.826+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:17.811 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:17.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.836+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) v1 -- 0x7f8dfc073440 con 0x7f8de0073700 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.836+0000 7f8df1ffb640 1 -- 172.21.15.172:0/3442096248 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8df4007300 con 0x7f8de0073700 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f8de0073700 msgr2=0x7f8de0075b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f8de0073700 0x7f8de0075b80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8de006b3e0 msgr2=0x7f8de006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8de006b3e0 0x7f8de006d8a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc145630 tx=0x7f8dec009190 comp rx=0 tx=0).stop 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc0c8090 msgr2=0x7f8dfc14d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 --2- 172.21.15.172:0/3442096248 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8dfc0c8090 0x7f8dfc14d3f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc06bdf0 tx=0x7f8df8004a40 comp rx=0 tx=0).stop 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e01f7f640 1 -- 172.21.15.172:0/3442096248 reap_dead start 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 shutdown_connections 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 >> 172.21.15.172:0/3442096248 conn(0x7f8dfc0cc030 msgr2=0x7f8dfc156960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 shutdown_connections 2024-03-16T19:30:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:17.837+0000 7f8e039e2640 1 -- 172.21.15.172:0/3442096248 wait complete. 2024-03-16T19:30:17.907 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:17.926+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:17.957 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:17.957 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:17.965 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:18.008 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:18.026+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:18.108 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:18.127+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:18.116 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:18.116 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:18.125 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:18.209 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:18.228+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:18.272 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:18.272 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:18.282 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:18.310 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:18.330+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:18.410 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:18.429+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:18.430 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:18.430 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:18.438 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:18.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:30:18.522 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:18.541+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:18.585 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:18.585 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:18.593 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:18.613 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:18.633+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:18.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.640+0000 7f9821748640 1 Processor -- start 2024-03-16T19:30:18.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.640+0000 7f9821748640 1 -- start start 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.641+0000 7f9821748640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c154b60 0x7f981c0ca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.641+0000 7f9821748640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f981c06f040 0x7f981c15af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.641+0000 7f9821748640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981c15ed90 con 0x7f981c15b450 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.641+0000 7f9821748640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981c06de30 con 0x7f981c06f040 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.641+0000 7f9821748640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c15a7e0 con 0x7f981c154b60 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.641+0000 7f981b7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f981c15b450 0x7f981c0ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40116/0 (socket says 172.21.15.172:40116) 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.641+0000 7f981b7fe640 1 -- 172.21.15.172:0/583199080 learned_addr learned my addr 172.21.15.172:0/583199080 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.641+0000 7f981affd640 1 --2- 172.21.15.172:0/583199080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c154b60 0x7f981c0ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.641+0000 7f981a7fc640 1 --2- 172.21.15.172:0/583199080 >> v2:172.21.15.174:3300/0 conn(0x7f981c06f040 0x7f981c15af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.642+0000 7f9819ffb640 1 -- 172.21.15.172:0/583199080 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265913243 0 0) 0x7f981c15ed90 con 0x7f981c15b450 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.642+0000 7f9819ffb640 1 -- 172.21.15.172:0/583199080 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9804003520 con 0x7f981c15b450 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.642+0000 7f981affd640 1 -- 172.21.15.172:0/583199080 >> v2:172.21.15.174:3300/0 conn(0x7f981c06f040 msgr2=0x7f981c15af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.642+0000 7f981affd640 1 --2- 172.21.15.172:0/583199080 >> v2:172.21.15.174:3300/0 conn(0x7f981c06f040 0x7f981c15af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.642+0000 7f981affd640 1 -- 172.21.15.172:0/583199080 >> v1:172.21.15.172:6789/0 conn(0x7f981c15b450 legacy=0x7f981c0ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.642+0000 7f981affd640 1 -- 172.21.15.172:0/583199080 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c0ca970 con 0x7f981c154b60 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.642+0000 7f981a7fc640 1 --2- 172.21.15.172:0/583199080 >> v2:172.21.15.174:3300/0 conn(0x7f981c06f040 0x7f981c15af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.642+0000 7f981affd640 1 --2- 172.21.15.172:0/583199080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c154b60 0x7f981c0ca430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f980c009d90 tx=0x7f980c02bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fed1eebb15e2708 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9819ffb640 1 -- 172.21.15.172:0/583199080 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c02cce0 con 0x7f981c154b60 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9819ffb640 1 -- 172.21.15.172:0/583199080 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c02ce40 con 0x7f981c154b60 2024-03-16T19:30:18.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9819ffb640 1 -- 172.21.15.172:0/583199080 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c034660 con 0x7f981c154b60 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9821748640 1 -- 172.21.15.172:0/583199080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c154b60 msgr2=0x7f981c0ca430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9821748640 1 --2- 172.21.15.172:0/583199080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c154b60 0x7f981c0ca430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f980c009d90 tx=0x7f980c02bf30 comp rx=0 tx=0).stop 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9821748640 1 -- 172.21.15.172:0/583199080 shutdown_connections 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9821748640 1 --2- 172.21.15.172:0/583199080 >> v2:172.21.15.174:3300/0 conn(0x7f981c06f040 0x7f981c15af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9821748640 1 --2- 172.21.15.172:0/583199080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c154b60 0x7f981c0ca430 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9821748640 1 -- 172.21.15.172:0/583199080 >> 172.21.15.172:0/583199080 conn(0x7f981c066e00 msgr2=0x7f981c0671e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9821748640 1 -- 172.21.15.172:0/583199080 shutdown_connections 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.643+0000 7f9821748640 1 -- 172.21.15.172:0/583199080 wait complete. 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.644+0000 7f9821748640 1 Processor -- start 2024-03-16T19:30:18.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.644+0000 7f9821748640 1 -- start start 2024-03-16T19:30:18.627 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f9821748640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981c06f040 0x7f981c152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.627 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f9821748640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f981c154b60 0x7f981c0aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.627 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f9821748640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c15b450 0x7f981c14d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f9821748640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c1574b0 con 0x7f981c06f040 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f9821748640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c15d7f0 con 0x7f981c154b60 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f9821748640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c157aa0 con 0x7f981c15b450 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f981affd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981c06f040 0x7f981c152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f981affd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981c06f040 0x7f981c152fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42354/0 (socket says 172.21.15.172:42354) 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f981affd640 1 -- 172.21.15.172:0/458734783 learned_addr learned my addr 172.21.15.172:0/458734783 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f981b7fe640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c15b450 0x7f981c14d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.645+0000 7f981a7fc640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f981c154b60 0x7f981c0aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f981affd640 1 -- 172.21.15.172:0/458734783 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c15b450 msgr2=0x7f981c14d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f981affd640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c15b450 0x7f981c14d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f981affd640 1 -- 172.21.15.172:0/458734783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f981c154b60 msgr2=0x7f981c0aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f981affd640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f981c154b60 0x7f981c0aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f981affd640 1 -- 172.21.15.172:0/458734783 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810009400 con 0x7f981c06f040 2024-03-16T19:30:18.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f981affd640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981c06f040 0x7f981c152fb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f980c009d90 tx=0x7f980c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.629 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f98037fe640 1 -- 172.21.15.172:0/458734783 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c03e670 con 0x7f981c06f040 2024-03-16T19:30:18.629 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f98037fe640 1 -- 172.21.15.172:0/458734783 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c03e7d0 con 0x7f981c06f040 2024-03-16T19:30:18.629 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c009a70 con 0x7f981c06f040 2024-03-16T19:30:18.629 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f98037fe640 1 -- 172.21.15.172:0/458734783 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c03eb80 con 0x7f981c06f040 2024-03-16T19:30:18.629 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.646+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c15cc00 con 0x7f981c06f040 2024-03-16T19:30:18.629 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.647+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c15d930 con 0x7f981c06f040 2024-03-16T19:30:18.630 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.647+0000 7f98037fe640 1 -- 172.21.15.172:0/458734783 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f980c03b040 con 0x7f981c06f040 2024-03-16T19:30:18.630 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.648+0000 7f98037fe640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980806b3b0 0x7f980806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.630 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.648+0000 7f981a7fc640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980806b3b0 0x7f980806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.630 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.648+0000 7f98037fe640 1 -- 172.21.15.172:0/458734783 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f980c0ac180 con 0x7f981c06f040 2024-03-16T19:30:18.630 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.648+0000 7f981a7fc640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980806b3b0 0x7f980806d870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f981c151840 tx=0x7f9810009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.652+0000 7f98037fe640 1 -- 172.21.15.172:0/458734783 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f981c15d930 con 0x7f981c06f040 2024-03-16T19:30:18.714 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:18.732+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:18.741 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:18.741 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:18.749 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:18.812 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.831+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f981c15d930 con 0x7f981c06f040 2024-03-16T19:30:18.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:18.832 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:18.851+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:18.896 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:18.896 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:18.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:18.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.932+0000 7f9ea8125640 1 Processor -- start 2024-03-16T19:30:18.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.933+0000 7f9ea8125640 1 -- start start 2024-03-16T19:30:18.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.933+0000 7f9ea8125640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9ea0154d00 0x7f9ea015de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.933+0000 7f9ea8125640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea00c7510 con 0x7f9ea0071e50 2024-03-16T19:30:18.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.933+0000 7f9ea8125640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea006b910 con 0x7f9ea0072230 2024-03-16T19:30:18.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.933+0000 7f9ea8125640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea006afe0 con 0x7f9ea0154d00 2024-03-16T19:30:18.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.933+0000 7f9ea66c2640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9ea0154d00 0x7f9ea015de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.933+0000 7f9ea5ec1640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9ea0071e50 0x7f9ea00c7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40128/0 (socket says 172.21.15.172:40128) 2024-03-16T19:30:18.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.933+0000 7f9ea5ec1640 1 -- 172.21.15.172:0/268559445 learned_addr learned my addr 172.21.15.172:0/268559445 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:18.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.934+0000 7f9ea66c2640 1 -- 172.21.15.172:0/268559445 >> v1:172.21.15.174:6789/0 conn(0x7f9ea0072230 legacy=0x7f9ea01545f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:18.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.934+0000 7f9ea66c2640 1 -- 172.21.15.172:0/268559445 >> v1:172.21.15.172:6789/0 conn(0x7f9ea0071e50 legacy=0x7f9ea00c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:18.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.934+0000 7f9ea66c2640 1 -- 172.21.15.172:0/268559445 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea006b2c0 con 0x7f9ea0154d00 2024-03-16T19:30:18.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.934+0000 7f9ea4ebf640 1 -- 172.21.15.172:0/268559445 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155468258 0 0) 0x7f9ea00c7510 con 0x7f9ea0071e50 2024-03-16T19:30:18.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.934+0000 7f9ea66c2640 1 --2- 172.21.15.172:0/268559445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 0x7f9ea015de30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009d90 tx=0x7f9e9802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205228f7e2b7227e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea4ebf640 1 -- 172.21.15.172:0/268559445 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9802cd80 con 0x7f9ea0154d00 2024-03-16T19:30:18.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea4ebf640 1 -- 172.21.15.172:0/268559445 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e98034440 con 0x7f9ea0154d00 2024-03-16T19:30:18.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea4ebf640 1 -- 172.21.15.172:0/268559445 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e980347d0 con 0x7f9ea0154d00 2024-03-16T19:30:18.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea8125640 1 -- 172.21.15.172:0/268559445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 msgr2=0x7f9ea015de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:18.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea8125640 1 --2- 172.21.15.172:0/268559445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 0x7f9ea015de30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009d90 tx=0x7f9e9802bfd0 comp rx=0 tx=0).stop 2024-03-16T19:30:18.916 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:18.935+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:18.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea8125640 1 -- 172.21.15.172:0/268559445 shutdown_connections 2024-03-16T19:30:18.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea8125640 1 --2- 172.21.15.172:0/268559445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 0x7f9ea015de30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:18.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea8125640 1 -- 172.21.15.172:0/268559445 >> 172.21.15.172:0/268559445 conn(0x7f9ea0072b40 msgr2=0x7f9ea0072f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:18.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea8125640 1 -- 172.21.15.172:0/268559445 shutdown_connections 2024-03-16T19:30:18.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.935+0000 7f9ea8125640 1 -- 172.21.15.172:0/268559445 wait complete. 2024-03-16T19:30:18.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea8125640 1 Processor -- start 2024-03-16T19:30:18.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea8125640 1 -- start start 2024-03-16T19:30:18.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea8125640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea0071e50 0x7f9ea015ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea8125640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea0072230 0x7f9ea01f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea8125640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 0x7f9ea01f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea8125640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea015d7d0 con 0x7f9ea0154d00 2024-03-16T19:30:18.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea8125640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00c5e30 con 0x7f9ea0071e50 2024-03-16T19:30:18.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea8125640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00c6110 con 0x7f9ea0072230 2024-03-16T19:30:18.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea66c2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 0x7f9ea01f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea66c2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 0x7f9ea01f1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42374/0 (socket says 172.21.15.172:42374) 2024-03-16T19:30:18.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea66c2640 1 -- 172.21.15.172:0/884099876 learned_addr learned my addr 172.21.15.172:0/884099876 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea5ec1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea0071e50 0x7f9ea015ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.937+0000 7f9ea56c0640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea0072230 0x7f9ea01f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9ea66c2640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea0072230 msgr2=0x7f9ea01f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9ea66c2640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea0072230 0x7f9ea01f1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9ea66c2640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea0071e50 msgr2=0x7f9ea015ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9ea66c2640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea0071e50 0x7f9ea015ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9ea66c2640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e98009a70 con 0x7f9ea0154d00 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9ea5ec1640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea0071e50 0x7f9ea015ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9ea66c2640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 0x7f9ea01f1c70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9e98037040 tx=0x7f9e98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98003b90 con 0x7f9ea0154d00 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e98003d10 con 0x7f9ea0154d00 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea00c63f0 con 0x7f9ea0154d00 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98004270 con 0x7f9ea0154d00 2024-03-16T19:30:18.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.938+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea00c69b0 con 0x7f9ea0154d00 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.939+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9ea00688d0 con 0x7f9ea0154d00 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.940+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9e980043d0 con 0x7f9ea0154d00 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.940+0000 7f9e967fc640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9e8406b3b0 0x7f9e8406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.940+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f9e980ac550 con 0x7f9ea0154d00 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.940+0000 7f9ea5ec1640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9e8406b3b0 0x7f9e8406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.940+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v36) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9e980b3070 con 0x7f9ea0154d00 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.940+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9ea00c6fa0 0x7f9ea01527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.940+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ea00c82c0 con 0x7f9ea00c6fa0 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.941+0000 7f9ea5ec1640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9e8406b3b0 0x7f9e8406d870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c007f70 tx=0x7f9e9c007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.941+0000 7f9ea56c0640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9ea00c6fa0 0x7f9ea01527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.941+0000 7f9ea56c0640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9ea00c6fa0 0x7f9ea01527c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.942+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ea00c82c0 con 0x7f9ea00c6fa0 2024-03-16T19:30:18.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.957+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ea00745a0 con 0x7f9ea00c6fa0 2024-03-16T19:30:18.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.958+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9ea00745a0 con 0x7f9ea00c6fa0 2024-03-16T19:30:18.939 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:18.939 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:18.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.958+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9ea0145320 0x7f9ea0145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.958+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ea006c240 con 0x7f9ea0145320 2024-03-16T19:30:18.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.959+0000 7f9ea5ec1640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9ea0145320 0x7f9ea0145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.959+0000 7f9ea5ec1640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9ea0145320 0x7f9ea0145700 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.959+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ea006c240 con 0x7f9ea0145320 2024-03-16T19:30:18.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.975+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ea0145c40 con 0x7f9ea0145320 2024-03-16T19:30:18.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.977+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9ea0145c40 con 0x7f9ea0145320 2024-03-16T19:30:18.957 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:18.958 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:18.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.977+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9ea01f3cd0 0x7f9ea0061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.977+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ea01f40b0 con 0x7f9ea01f3cd0 2024-03-16T19:30:18.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.977+0000 7f9ea56c0640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9ea01f3cd0 0x7f9ea0061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.977+0000 7f9ea56c0640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9ea01f3cd0 0x7f9ea0061930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.978+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ea01f40b0 con 0x7f9ea01f3cd0 2024-03-16T19:30:18.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.992+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ea0062140 con 0x7f9ea01f3cd0 2024-03-16T19:30:18.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.994+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9ea0062140 con 0x7f9ea01f3cd0 2024-03-16T19:30:18.974 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:18.975 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:18.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.994+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9ea00626d0 0x7f9ea01511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.994+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ea0151700 con 0x7f9ea00626d0 2024-03-16T19:30:18.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.994+0000 7f9ea66c2640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9ea00626d0 0x7f9ea01511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.994+0000 7f9ea66c2640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9ea00626d0 0x7f9ea01511c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:18.995+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ea0151700 con 0x7f9ea00626d0 2024-03-16T19:30:18.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.009+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ea0151b50 con 0x7f9ea00626d0 2024-03-16T19:30:18.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.011+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9ea0151b50 con 0x7f9ea00626d0 2024-03-16T19:30:18.992 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:18.992 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:18.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.011+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9ea002c2f0 0x7f9ea002c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:18.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.011+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ea0151cc0 con 0x7f9ea002c2f0 2024-03-16T19:30:18.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.012+0000 7f9ea5ec1640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9ea002c2f0 0x7f9ea002c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:18.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.012+0000 7f9ea5ec1640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9ea002c2f0 0x7f9ea002c710 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:18.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.013+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ea0151cc0 con 0x7f9ea002c2f0 2024-03-16T19:30:19.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.027+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ea0151cc0 con 0x7f9ea002c2f0 2024-03-16T19:30:19.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.028+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9ea0151cc0 con 0x7f9ea002c2f0 2024-03-16T19:30:19.009 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:19.010 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:19.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.029+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9ea02049e0 0x7f9ea0206dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:19.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.029+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ea002d160 con 0x7f9ea02049e0 2024-03-16T19:30:19.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.029+0000 7f9ea56c0640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9ea02049e0 0x7f9ea0206dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:19.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.029+0000 7f9ea56c0640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9ea02049e0 0x7f9ea0206dd0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:19.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.030+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ea002d160 con 0x7f9ea02049e0 2024-03-16T19:30:19.016 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:19.016 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:19.017 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:19.017 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:19.017 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:19.036+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:19.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.044+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ea002d360 con 0x7f9ea02049e0 2024-03-16T19:30:19.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:19.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.046+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9ea002d360 con 0x7f9ea02049e0 2024-03-16T19:30:19.026 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:19.027 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:19.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.046+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9ea0207820 0x7f9ea0209c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:19.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.046+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ea002d2d0 con 0x7f9ea0207820 2024-03-16T19:30:19.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.046+0000 7f9ea66c2640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9ea0207820 0x7f9ea0209c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:19.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.047+0000 7f9ea66c2640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9ea0207820 0x7f9ea0209c10 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:19.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.047+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ea002d2d0 con 0x7f9ea0207820 2024-03-16T19:30:19.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.062+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ea002d2d0 con 0x7f9ea0207820 2024-03-16T19:30:19.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.063+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9ea002d2d0 con 0x7f9ea0207820 2024-03-16T19:30:19.044 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:19.045 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:19.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.064+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9ea020a660 0x7f9ea020ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:19.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.064+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ea020f820 con 0x7f9ea020a660 2024-03-16T19:30:19.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.064+0000 7f9ea5ec1640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9ea020a660 0x7f9ea020ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:19.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.064+0000 7f9ea5ec1640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9ea020a660 0x7f9ea020ca50 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:19.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.065+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ea020f820 con 0x7f9ea020a660 2024-03-16T19:30:19.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.079+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9ea020cf90 con 0x7f9ea020a660 2024-03-16T19:30:19.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.081+0000 7f9e967fc640 1 -- 172.21.15.172:0/884099876 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9ea020cf90 con 0x7f9ea020a660 2024-03-16T19:30:19.062 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:19.063 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:19.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.081+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9ea00c6fa0 msgr2=0x7f9ea01527c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.081+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9ea00c6fa0 0x7f9ea01527c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.081+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9ea0145320 msgr2=0x7f9ea0145700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.081+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9ea0145320 0x7f9ea0145700 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.081+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9ea01f3cd0 msgr2=0x7f9ea0061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.081+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9ea01f3cd0 0x7f9ea0061930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9ea00626d0 msgr2=0x7f9ea01511c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9ea00626d0 0x7f9ea01511c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea66c2640 1 -- 172.21.15.172:0/884099876 reap_dead start 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9ea002c2f0 msgr2=0x7f9ea002c710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9ea002c2f0 0x7f9ea002c710 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea66c2640 1 -- 172.21.15.172:0/884099876 reap_dead start 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9ea02049e0 msgr2=0x7f9ea0206dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9ea02049e0 0x7f9ea0206dd0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9ea0207820 msgr2=0x7f9ea0209c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9ea0207820 0x7f9ea0209c10 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9ea020a660 msgr2=0x7f9ea020ca50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9ea020a660 0x7f9ea020ca50 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9e8406b3b0 msgr2=0x7f9e8406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9e8406b3b0 0x7f9e8406d870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c007f70 tx=0x7f9e9c007f00 comp rx=0 tx=0).stop 2024-03-16T19:30:19.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 msgr2=0x7f9ea01f1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 --2- 172.21.15.172:0/884099876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ea0154d00 0x7f9ea01f1c70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9e98037040 tx=0x7f9e98004a40 comp rx=0 tx=0).stop 2024-03-16T19:30:19.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea66c2640 1 -- 172.21.15.172:0/884099876 reap_dead start 2024-03-16T19:30:19.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 shutdown_connections 2024-03-16T19:30:19.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 >> 172.21.15.172:0/884099876 conn(0x7f9ea0072b40 msgr2=0x7f9ea0074880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:19.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 shutdown_connections 2024-03-16T19:30:19.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.082+0000 7f9ea8125640 1 -- 172.21.15.172:0/884099876 wait complete. 2024-03-16T19:30:19.117 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:19.136+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:19.135 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:19.135 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:19.135 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:19.135 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:19.135 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:19.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:19.218 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:19.238+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:19.251 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:19.251 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:19.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:19.251 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:19.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.251 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.676780569999998, 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:30:19.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:30:19.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.255 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.886554398999998, 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:30:19.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.259 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.885794164000004, 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:19.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:30:19.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:19.263 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.884980208999998, 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:30:19.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:30:19.267 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.884340283999997, 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.883682186999998, 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.883149611999997, 2024-03-16T19:30:19.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:30:19.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.824977709000002, 2024-03-16T19:30:19.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:30:19.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:30:19.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.824375515, 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:30:19.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.823732569999997, 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.823228608999997, 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.292 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:30:19.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:30:19.296 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 60.665082343000002, 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:30:19.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 60.664552207, 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:30:19.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:30:19.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:30:19.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.912582948000001, 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.912558412000003, 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.912553881999997, 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:19.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:30:19.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:30:19.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.912544070000003, 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:19.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:30:19.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:19.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:19.325 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.912540217, 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:30:19.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:30:19.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.912536901000003, 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:30:19.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.911341317999998, 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.338 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:19.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:19.340 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:19.358+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:19.440 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:19.458+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:19.488 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:19.488 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:19.488 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:19.488 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:19.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:19.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.520+0000 7f98037fe640 1 -- 172.21.15.172:0/458734783 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v37) v1 ==== 87+0+0 (secure 0 0 0) 0x7f981c15d930 con 0x7f981c06f040 2024-03-16T19:30:19.500 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980806b3b0 msgr2=0x7f980806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980806b3b0 0x7f980806d870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f981c151840 tx=0x7f9810009190 comp rx=0 tx=0).stop 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981c06f040 msgr2=0x7f981c152fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981c06f040 0x7f981c152fb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f980c009d90 tx=0x7f980c004a40 comp rx=0 tx=0).stop 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 shutdown_connections 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980806b3b0 0x7f980806d870 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f981c15b450 0x7f981c14d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f981c154b60 0x7f981c0aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 --2- 172.21.15.172:0/458734783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f981c06f040 0x7f981c152fb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 >> 172.21.15.172:0/458734783 conn(0x7f981c066e00 msgr2=0x7f981c0c96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 shutdown_connections 2024-03-16T19:30:19.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:19.522+0000 7f9821748640 1 -- 172.21.15.172:0/458734783 wait complete. 2024-03-16T19:30:19.556 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:19.576+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:19.604 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:19.605 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:19.605 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:19.605 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:19.605 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:19.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:19.642 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:19.660+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:19.743 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:19.762+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:19.762 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:19.762 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.424073683000003, 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:19.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:30:19.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.424056569000001, 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.424049289000003, 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.424035422000003, 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:30:19.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:30:19.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.423362587, 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:19.786 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.421064336999997, 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.421019235999999, 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.420991180999998, 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:30:19.794 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.798 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.420969767000003, 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:19.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:30:19.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.802 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.420947415999997, 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:19.806 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.420927016999997, 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.420821025000002, 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.814 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.419029313999999, 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:30:19.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:30:19.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.418981813000002, 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:30:19.818 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:19.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:30:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.821 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.418742561999998, 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:30 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.418586982999997, 2024-03-16T19:30:19.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126666, 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362824+0000", 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7999999999999995e-07 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362828+0000", 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0879999999999997e-06 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362837+0000", 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.704e-06 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362843+0000", 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9390000000000002e-06 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362877+0000", 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4779e-05 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362883+0000", 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9009999999999999e-06 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362950+0000", 2024-03-16T19:30:19.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6674999999999994e-05 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.410244417000001, 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:19.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:30:19.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:30:19.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.408904540999998, 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:30:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.40591355899999998, 2024-03-16T19:30:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:46 2.6 e36/18 ondisk, result = 0)", 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.376767+0000", 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.404643482, 2024-03-16T19:30:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013778799999999999, 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376767+0000", 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376767+0000", 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376768+0000", 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376771+0000", 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2509999999999999e-06 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376781+0000", 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349999999999999e-05 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376788+0000", 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7900000000000002e-06 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376826+0000", 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7886000000000003e-05 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376832+0000", 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8340000000000004e-06 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376905+0000", 2024-03-16T19:30:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2966999999999994e-05 2024-03-16T19:30:19.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.846 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:19.846 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:19.846 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:19.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:19.847 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:19.867+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:19.947 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:19.966+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:19.997 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:19.997 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:19.997 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:19.997 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:20.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:20.049 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:20.069+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:20.112 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:20.112 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:20.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:20.149 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:20.168+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:20.250 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:20.269+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930693169999998, 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053568110000000004, 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.269 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357146+0000", 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5860000000000001e-06 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357154+0000", 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5439999999999999e-06 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357217+0000", 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2986999999999999e-05 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357229+0000", 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1853e-05 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357343+0000", 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113663 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357504+0000", 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016176 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:20.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013908599999999999 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360705+0000", 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030619419999999998 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362409+0000", 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001704021 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362456+0000", 2024-03-16T19:30:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6140000000000002e-05 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362496+0000", 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0228999999999997e-05 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930673143, 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078702890000000008, 2024-03-16T19:30:20.274 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4299999999999998e-07 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357163+0000", 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2559999999999998e-06 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357254+0000", 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1701999999999997e-05 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357258+0000", 2024-03-16T19:30:20.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5719999999999999e-06 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357931+0000", 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067308000000000005 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357999+0000", 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8323000000000003e-05 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358113+0000", 2024-03-16T19:30:20.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011366699999999999 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362505+0000", 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004391509 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024372970000000002 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364992+0000", 2024-03-16T19:30:20.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9741999999999999e-05 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365029+0000", 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7697999999999997e-05 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930666340999998, 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079944349999999994, 2024-03-16T19:30:20.279 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6e-07 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357172+0000", 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7960000000000001e-06 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357285+0000", 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113183 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357290+0000", 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6190000000000002e-06 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358306+0000", 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010160670000000001 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358400+0000", 2024-03-16T19:30:20.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3854e-05 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358492+0000", 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2954000000000005e-05 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362510+0000", 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040175469999999998 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:30:20.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025680310000000001 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365123+0000", 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4515000000000003e-05 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365160+0000", 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7608999999999999e-05 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930659445000003, 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080787759999999993, 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.36e-07 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1270000000000001e-06 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357297+0000", 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121885 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357300+0000", 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049999999999999e-06 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358709+0000", 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001409083 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358776+0000", 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6489999999999995e-05 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358885+0000", 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109271 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362514+0000", 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036292199999999998 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365197+0000", 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026832100000000001 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365226+0000", 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9136999999999999e-05 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365251+0000", 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5012e-05 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.289 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930656460999998, 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081787609999999997, 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:20.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357177+0000", 2024-03-16T19:30:20.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5090000000000001e-06 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357306+0000", 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012856900000000001 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357308+0000", 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3700000000000002e-06 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359071+0000", 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017631210000000001 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359182+0000", 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110265 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359280+0000", 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8595e-05 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362520+0000", 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032394120000000001 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365297+0000", 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002777542 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365328+0000", 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0899999999999999e-05 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365354+0000", 2024-03-16T19:30:20.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6276000000000001e-05 2024-03-16T19:30:20.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.297 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.297 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.297 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930653773000003, 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083197830000000007, 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357179+0000", 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357187+0000", 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357314+0000", 2024-03-16T19:30:20.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126978 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357316+0000", 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3800000000000001e-06 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359512+0000", 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021950799999999999 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359585+0000", 2024-03-16T19:30:20.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3168000000000001e-05 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359702+0000", 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011741 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362525+0000", 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028225419999999999 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365408+0000", 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028829010000000002 2024-03-16T19:30:20.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365460+0000", 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839e-05 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365498+0000", 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7673999999999999e-05 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930642648000003, 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084493169999999996, 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357190+0000", 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-07 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357191+0000", 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7069999999999999e-06 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357325+0000", 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133585 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357330+0000", 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0799999999999996e-06 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359915+0000", 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025849290000000001 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9741999999999995e-05 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360121+0000", 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362826+0000", 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027046819999999999 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365564+0000", 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027387179999999998 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365606+0000", 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1715999999999998e-05 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365639+0000", 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660999999999998e-05 2024-03-16T19:30:20.307 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930639978999999, 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085533069999999996, 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0699999999999999e-07 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:20.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3030000000000001e-06 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357339+0000", 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014513199999999999 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000003e-06 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030087619999999999 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360414+0000", 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1297999999999995e-05 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360498+0000", 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4538999999999998e-05 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362832+0000", 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023338410000000001 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365670+0000", 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028379550000000001 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365724+0000", 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3533999999999999e-05 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365745+0000", 2024-03-16T19:30:20.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1725999999999998e-05 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930636755000002, 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052850789999999998, 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:20.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357196+0000", 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:20.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.173e-06 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357359+0000", 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016222400000000001 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357365+0000", 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7320000000000003e-06 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357451+0000", 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6494000000000002e-05 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357555+0000", 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010401200000000001 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8224000000000001e-05 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362294+0000", 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046502399999999999 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362388+0000", 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4448000000000001e-05 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362433+0000", 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4888000000000001e-05 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362480+0000", 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7441000000000002e-05 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.930634718, 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053762239999999998, 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357198+0000", 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357199+0000", 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357415+0000", 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021638699999999999 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0069e-05 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358132+0000", 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070650599999999997 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358207+0000", 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5197999999999994e-05 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:20.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358342+0000", 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013503199999999999 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362377+0000", 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040354600000000003 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362493+0000", 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115951 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362536+0000", 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3189999999999998e-05 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362574+0000", 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7113999999999997e-05 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:6 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.323 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.927575816999997, 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045449999999999999, 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360257+0000", 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7400000000000002e-07 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360260+0000", 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3550000000000001e-06 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360270+0000", 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360277+0000", 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5760000000000002e-06 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360697+0000", 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042039200000000001 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360702+0000", 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7140000000000001e-06 2024-03-16T19:30:20.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360711+0000", 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1209999999999996e-06 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:22 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.925631641999999, 2024-03-16T19:30:20.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100673, 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362201+0000", 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5799999999999998e-07 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362206+0000", 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8579999999999999e-06 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362219+0000", 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.239e-05 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362226+0000", 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2710000000000003e-06 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362281+0000", 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4756000000000002e-05 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362288+0000", 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.075e-06 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362301+0000", 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3365e-05 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:23 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.925520372999998, 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017205200000000001, 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.331 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4499999999999997e-07 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362315+0000", 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1350000000000001e-06 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362322+0000", 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9469999999999997e-06 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362328+0000", 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362366+0000", 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8160000000000001e-05 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362373+0000", 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5799999999999997e-06 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362484+0000", 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111173 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:7 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.922976501000001, 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018177499999999999, 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364861+0000", 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3699999999999997e-06 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364875+0000", 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4059e-05 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364881+0000", 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5270000000000004e-06 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364929+0000", 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000002e-05 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3699999999999997e-06 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365037+0000", 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100819 2024-03-16T19:30:20.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:8 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.922919444000001, 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025451199999999998, 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:20.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364916+0000", 2024-03-16T19:30:20.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8109999999999999e-06 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364924+0000", 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1419999999999993e-06 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364930+0000", 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2260000000000002e-06 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365071+0000", 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014030100000000001 2024-03-16T19:30:20.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365075+0000", 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365167+0000", 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2196999999999998e-05 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.342 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:9 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.922893203999998, 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032082900000000001, 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.22e-07 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3980000000000002e-06 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364947+0000", 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3090000000000003e-06 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364951+0000", 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1309999999999999e-06 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365191+0000", 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023992600000000001 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365195+0000", 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.704e-06 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365260+0000", 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5139000000000001e-05 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:10 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.922864818000001, 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039415499999999998, 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.346 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364968+0000", 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364970+0000", 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4650000000000001e-06 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364977+0000", 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.443e-06 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364981+0000", 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2320000000000003e-06 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365289+0000", 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030856900000000002 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365294+0000", 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6920000000000001e-06 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365361+0000", 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7358000000000001e-05 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:11 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.922844318000003, 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051803199999999998, 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9299999999999999e-07 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364993+0000", 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9080000000000003e-06 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4240000000000001e-06 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365002+0000", 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7570000000000002e-06 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365395+0000", 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039237199999999999 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365401+0000", 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9380000000000003e-06 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365506+0000", 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010433999999999999 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:12 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.922822314000001, 2024-03-16T19:30:20.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063430500000000005, 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9500000000000001e-07 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365012+0000", 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7129999999999999e-06 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365015+0000", 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2629999999999999e-06 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365017+0000", 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1749999999999999e-06 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365544+0000", 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052690099999999998 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365559+0000", 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4610000000000001e-05 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365644+0000", 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5447999999999999e-05 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:13 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.922811815999999, 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00073113500000000005, 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1000000000000001e-07 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365021+0000", 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.23e-07 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365024+0000", 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7369999999999998e-06 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365026+0000", 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1289999999999999e-06 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365663+0000", 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063679600000000002 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365667+0000", 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.233e-06 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365751+0000", 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4307000000000004e-05 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.361 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:20.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:20.362 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:20.382+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:20.463 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:20.482+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:20.509 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:20.509 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:20.510 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:20.510 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:20.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:20.563 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:20.582+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:20.664 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:20.683+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:20.665 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:20.665 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:20.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:20.765 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:20.785+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:20.781 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:20.781 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:20.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.443478712000001, 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.443464239000001, 2024-03-16T19:30:20.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.44295975, 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:30:20.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:30:20.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:30:20.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.442944353999998, 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.793 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.442485671, 2024-03-16T19:30:20.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.441939501, 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.441382410000003, 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:30:20.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:30:20.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.440180662000003, 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:30:20.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:30:20.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.439601953, 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:20.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:20.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:30:20.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.439552425999999, 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:20.817 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:30:20.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:30:20.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.438591783, 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.434877585000002, 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:30:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.994476673000001, 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.428923670000003, 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:30:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.921324945000002, 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:20.841 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:20.885 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:20.904+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:20.948 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:20.948 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:20.949 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:20.949 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:20.957 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:20.967 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:20.987+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:21.068 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:21.088+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:21.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:21.131 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:21.131 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:21.132 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:21.132 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:21.132 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:21.143 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:21.169 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:21.188+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:21.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.192+0000 7f6cb5d5c640 1 Processor -- start 2024-03-16T19:30:21.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.192+0000 7f6cb5d5c640 1 -- start start 2024-03-16T19:30:21.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.193+0000 7f6cb5d5c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6cb00c9cc0 0x7f6cb0070880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.193+0000 7f6cb5d5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cb0070dc0 0x7f6cb015b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.193+0000 7f6cb5d5c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00c9380 con 0x7f6cb0154de0 2024-03-16T19:30:21.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.193+0000 7f6cb5d5c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb006d9e0 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.193+0000 7f6cb5d5c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb00cab00 con 0x7f6cb0070dc0 2024-03-16T19:30:21.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.193+0000 7f6caeffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f6cb00c9cc0 0x7f6cb0070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.193+0000 7f6caf7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6cb0154de0 0x7f6cb015ead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40140/0 (socket says 172.21.15.172:40140) 2024-03-16T19:30:21.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.193+0000 7f6caf7fe640 1 -- 172.21.15.172:0/2944416392 learned_addr learned my addr 172.21.15.172:0/2944416392 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:21.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.193+0000 7f6caffff640 1 --2- 172.21.15.172:0/2944416392 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cb0070dc0 0x7f6cb015b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.194+0000 7f6caeffd640 1 -- 172.21.15.172:0/2944416392 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cb0070dc0 msgr2=0x7f6cb015b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:21.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.194+0000 7f6caeffd640 1 --2- 172.21.15.172:0/2944416392 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cb0070dc0 0x7f6cb015b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.194+0000 7f6caeffd640 1 -- 172.21.15.172:0/2944416392 >> v1:172.21.15.172:6789/0 conn(0x7f6cb0154de0 legacy=0x7f6cb015ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.194+0000 7f6caeffd640 1 -- 172.21.15.172:0/2944416392 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb00c90a0 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.194+0000 7f6cae7fc640 1 -- 172.21.15.172:0/2944416392 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063518014 0 0) 0x7f6cb00c9380 con 0x7f6cb0154de0 2024-03-16T19:30:21.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.194+0000 7f6caeffd640 1 --2- 172.21.15.172:0/2944416392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 0x7f6cb0070880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6c980097a0 tx=0x7f6c9802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3e2062608b3453 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.194+0000 7f6cae7fc640 1 -- 172.21.15.172:0/2944416392 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9802e070 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cae7fc640 1 -- 172.21.15.172:0/2944416392 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c980315a0 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cae7fc640 1 -- 172.21.15.172:0/2944416392 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98031930 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/2944416392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 msgr2=0x7f6cb0070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/2944416392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 0x7f6cb0070880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6c980097a0 tx=0x7f6c9802b840 comp rx=0 tx=0).stop 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/2944416392 shutdown_connections 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/2944416392 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cb0070dc0 0x7f6cb015b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/2944416392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 0x7f6cb0070880 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/2944416392 >> 172.21.15.172:0/2944416392 conn(0x7f6cb006e880 msgr2=0x7f6cb006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/2944416392 shutdown_connections 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.195+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/2944416392 wait complete. 2024-03-16T19:30:21.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.196+0000 7f6cb5d5c640 1 Processor -- start 2024-03-16T19:30:21.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.196+0000 7f6cb5d5c640 1 -- start start 2024-03-16T19:30:21.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6cb5d5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cb0070dc0 0x7f6cb015d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6cb5d5c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 0x7f6cb01566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6cb5d5c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb0154de0 0x7f6cb014fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6cb5d5c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb00b0810 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6cb5d5c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb0155830 con 0x7f6cb0154de0 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6cb5d5c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb00af8f0 con 0x7f6cb0070dc0 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6caeffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 0x7f6cb01566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6caeffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 0x7f6cb01566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42388/0 (socket says 172.21.15.172:42388) 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6caeffd640 1 -- 172.21.15.172:0/567494041 learned_addr learned my addr 172.21.15.172:0/567494041 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6caf7fe640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cb0070dc0 0x7f6cb015d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6caffff640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb0154de0 0x7f6cb014fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6caeffd640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cb0070dc0 msgr2=0x7f6cb015d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6caeffd640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cb0070dc0 0x7f6cb015d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.197+0000 7f6caeffd640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb0154de0 msgr2=0x7f6cb014fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.198+0000 7f6caeffd640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb0154de0 0x7f6cb014fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.198+0000 7f6caeffd640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c98009400 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.198+0000 7f6caeffd640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 0x7f6cb01566b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6c980097a0 tx=0x7f6c980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.198+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98004230 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.198+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c98004390 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.198+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb00afb90 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.198+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003850 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.198+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb00b0150 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.199+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6cb00b0950 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.199+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6c980039b0 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.200+0000 7f6ca7fff640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6c9006b3e0 0x7f6c9006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.200+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f6c980abad0 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.200+0000 7f6caf7fe640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6c9006b3e0 0x7f6c9006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.200+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v37) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6cb00b0950 con 0x7f6cb00c9cc0 2024-03-16T19:30:21.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.200+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6cb01f0080 0x7f6cb015b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.200+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb0150670 con 0x7f6cb01f0080 2024-03-16T19:30:21.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.200+0000 7f6caffff640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6cb01f0080 0x7f6cb015b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.200+0000 7f6caf7fe640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6c9006b3e0 0x7f6c9006d8a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00012e0 tx=0x7f6ca0001630 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.200+0000 7f6caffff640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6cb01f0080 0x7f6cb015b0e0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.201+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6cb0150670 con 0x7f6cb01f0080 2024-03-16T19:30:21.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.216+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6cb00b0950 con 0x7f6cb01f0080 2024-03-16T19:30:21.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.217+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6cb00b0950 con 0x7f6cb01f0080 2024-03-16T19:30:21.198 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:21.199 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:21.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.217+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6cb006bf60 0x7f6cb01f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.217+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb00b10d0 con 0x7f6cb006bf60 2024-03-16T19:30:21.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.217+0000 7f6caf7fe640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6cb006bf60 0x7f6cb01f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.218+0000 7f6caf7fe640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6cb006bf60 0x7f6cb01f3b00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.218+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6cb00b10d0 con 0x7f6cb006bf60 2024-03-16T19:30:21.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.233+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6cb006b960 con 0x7f6cb006bf60 2024-03-16T19:30:21.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.235+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6cb006b960 con 0x7f6cb006bf60 2024-03-16T19:30:21.215 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:21.216 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:21.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.235+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6cb0061930 0x7f6cb0061d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.235+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb00cb220 con 0x7f6cb0061930 2024-03-16T19:30:21.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.235+0000 7f6caeffd640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6cb0061930 0x7f6cb0061d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.235+0000 7f6caeffd640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6cb0061930 0x7f6cb0061d10 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.236+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6cb00cb220 con 0x7f6cb0061930 2024-03-16T19:30:21.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.250+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6cb006d5a0 con 0x7f6cb0061930 2024-03-16T19:30:21.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.251+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6cb006d5a0 con 0x7f6cb0061930 2024-03-16T19:30:21.232 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:21.232 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:21.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.251+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6cb015bef0 0x7f6cb015c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.251+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb0062760 con 0x7f6cb015bef0 2024-03-16T19:30:21.233 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.252+0000 7f6caffff640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6cb015bef0 0x7f6cb015c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.233 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.252+0000 7f6caffff640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6cb015bef0 0x7f6cb015c350 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.233 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.252+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6cb0062760 con 0x7f6cb015bef0 2024-03-16T19:30:21.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.266+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6cb015c890 con 0x7f6cb015bef0 2024-03-16T19:30:21.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.268+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6cb015c890 con 0x7f6cb015bef0 2024-03-16T19:30:21.248 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:21.249 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:21.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.268+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6cb002bde0 0x7f6cb002c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.268+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb00b0590 con 0x7f6cb002bde0 2024-03-16T19:30:21.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.268+0000 7f6caf7fe640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6cb002bde0 0x7f6cb002c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.268+0000 7f6caf7fe640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6cb002bde0 0x7f6cb002c240 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.269+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6ca00244f0 con 0x7f6cb002bde0 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.914097710999997, 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:21.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:21.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:21.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:21.256 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.912870288999997, 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:21.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:21.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.912467583000002, 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:21.260 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:21.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.911858832, 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:21.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.911414962999999, 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:21.268 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:21.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:21.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.911272187000002, 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:21.272 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:21.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:21.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.275 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.911257646000003, 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:21.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:21.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:21.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.908379951000001, 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:21.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.283 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:21.284 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:21.284 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:21.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.283+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6cb002c780 con 0x7f6cb002bde0 2024-03-16T19:30:21.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.284+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6cb002c780 con 0x7f6cb002bde0 2024-03-16T19:30:21.285 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:21.286 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:21.287 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:21.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.284+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6cb002ce00 0x7f6cb0206980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.284+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb002d220 con 0x7f6cb002ce00 2024-03-16T19:30:21.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.285+0000 7f6caeffd640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6cb002ce00 0x7f6cb0206980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.285+0000 7f6caeffd640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6cb002ce00 0x7f6cb0206980 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.286+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6cb002d220 con 0x7f6cb002ce00 2024-03-16T19:30:21.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.300+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6cb002d220 con 0x7f6cb002ce00 2024-03-16T19:30:21.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.303+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6cb002d220 con 0x7f6cb002ce00 2024-03-16T19:30:21.287 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:21.288 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:21.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.303+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6cb02073d0 0x7f6cb02097c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.303+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb020c590 con 0x7f6cb02073d0 2024-03-16T19:30:21.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.303+0000 7f6caffff640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6cb02073d0 0x7f6cb02097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.303+0000 7f6caffff640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6cb02073d0 0x7f6cb02097c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.304+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6cb020c590 con 0x7f6cb02073d0 2024-03-16T19:30:21.289 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:21.304+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:21.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.318+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6cb0209d00 con 0x7f6cb02073d0 2024-03-16T19:30:21.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.319+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6cb0209d00 con 0x7f6cb02073d0 2024-03-16T19:30:21.300 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:21.301 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:21.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.320+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6cb020a380 0x7f6cb020c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.320+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb020f640 con 0x7f6cb020a380 2024-03-16T19:30:21.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.320+0000 7f6caf7fe640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6cb020a380 0x7f6cb020c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.320+0000 7f6caf7fe640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6cb020a380 0x7f6cb020c770 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.321+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6cb020f640 con 0x7f6cb020a380 2024-03-16T19:30:21.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.335+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6cb020f610 con 0x7f6cb020a380 2024-03-16T19:30:21.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.336+0000 7f6ca7fff640 1 -- 172.21.15.172:0/567494041 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6cb020f610 con 0x7f6cb020a380 2024-03-16T19:30:21.317 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:21.318 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:21.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.336+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6cb01f0080 msgr2=0x7f6cb015b0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.336+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6cb01f0080 0x7f6cb015b0e0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.336+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6cb006bf60 msgr2=0x7f6cb01f3b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.336+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6cb006bf60 0x7f6cb01f3b00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.336+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6cb0061930 msgr2=0x7f6cb0061d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6cb0061930 0x7f6cb0061d10 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6cb015bef0 msgr2=0x7f6cb015c350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6cb015bef0 0x7f6cb015c350 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6cb002bde0 msgr2=0x7f6cb002c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6cb002bde0 0x7f6cb002c240 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6caffff640 1 -- 172.21.15.172:0/567494041 reap_dead start 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6cb002ce00 msgr2=0x7f6cb0206980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6cb002ce00 0x7f6cb0206980 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6caffff640 1 -- 172.21.15.172:0/567494041 reap_dead start 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6caffff640 1 -- 172.21.15.172:0/567494041 reap_dead start 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6cb02073d0 msgr2=0x7f6cb02097c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6cb02073d0 0x7f6cb02097c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6cb020a380 msgr2=0x7f6cb020c770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6cb020a380 0x7f6cb020c770 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6c9006b3e0 msgr2=0x7f6c9006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6c9006b3e0 0x7f6c9006d8a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00012e0 tx=0x7f6ca0001630 comp rx=0 tx=0).stop 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 msgr2=0x7f6cb01566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 0x7f6cb01566b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6c980097a0 tx=0x7f6c980046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 shutdown_connections 2024-03-16T19:30:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6cb020a380 0x7f6cb020c770 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6cb02073d0 0x7f6cb02097c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00c9cc0 0x7f6cb01566b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 --2- 172.21.15.172:0/567494041 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6c9006b3e0 0x7f6c9006d8a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 >> 172.21.15.172:0/567494041 conn(0x7f6cb006e880 msgr2=0x7f6cb0159970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 shutdown_connections 2024-03-16T19:30:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.337+0000 7f6cb5d5c640 1 -- 172.21.15.172:0/567494041 wait complete. 2024-03-16T19:30:21.385 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:21.404+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:21.399 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:21.399 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:21.409 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:21.493 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:21.512+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:21.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:30:21.561 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:21.561 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:21.570 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:21.611 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:21.630+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:21.617 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.637+0000 7fcbda719640 1 Processor -- start 2024-03-16T19:30:21.617 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.637+0000 7fcbda719640 1 -- start start 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.637+0000 7fcbda719640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcbd406baf0 0x7fcbd406b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.637+0000 7fcbda719640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd41580b0 0x7fcbd4161fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.637+0000 7fcbda719640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd40c1eb0 con 0x7fcbd406aed0 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.637+0000 7fcbda719640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd40c1150 con 0x7fcbd406baf0 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.637+0000 7fcbda719640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd406b7f0 con 0x7fcbd41580b0 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd37fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcbd406baf0 0x7fcbd406b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd37fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcbd406baf0 0x7fcbd406b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42404/0 (socket says 172.21.15.172:42404) 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd3fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcbd406aed0 0x7fcbd41579a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40148/0 (socket says 172.21.15.172:40148) 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd37fe640 1 -- 172.21.15.172:0/3754956614 learned_addr learned my addr 172.21.15.172:0/3754956614 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd8cb6640 1 --2- 172.21.15.172:0/3754956614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd41580b0 0x7fcbd4161fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd37fe640 1 -- 172.21.15.172:0/3754956614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd41580b0 msgr2=0x7fcbd4161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:21.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd37fe640 1 --2- 172.21.15.172:0/3754956614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd41580b0 0x7fcbd4161fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd37fe640 1 -- 172.21.15.172:0/3754956614 >> v1:172.21.15.172:6789/0 conn(0x7fcbd406aed0 legacy=0x7fcbd41579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd37fe640 1 -- 172.21.15.172:0/3754956614 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd40c1350 con 0x7fcbd406baf0 2024-03-16T19:30:21.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.638+0000 7fcbd2ffd640 1 -- 172.21.15.172:0/3754956614 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209329918 0 0) 0x7fcbd40c1eb0 con 0x7fcbd406aed0 2024-03-16T19:30:21.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbd37fe640 1 --2- 172.21.15.172:0/3754956614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd406baf0 0x7fcbd406b2b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800bb10 tx=0x7fcbc802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79536422be5a9fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbd2ffd640 1 -- 172.21.15.172:0/3754956614 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8036440 con 0x7fcbd406baf0 2024-03-16T19:30:21.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbd2ffd640 1 -- 172.21.15.172:0/3754956614 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc80365a0 con 0x7fcbd406baf0 2024-03-16T19:30:21.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbd2ffd640 1 -- 172.21.15.172:0/3754956614 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8036930 con 0x7fcbd406baf0 2024-03-16T19:30:21.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbda719640 1 -- 172.21.15.172:0/3754956614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd406baf0 msgr2=0x7fcbd406b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbda719640 1 --2- 172.21.15.172:0/3754956614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd406baf0 0x7fcbd406b2b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800bb10 tx=0x7fcbc802e150 comp rx=0 tx=0).stop 2024-03-16T19:30:21.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbda719640 1 -- 172.21.15.172:0/3754956614 shutdown_connections 2024-03-16T19:30:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbda719640 1 --2- 172.21.15.172:0/3754956614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd41580b0 0x7fcbd4161fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbda719640 1 --2- 172.21.15.172:0/3754956614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd406baf0 0x7fcbd406b2b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbda719640 1 -- 172.21.15.172:0/3754956614 >> 172.21.15.172:0/3754956614 conn(0x7fcbd40cc030 msgr2=0x7fcbd40cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbda719640 1 -- 172.21.15.172:0/3754956614 shutdown_connections 2024-03-16T19:30:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.639+0000 7fcbda719640 1 -- 172.21.15.172:0/3754956614 wait complete. 2024-03-16T19:30:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.640+0000 7fcbda719640 1 Processor -- start 2024-03-16T19:30:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbda719640 1 -- start start 2024-03-16T19:30:21.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbda719640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbd406aed0 0x7fcbd414c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbda719640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd406baf0 0x7fcbd40b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbda719640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd41580b0 0x7fcbd41453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbda719640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd40cae00 con 0x7fcbd41580b0 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbda719640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd40c8b20 con 0x7fcbd406aed0 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbda719640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd40cb990 con 0x7fcbd406baf0 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbd8cb6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd41580b0 0x7fcbd41453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbd8cb6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd41580b0 0x7fcbd41453e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42406/0 (socket says 172.21.15.172:42406) 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbd8cb6640 1 -- 172.21.15.172:0/177333085 learned_addr learned my addr 172.21.15.172:0/177333085 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbd3fff640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbd406aed0 0x7fcbd414c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.641+0000 7fcbd37fe640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd406baf0 0x7fcbd40b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbd8cb6640 1 -- 172.21.15.172:0/177333085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd406baf0 msgr2=0x7fcbd40b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbd8cb6640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd406baf0 0x7fcbd40b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbd8cb6640 1 -- 172.21.15.172:0/177333085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbd406aed0 msgr2=0x7fcbd414c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbd8cb6640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbd406aed0 0x7fcbd414c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbd8cb6640 1 -- 172.21.15.172:0/177333085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc800b770 con 0x7fcbd41580b0 2024-03-16T19:30:21.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbd8cb6640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd41580b0 0x7fcbd41453e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00ec10 tx=0x7fcbcc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbd0ff9640 1 -- 172.21.15.172:0/177333085 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc00cd20 con 0x7fcbd41580b0 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbd0ff9640 1 -- 172.21.15.172:0/177333085 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc00f040 con 0x7fcbd41580b0 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd40cbc70 con 0x7fcbd41580b0 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbd0ff9640 1 -- 172.21.15.172:0/177333085 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0146c0 con 0x7fcbd41580b0 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.642+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd40cab20 con 0x7fcbd41580b0 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.643+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd4075a70 con 0x7fcbd41580b0 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.644+0000 7fcbd0ff9640 1 -- 172.21.15.172:0/177333085 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fcbcc014a30 con 0x7fcbd41580b0 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.644+0000 7fcbd0ff9640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbc006b3b0 0x7fcbc006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.644+0000 7fcbd0ff9640 1 -- 172.21.15.172:0/177333085 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fcbcc08a730 con 0x7fcbd41580b0 2024-03-16T19:30:21.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.644+0000 7fcbd3fff640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbc006b3b0 0x7fcbc006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:21.627 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.644+0000 7fcbd3fff640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbc006b3b0 0x7fcbc006d870 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4000f60 tx=0x7fcbc4007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:21.628 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.648+0000 7fcbd0ff9640 1 -- 172.21.15.172:0/177333085 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbcc055450 con 0x7fcbd41580b0 2024-03-16T19:30:21.689 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:21.707+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:21.719 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:21.719 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:21.728 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:21.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:21.828+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fcbd4142b70 con 0x7fcbd41580b0 2024-03-16T19:30:21.812 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:21.831+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:21.875 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:21.929 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:21.931 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:21.931 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:21.949+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:22.031 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:22.051+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:22.079 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:22.079 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:22.087 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:22.135 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:22.153+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:22.195 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:22.195 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:22.203 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:22.244 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:22.263+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:22.338 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:22.357+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:22.357 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:22.357 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:22.366 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:22.438 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:22.457+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:22.472 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:22.472 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:22.481 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:22.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.530+0000 7fcbd0ff9640 1 -- 172.21.15.172:0/177333085 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v38) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcbcc05f070 con 0x7fcbd41580b0 2024-03-16T19:30:22.511 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:30:22.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbc006b3b0 msgr2=0x7fcbc006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:22.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbc006b3b0 0x7fcbc006d870 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4000f60 tx=0x7fcbc4007d50 comp rx=0 tx=0).stop 2024-03-16T19:30:22.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd41580b0 msgr2=0x7fcbd41453e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:22.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd41580b0 0x7fcbd41453e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00ec10 tx=0x7fcbcc00c490 comp rx=0 tx=0).stop 2024-03-16T19:30:22.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 shutdown_connections 2024-03-16T19:30:22.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbd41580b0 0x7fcbd41453e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:22.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbc006b3b0 0x7fcbc006d870 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:22.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbd406baf0 0x7fcbd40b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:22.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 --2- 172.21.15.172:0/177333085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbd406aed0 0x7fcbd414c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:22.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.533+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 >> 172.21.15.172:0/177333085 conn(0x7fcbd40cc030 msgr2=0x7fcbd40c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:22.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.534+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 shutdown_connections 2024-03-16T19:30:22.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.534+0000 7fcbda719640 1 -- 172.21.15.172:0/177333085 wait complete. 2024-03-16T19:30:22.565 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:22.584+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:22.632 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:22.632 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:22.639 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:22.658+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:22.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:22.740 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:22.760+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:22.752 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:22.752 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:22.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:22.828 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:30:22.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-16T19:30:22.841 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:22.860+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:22.911 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:22.911 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:22.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:22.942 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:22.961+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:22.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.969+0000 7f7ddb64e640 1 Processor -- start 2024-03-16T19:30:22.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.969+0000 7f7ddb64e640 1 -- start start 2024-03-16T19:30:22.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.970+0000 7f7ddb64e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7dd4154a90 0x7f7dd40c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:22.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.970+0000 7f7ddb64e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd40cc280 0x7f7dd40cae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:22.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.970+0000 7f7ddb64e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd415df50 con 0x7f7dd4154e70 2024-03-16T19:30:22.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.970+0000 7f7ddb64e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd415e1f0 con 0x7f7dd4154a90 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.970+0000 7f7ddb64e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd415e4d0 con 0x7f7dd40cc280 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.970+0000 7f7dd93ea640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7dd4154a90 0x7f7dd40c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.970+0000 7f7dd8be9640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7dd4154e70 0x7f7dd406b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40160/0 (socket says 172.21.15.172:40160) 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.970+0000 7f7dd8be9640 1 -- 172.21.15.172:0/2403883199 learned_addr learned my addr 172.21.15.172:0/2403883199 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.970+0000 7f7dd9beb640 1 --2- 172.21.15.172:0/2403883199 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd40cc280 0x7f7dd40cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.971+0000 7f7dd3fff640 1 -- 172.21.15.172:0/2403883199 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907469857 0 0) 0x7f7dd415df50 con 0x7f7dd4154e70 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.971+0000 7f7dd3fff640 1 -- 172.21.15.172:0/2403883199 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dbc003520 con 0x7f7dd4154e70 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.971+0000 7f7dd93ea640 1 -- 172.21.15.172:0/2403883199 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd40cc280 msgr2=0x7f7dd40cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.971+0000 7f7dd93ea640 1 --2- 172.21.15.172:0/2403883199 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd40cc280 0x7f7dd40cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.971+0000 7f7dd93ea640 1 -- 172.21.15.172:0/2403883199 >> v1:172.21.15.172:6789/0 conn(0x7f7dd4154e70 legacy=0x7f7dd406b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.971+0000 7f7dd93ea640 1 -- 172.21.15.172:0/2403883199 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd415e7b0 con 0x7f7dd4154a90 2024-03-16T19:30:22.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.972+0000 7f7dd93ea640 1 --2- 172.21.15.172:0/2403883199 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154a90 0x7f7dd40c50b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0097a0 tx=0x7f7dcc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793a935684559f19 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:22.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.972+0000 7f7dd3fff640 1 -- 172.21.15.172:0/2403883199 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc030610 con 0x7f7dd4154a90 2024-03-16T19:30:22.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.972+0000 7f7dd3fff640 1 -- 172.21.15.172:0/2403883199 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc030770 con 0x7f7dd4154a90 2024-03-16T19:30:22.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.972+0000 7f7dd3fff640 1 -- 172.21.15.172:0/2403883199 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc030b00 con 0x7f7dd4154a90 2024-03-16T19:30:22.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.973+0000 7f7ddb64e640 1 -- 172.21.15.172:0/2403883199 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154a90 msgr2=0x7f7dd40c50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:22.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.973+0000 7f7ddb64e640 1 --2- 172.21.15.172:0/2403883199 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154a90 0x7f7dd40c50b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0097a0 tx=0x7f7dcc02b960 comp rx=0 tx=0).stop 2024-03-16T19:30:22.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.973+0000 7f7ddb64e640 1 -- 172.21.15.172:0/2403883199 shutdown_connections 2024-03-16T19:30:22.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.973+0000 7f7ddb64e640 1 --2- 172.21.15.172:0/2403883199 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd40cc280 0x7f7dd40cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:22.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.973+0000 7f7ddb64e640 1 --2- 172.21.15.172:0/2403883199 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154a90 0x7f7dd40c50b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:22.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.973+0000 7f7ddb64e640 1 -- 172.21.15.172:0/2403883199 >> 172.21.15.172:0/2403883199 conn(0x7f7dd4067250 msgr2=0x7f7dd40d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:22.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.973+0000 7f7ddb64e640 1 -- 172.21.15.172:0/2403883199 shutdown_connections 2024-03-16T19:30:22.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.973+0000 7f7ddb64e640 1 -- 172.21.15.172:0/2403883199 wait complete. 2024-03-16T19:30:22.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.974+0000 7f7ddb64e640 1 Processor -- start 2024-03-16T19:30:22.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.974+0000 7f7ddb64e640 1 -- start start 2024-03-16T19:30:22.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7ddb64e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dd40cc280 0x7f7dd4153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:22.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7ddb64e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd4154a90 0x7f7dd40b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:22.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7ddb64e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154e70 0x7f7dd4146c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:22.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7ddb64e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4150de0 con 0x7f7dd4154e70 2024-03-16T19:30:22.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7ddb64e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4156120 con 0x7f7dd40cc280 2024-03-16T19:30:22.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7ddb64e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4150540 con 0x7f7dd4154a90 2024-03-16T19:30:22.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd9beb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154e70 0x7f7dd4146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:22.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd9beb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154e70 0x7f7dd4146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42424/0 (socket says 172.21.15.172:42424) 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd9beb640 1 -- 172.21.15.172:0/3314789098 learned_addr learned my addr 172.21.15.172:0/3314789098 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd8be9640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd4154a90 0x7f7dd40b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd93ea640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dd40cc280 0x7f7dd4153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd9beb640 1 -- 172.21.15.172:0/3314789098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd4154a90 msgr2=0x7f7dd40b55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd9beb640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd4154a90 0x7f7dd40b55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd9beb640 1 -- 172.21.15.172:0/3314789098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dd40cc280 msgr2=0x7f7dd4153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd9beb640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dd40cc280 0x7f7dd4153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd9beb640 1 -- 172.21.15.172:0/3314789098 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc009400 con 0x7f7dd4154e70 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd8be9640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd4154a90 0x7f7dd40b55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd9beb640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154e70 0x7f7dd4146c40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400efc0 tx=0x7f7dc400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd1ffb640 1 -- 172.21.15.172:0/3314789098 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4009ca0 con 0x7f7dd4154e70 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd1ffb640 1 -- 172.21.15.172:0/3314789098 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc400f040 con 0x7f7dd4154e70 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7dd1ffb640 1 -- 172.21.15.172:0/3314789098 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40146c0 con 0x7f7dd4154e70 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.975+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd4150820 con 0x7f7dd4154e70 2024-03-16T19:30:22.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.976+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd4153230 con 0x7f7dd4154e70 2024-03-16T19:30:22.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.977+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd4153090 con 0x7f7dd4154e70 2024-03-16T19:30:22.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.977+0000 7f7dd1ffb640 1 -- 172.21.15.172:0/3314789098 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7dc4014820 con 0x7f7dd4154e70 2024-03-16T19:30:22.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.977+0000 7f7dd1ffb640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7dc006b3b0 0x7f7dc006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:22.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.977+0000 7f7dd1ffb640 1 -- 172.21.15.172:0/3314789098 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f7dc408ab30 con 0x7f7dd4154e70 2024-03-16T19:30:22.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.978+0000 7f7dd93ea640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7dc006b3b0 0x7f7dc006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:22.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.978+0000 7f7dd93ea640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7dc006b3b0 0x7f7dc006d870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc036000 tx=0x7f7dcc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:22.982+0000 7f7dd1ffb640 1 -- 172.21.15.172:0/3314789098 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dd4153090 con 0x7f7dd4154e70 2024-03-16T19:30:23.030 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:23.030 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:23.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:23.043 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:23.061+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:23.143 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:23.163+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:23.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.173+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f7dd4153090 con 0x7f7dd4154e70 2024-03-16T19:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:23.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:23.243 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:23.262+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:23.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:23.342 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:23.342 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:23.344 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:23.363+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:23.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:23.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.449+0000 7f889bfd7640 1 Processor -- start 2024-03-16T19:30:23.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.449+0000 7f889bfd7640 1 -- start start 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889bfd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 0x7f88940cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889bfd7640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f88940cb5e0 0x7f88940b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889bfd7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88940b4270 con 0x7f8894154ed0 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889bfd7640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889415b250 con 0x7f88940cb5e0 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889bfd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88941552b0 con 0x7f88940b3630 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889a574640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f88940cb5e0 0x7f88940b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f8899572640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 0x7f88940cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889a574640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f88940cb5e0 0x7f88940b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42432/0 (socket says 172.21.15.172:42432) 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889a574640 1 -- 172.21.15.172:0/2235006433 learned_addr learned my addr 172.21.15.172:0/2235006433 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:23.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f8899d73640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f8894154ed0 0x7f889415ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:58888/0 (socket says 172.21.15.172:58888) 2024-03-16T19:30:23.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889a574640 1 -- 172.21.15.172:0/2235006433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 msgr2=0x7f88940cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:23.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889a574640 1 --2- 172.21.15.172:0/2235006433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 0x7f88940cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.450+0000 7f889a574640 1 -- 172.21.15.172:0/2235006433 >> v1:172.21.15.174:6789/0 conn(0x7f8894154ed0 legacy=0x7f889415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.451+0000 7f889a574640 1 -- 172.21.15.172:0/2235006433 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889415b4b0 con 0x7f88940cb5e0 2024-03-16T19:30:23.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.451+0000 7f8898d71640 1 -- 172.21.15.172:0/2235006433 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25736301 0 0) 0x7f88940b4270 con 0x7f8894154ed0 2024-03-16T19:30:23.432 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.451+0000 7f8899572640 1 --2- 172.21.15.172:0/2235006433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 0x7f88940cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.451+0000 7f889a574640 1 --2- 172.21.15.172:0/2235006433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88940cb5e0 0x7f88940b4700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8884009870 tx=0x7f888402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e667dad13f01f1e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.451+0000 7f8898d71640 1 -- 172.21.15.172:0/2235006433 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884030840 con 0x7f88940cb5e0 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.451+0000 7f8898d71640 1 -- 172.21.15.172:0/2235006433 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88840309a0 con 0x7f88940cb5e0 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.451+0000 7f8898d71640 1 -- 172.21.15.172:0/2235006433 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884030d30 con 0x7f88940cb5e0 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.452+0000 7f889bfd7640 1 -- 172.21.15.172:0/2235006433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88940cb5e0 msgr2=0x7f88940b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.452+0000 7f889bfd7640 1 --2- 172.21.15.172:0/2235006433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88940cb5e0 0x7f88940b4700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8884009870 tx=0x7f888402b980 comp rx=0 tx=0).stop 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.452+0000 7f889bfd7640 1 -- 172.21.15.172:0/2235006433 shutdown_connections 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.452+0000 7f889bfd7640 1 --2- 172.21.15.172:0/2235006433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88940cb5e0 0x7f88940b4700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.452+0000 7f889bfd7640 1 --2- 172.21.15.172:0/2235006433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 0x7f88940cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.452+0000 7f889bfd7640 1 -- 172.21.15.172:0/2235006433 >> 172.21.15.172:0/2235006433 conn(0x7f889406bf60 msgr2=0x7f889406d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:23.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.452+0000 7f889bfd7640 1 -- 172.21.15.172:0/2235006433 shutdown_connections 2024-03-16T19:30:23.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.452+0000 7f889bfd7640 1 -- 172.21.15.172:0/2235006433 wait complete. 2024-03-16T19:30:23.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.453+0000 7f889bfd7640 1 Processor -- start 2024-03-16T19:30:23.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.453+0000 7f889bfd7640 1 -- start start 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889bfd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 0x7f88941f1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889bfd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940cb5e0 0x7f88940c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889bfd7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8894154ed0 0x7f88940c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889bfd7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88940c88c0 con 0x7f8894154ed0 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889bfd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88940c6ce0 con 0x7f88940cb5e0 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889bfd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88941f0b80 con 0x7f88940b3630 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889a574640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8894154ed0 0x7f88940c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889a574640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8894154ed0 0x7f88940c5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42442/0 (socket says 172.21.15.172:42442) 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889a574640 1 -- 172.21.15.172:0/3394864804 learned_addr learned my addr 172.21.15.172:0/3394864804 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f8899572640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940cb5e0 0x7f88940c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f8899d73640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 0x7f88941f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889a574640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 msgr2=0x7f88941f1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889a574640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940b3630 0x7f88941f1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889a574640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940cb5e0 msgr2=0x7f88940c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889a574640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940cb5e0 0x7f88940c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.454+0000 7f889a574640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8888009400 con 0x7f8894154ed0 2024-03-16T19:30:23.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.455+0000 7f889a574640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8894154ed0 0x7f88940c5070 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f888402b9c0 tx=0x7f8884004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.455+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884039c90 con 0x7f8894154ed0 2024-03-16T19:30:23.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.455+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8884039e10 con 0x7f8894154ed0 2024-03-16T19:30:23.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.455+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88840094d0 con 0x7f8894154ed0 2024-03-16T19:30:23.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.455+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88840446c0 con 0x7f8894154ed0 2024-03-16T19:30:23.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.455+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88940ca800 con 0x7f8894154ed0 2024-03-16T19:30:23.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.456+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8894068d20 con 0x7f8894154ed0 2024-03-16T19:30:23.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.456+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f888404d3b0 con 0x7f8894154ed0 2024-03-16T19:30:23.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.457+0000 7f88927fc640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f888006b3e0 0x7f888006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.457+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f888403e070 con 0x7f8894154ed0 2024-03-16T19:30:23.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.457+0000 7f8899d73640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f888006b3e0 0x7f888006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.457+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v38) v1 ==== 54+0+16 (secure 0 0 0) 0x7f88840b3070 con 0x7f8894154ed0 2024-03-16T19:30:23.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.457+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f88941f0730 0x7f889414e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.457+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f889415ba70 con 0x7f88941f0730 2024-03-16T19:30:23.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.457+0000 7f8899572640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f88941f0730 0x7f889414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.457+0000 7f8899d73640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f888006b3e0 0x7f888006d8a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f888c0012e0 tx=0x7f888c001630 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.458+0000 7f8899572640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f88941f0730 0x7f889414e900 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.458+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8888009e00 con 0x7f88941f0730 2024-03-16T19:30:23.445 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:23.465+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:23.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.473+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8894153b40 con 0x7f88941f0730 2024-03-16T19:30:23.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.475+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8894153b40 con 0x7f88941f0730 2024-03-16T19:30:23.455 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:23.456 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:23.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.475+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f88940c9430 0x7f88941f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.475+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f889414f350 con 0x7f88940c9430 2024-03-16T19:30:23.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.475+0000 7f8899d73640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f88940c9430 0x7f88941f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.475+0000 7f8899d73640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f88940c9430 0x7f88941f3be0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.476+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f889414f350 con 0x7f88940c9430 2024-03-16T19:30:23.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.491+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88940681c0 con 0x7f88940c9430 2024-03-16T19:30:23.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.493+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f88940681c0 con 0x7f88940c9430 2024-03-16T19:30:23.473 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:23.474 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:23.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.493+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f88940cc0f0 0x7f88941f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.493+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88940c6440 con 0x7f88940cc0f0 2024-03-16T19:30:23.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.493+0000 7f8899572640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f88940cc0f0 0x7f88941f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.494+0000 7f8899572640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f88940cc0f0 0x7f88941f4630 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.494+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f88940c6440 con 0x7f88940cc0f0 2024-03-16T19:30:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.508+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88940c6720 con 0x7f88940cc0f0 2024-03-16T19:30:23.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.510+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f88940c6720 con 0x7f88940cc0f0 2024-03-16T19:30:23.490 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:23.491 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:23.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.510+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8894062200 0x7f889414f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.510+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88940c6280 con 0x7f8894062200 2024-03-16T19:30:23.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.510+0000 7f889a574640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8894062200 0x7f889414f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.511+0000 7f889a574640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8894062200 0x7f889414f790 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.511+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f88940c6280 con 0x7f8894062200 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.166550342000001, 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.505 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:30:23.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.166533227999999, 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:23.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.166525948, 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:30:23.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:23.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.166512081, 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:30:23.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:30:23.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.165839245999997, 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:23.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:30:23.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:30:23.528 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.163540996000002, 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.163495894999997, 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:23.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:30:23.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:30:23.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.163467840000003, 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.163446426, 2024-03-16T19:30:23.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:23.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:30:23.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.163424075000002, 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:30:23.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:30:23.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.163403676000001, 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:30:23.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.551 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.163297684, 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:30:23.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:23.555 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.161505972999997, 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:30:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.161458472, 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:30:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:30:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.161219221000003, 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:30 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.161063642000002, 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126666, 2024-03-16T19:30:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362823+0000", 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362824+0000", 2024-03-16T19:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7999999999999995e-07 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362828+0000", 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0879999999999997e-06 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362837+0000", 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.704e-06 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362843+0000", 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9390000000000002e-06 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362877+0000", 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4779e-05 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362883+0000", 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9009999999999999e-06 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362950+0000", 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6674999999999994e-05 2024-03-16T19:30:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.152721075999999, 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.151381200000003, 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.1483902180000003, 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:30:23.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:30:23.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:46 2.6 e36/18 ondisk, result = 0)", 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.376767+0000", 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.1471201410000003, 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013778799999999999, 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376767+0000", 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376767+0000", 2024-03-16T19:30:23.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376768+0000", 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376771+0000", 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2509999999999999e-06 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376781+0000", 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349999999999999e-05 2024-03-16T19:30:23.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376788+0000", 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7900000000000002e-06 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376826+0000", 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7886000000000003e-05 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376832+0000", 2024-03-16T19:30:23.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8340000000000004e-06 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376905+0000", 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2966999999999994e-05 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:23.586 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:23.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.526+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8894062870 con 0x7f8894062200 2024-03-16T19:30:23.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.527+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8894062870 con 0x7f8894062200 2024-03-16T19:30:23.587 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:23.588 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:23.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.528+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f88941501e0 0x7f889402bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.528+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88941535c0 con 0x7f88941501e0 2024-03-16T19:30:23.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.528+0000 7f8899d73640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f88941501e0 0x7f889402bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.528+0000 7f8899d73640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f88941501e0 0x7f889402bde0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.529+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f88941535c0 con 0x7f88941501e0 2024-03-16T19:30:23.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.531+0000 7f7dd1ffb640 1 -- 172.21.15.172:0/3314789098 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v39) v1 ==== 113+0+0 (secure 0 0 0) 0x7f7dd4153090 con 0x7f7dd4154e70 2024-03-16T19:30:23.589 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.0 to 1 (10000) 2024-03-16T19:30:23.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7dc006b3b0 msgr2=0x7f7dc006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7dc006b3b0 0x7f7dc006d870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc036000 tx=0x7f7dcc008030 comp rx=0 tx=0).stop 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154e70 msgr2=0x7f7dd4146c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154e70 0x7f7dd4146c40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400efc0 tx=0x7f7dc400c490 comp rx=0 tx=0).stop 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 shutdown_connections 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7dd4154e70 0x7f7dd4146c40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7dc006b3b0 0x7f7dc006d870 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dd4154a90 0x7f7dd40b55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 --2- 172.21.15.172:0/3314789098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dd40cc280 0x7f7dd4153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 >> 172.21.15.172:0/3314789098 conn(0x7f7dd4067250 msgr2=0x7f7dd40c50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 shutdown_connections 2024-03-16T19:30:23.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.535+0000 7f7ddb64e640 1 -- 172.21.15.172:0/3314789098 wait complete. 2024-03-16T19:30:23.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-16T19:30:23.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.553+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88941535c0 con 0x7f88941501e0 2024-03-16T19:30:23.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.555+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f88941535c0 con 0x7f88941501e0 2024-03-16T19:30:23.592 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:23.593 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:23.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.555+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f889402c830 0x7f889402cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.555+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8894153360 con 0x7f889402c830 2024-03-16T19:30:23.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.555+0000 7f8899572640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f889402c830 0x7f889402cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.555+0000 7f8899572640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f889402c830 0x7f889402cc90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.556+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8894153360 con 0x7f889402c830 2024-03-16T19:30:23.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.570+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f889402d200 con 0x7f889402c830 2024-03-16T19:30:23.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.572+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f889402d200 con 0x7f889402c830 2024-03-16T19:30:23.594 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:23.595 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:23.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.572+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8894207480 0x7f8894209870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.572+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88941505e0 con 0x7f8894207480 2024-03-16T19:30:23.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.572+0000 7f889a574640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8894207480 0x7f8894209870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.573+0000 7f889a574640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8894207480 0x7f8894209870 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.573+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f88941505e0 con 0x7f8894207480 2024-03-16T19:30:23.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.587+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88941505e0 con 0x7f8894207480 2024-03-16T19:30:23.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.589+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f88941505e0 con 0x7f8894207480 2024-03-16T19:30:23.596 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:23.596 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:23.606+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:23.596 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:23.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.589+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f889420a2c0 0x7f889420c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.589+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f889420f5f0 con 0x7f889420a2c0 2024-03-16T19:30:23.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.590+0000 7f8899d73640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f889420a2c0 0x7f889420c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.590+0000 7f8899d73640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f889420a2c0 0x7f889420c6b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.591+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f889420f5f0 con 0x7f889420a2c0 2024-03-16T19:30:23.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.605+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8894062700 con 0x7f889420a2c0 2024-03-16T19:30:23.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.606+0000 7f88927fc640 1 -- 172.21.15.172:0/3394864804 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8894062700 con 0x7f889420a2c0 2024-03-16T19:30:23.597 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:23.599 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:23.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f88941f0730 msgr2=0x7f889414e900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f88941f0730 0x7f889414e900 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f88940c9430 msgr2=0x7f88941f3be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f88940c9430 0x7f88941f3be0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f88940cc0f0 msgr2=0x7f88941f4630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f88940cc0f0 0x7f88941f4630 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8894062200 msgr2=0x7f889414f790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8894062200 0x7f889414f790 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f88941501e0 msgr2=0x7f889402bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889a574640 1 -- 172.21.15.172:0/3394864804 reap_dead start 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f88941501e0 0x7f889402bde0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889a574640 1 -- 172.21.15.172:0/3394864804 reap_dead start 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f889402c830 msgr2=0x7f889402cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f889402c830 0x7f889402cc90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8894207480 msgr2=0x7f8894209870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8894207480 0x7f8894209870 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f889420a2c0 msgr2=0x7f889420c6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f889420a2c0 0x7f889420c6b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f888006b3e0 msgr2=0x7f888006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f888006b3e0 0x7f888006d8a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f888c0012e0 tx=0x7f888c001630 comp rx=0 tx=0).stop 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8894154ed0 msgr2=0x7f88940c5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 --2- 172.21.15.172:0/3394864804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8894154ed0 0x7f88940c5070 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f888402b9c0 tx=0x7f8884004730 comp rx=0 tx=0).stop 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889a574640 1 -- 172.21.15.172:0/3394864804 reap_dead start 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 shutdown_connections 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 >> 172.21.15.172:0/3394864804 conn(0x7f889406bf60 msgr2=0x7f88941540b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 shutdown_connections 2024-03-16T19:30:23.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.607+0000 7f889bfd7640 1 -- 172.21.15.172:0/3394864804 wait complete. 2024-03-16T19:30:23.688 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:23.707+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:23.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.763+0000 7f8a34179640 1 Processor -- start 2024-03-16T19:30:23.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.763+0000 7f8a34179640 1 -- start start 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a34179640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c06dc00 0x7f8a2c166220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a34179640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c1583a0 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a34179640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c06c620 con 0x7f8a2c157fc0 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a34179640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c15ed70 con 0x7f8a2c06dc00 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a31714640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8a2c06f3d0 0x7f8a2c06e690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40170/0 (socket says 172.21.15.172:40170) 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a31714640 1 -- 172.21.15.172:0/3277974286 learned_addr learned my addr 172.21.15.172:0/3277974286 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a32716640 1 --2- 172.21.15.172:0/3277974286 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c06dc00 0x7f8a2c166220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a30f13640 1 -- 172.21.15.172:0/3277974286 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287694275 0 0) 0x7f8a2c1583a0 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a30f13640 1 -- 172.21.15.172:0/3277974286 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a18003520 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a30f13640 1 -- 172.21.15.172:0/3277974286 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950298518 0 0) 0x7f8a2c06c620 con 0x7f8a2c157fc0 2024-03-16T19:30:23.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a30f13640 1 -- 172.21.15.172:0/3277974286 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a2c1583a0 con 0x7f8a2c157fc0 2024-03-16T19:30:23.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a32716640 1 -- 172.21.15.172:0/3277974286 >> v1:172.21.15.174:6789/0 conn(0x7f8a2c157fc0 legacy=0x7f8a2c0c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a32716640 1 -- 172.21.15.172:0/3277974286 >> v1:172.21.15.172:6789/0 conn(0x7f8a2c06f3d0 legacy=0x7f8a2c06e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.764+0000 7f8a32716640 1 -- 172.21.15.172:0/3277974286 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c06c340 con 0x7f8a2c06dc00 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.765+0000 7f8a32716640 1 --2- 172.21.15.172:0/3277974286 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c06dc00 0x7f8a2c166220 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009dc0 tx=0x7f8a2402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=819fe0adc8320b28 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.765+0000 7f8a30f13640 1 -- 172.21.15.172:0/3277974286 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2402cc40 con 0x7f8a2c06dc00 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.765+0000 7f8a30f13640 1 -- 172.21.15.172:0/3277974286 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2402cda0 con 0x7f8a2c06dc00 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.765+0000 7f8a30f13640 1 -- 172.21.15.172:0/3277974286 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a24034660 con 0x7f8a2c06dc00 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.766+0000 7f8a34179640 1 -- 172.21.15.172:0/3277974286 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c06dc00 msgr2=0x7f8a2c166220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.766+0000 7f8a34179640 1 --2- 172.21.15.172:0/3277974286 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c06dc00 0x7f8a2c166220 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009dc0 tx=0x7f8a2402be90 comp rx=0 tx=0).stop 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.766+0000 7f8a34179640 1 -- 172.21.15.172:0/3277974286 shutdown_connections 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.766+0000 7f8a34179640 1 --2- 172.21.15.172:0/3277974286 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c06dc00 0x7f8a2c166220 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.766+0000 7f8a34179640 1 -- 172.21.15.172:0/3277974286 >> 172.21.15.172:0/3277974286 conn(0x7f8a2c066c20 msgr2=0x7f8a2c067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.766+0000 7f8a34179640 1 -- 172.21.15.172:0/3277974286 shutdown_connections 2024-03-16T19:30:23.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.766+0000 7f8a34179640 1 -- 172.21.15.172:0/3277974286 wait complete. 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.767+0000 7f8a34179640 1 Processor -- start 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.767+0000 7f8a34179640 1 -- start start 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a34179640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c06dc00 0x7f8a2c1f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a34179640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a2c06f3d0 0x7f8a2c1e3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a34179640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c157fc0 0x7f8a2c1e4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a34179640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0c5e80 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a34179640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0c6160 con 0x7f8a2c06dc00 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a34179640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0c6440 con 0x7f8a2c157fc0 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31714640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a2c06f3d0 0x7f8a2c1e3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31714640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a2c06f3d0 0x7f8a2c1e3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42450/0 (socket says 172.21.15.172:42450) 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31714640 1 -- 172.21.15.172:0/1999258373 learned_addr learned my addr 172.21.15.172:0/1999258373 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a32716640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c157fc0 0x7f8a2c1e4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31f15640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c06dc00 0x7f8a2c1f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31714640 1 -- 172.21.15.172:0/1999258373 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c157fc0 msgr2=0x7f8a2c1e4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31714640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c157fc0 0x7f8a2c1e4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31714640 1 -- 172.21.15.172:0/1999258373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c06dc00 msgr2=0x7f8a2c1f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31714640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c06dc00 0x7f8a2c1f34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31714640 1 -- 172.21.15.172:0/1999258373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24009a70 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:23.750 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:23.751 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:23.751 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a31714640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a2c06f3d0 0x7f8a2c1e3e10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800c500 tx=0x7f8a2800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.768+0000 7f8a227fc640 1 -- 172.21.15.172:0/1999258373 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28005ce0 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.769+0000 7f8a227fc640 1 -- 172.21.15.172:0/1999258373 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a28005e60 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.769+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2c1f3f70 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.769+0000 7f8a227fc640 1 -- 172.21.15.172:0/1999258373 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28007330 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.769+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2c15bc30 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.770+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2c0681c0 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.770+0000 7f8a227fc640 1 -- 172.21.15.172:0/1999258373 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8a28007490 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.770+0000 7f8a227fc640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8a1006b3e0 0x7f8a1006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.771+0000 7f8a227fc640 1 -- 172.21.15.172:0/1999258373 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f8a280891f0 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.771+0000 7f8a31f15640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8a1006b3e0 0x7f8a1006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:23.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.771+0000 7f8a31f15640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8a1006b3e0 0x7f8a1006d8a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c007fd0 tx=0x7f8a1c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:23.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.775+0000 7f8a227fc640 1 -- 172.21.15.172:0/1999258373 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a28053fd0 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:23.802 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:23.822+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:23.865 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:23.865 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:23.865 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:23.865 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:23.866 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:23.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:23.915 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:23.935+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:23.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:23.964+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f8a2c06f1b0 con 0x7f8a2c06f3d0 2024-03-16T19:30:23.990 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:24.009+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:24.022 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:24.022 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:24.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:24.022 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:24.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.022 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.022 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683977044999999, 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053568110000000004, 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357146+0000", 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5860000000000001e-06 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357154+0000", 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5439999999999999e-06 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357217+0000", 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2986999999999999e-05 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357229+0000", 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1853e-05 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357343+0000", 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113663 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357504+0000", 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016176 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013908599999999999 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360705+0000", 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030619419999999998 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362409+0000", 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001704021 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362456+0000", 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6140000000000002e-05 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362496+0000", 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0228999999999997e-05 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:24.027 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683957018000001, 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078702890000000008, 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4299999999999998e-07 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357163+0000", 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2559999999999998e-06 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357254+0000", 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1701999999999997e-05 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357258+0000", 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5719999999999999e-06 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357931+0000", 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067308000000000005 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357999+0000", 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8323000000000003e-05 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:24.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358113+0000", 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011366699999999999 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362505+0000", 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004391509 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024372970000000002 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364992+0000", 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9741999999999999e-05 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365029+0000", 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7697999999999997e-05 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683950216, 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079944349999999994, 2024-03-16T19:30:24.032 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6e-07 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357172+0000", 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7960000000000001e-06 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357285+0000", 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113183 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357290+0000", 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6190000000000002e-06 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358306+0000", 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010160670000000001 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358400+0000", 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3854e-05 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358492+0000", 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2954000000000005e-05 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362510+0000", 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040175469999999998 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025680310000000001 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365123+0000", 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4515000000000003e-05 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365160+0000", 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7608999999999999e-05 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683943319999997, 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080787759999999993, 2024-03-16T19:30:24.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.36e-07 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1270000000000001e-06 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357297+0000", 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121885 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357300+0000", 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049999999999999e-06 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358709+0000", 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001409083 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358776+0000", 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6489999999999995e-05 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358885+0000", 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109271 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362514+0000", 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036292199999999998 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365197+0000", 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026832100000000001 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365226+0000", 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9136999999999999e-05 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365251+0000", 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5012e-05 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683940335999999, 2024-03-16T19:30:24.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081787609999999997, 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357177+0000", 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5090000000000001e-06 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357306+0000", 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012856900000000001 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357308+0000", 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3700000000000002e-06 2024-03-16T19:30:24.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359071+0000", 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017631210000000001 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359182+0000", 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110265 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359280+0000", 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8595e-05 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362520+0000", 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032394120000000001 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365297+0000", 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002777542 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365328+0000", 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0899999999999999e-05 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365354+0000", 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6276000000000001e-05 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683937647999997, 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083197830000000007, 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357179+0000", 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357187+0000", 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357314+0000", 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126978 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357316+0000", 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3800000000000001e-06 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359512+0000", 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021950799999999999 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359585+0000", 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3168000000000001e-05 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359702+0000", 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011741 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362525+0000", 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028225419999999999 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365408+0000", 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028829010000000002 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365460+0000", 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839e-05 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365498+0000", 2024-03-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7673999999999999e-05 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683926522999997, 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084493169999999996, 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:24.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357190+0000", 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-07 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357191+0000", 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7069999999999999e-06 2024-03-16T19:30:24.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357325+0000", 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133585 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357330+0000", 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0799999999999996e-06 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359915+0000", 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025849290000000001 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9741999999999995e-05 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360121+0000", 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362826+0000", 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027046819999999999 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365564+0000", 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027387179999999998 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365606+0000", 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1715999999999998e-05 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365639+0000", 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660999999999998e-05 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683923854, 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085533069999999996, 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0699999999999999e-07 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3030000000000001e-06 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357339+0000", 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014513199999999999 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000003e-06 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030087619999999999 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360414+0000", 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1297999999999995e-05 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360498+0000", 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4538999999999998e-05 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362832+0000", 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023338410000000001 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365670+0000", 2024-03-16T19:30:24.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028379550000000001 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365724+0000", 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3533999999999999e-05 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365745+0000", 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1725999999999998e-05 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.061 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683920630000003, 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052850789999999998, 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:24.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357196+0000", 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.173e-06 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357359+0000", 2024-03-16T19:30:24.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016222400000000001 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357365+0000", 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7320000000000003e-06 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357451+0000", 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6494000000000002e-05 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357555+0000", 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010401200000000001 2024-03-16T19:30:24.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8224000000000001e-05 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362294+0000", 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046502399999999999 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362388+0000", 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4448000000000001e-05 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362433+0000", 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4888000000000001e-05 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362480+0000", 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7441000000000002e-05 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:24.066 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.683918593000001, 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053762239999999998, 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357198+0000", 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357199+0000", 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357415+0000", 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021638699999999999 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0069e-05 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358132+0000", 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070650599999999997 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358207+0000", 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5197999999999994e-05 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358342+0000", 2024-03-16T19:30:24.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013503199999999999 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362377+0000", 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040354600000000003 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362493+0000", 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115951 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362536+0000", 2024-03-16T19:30:24.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3189999999999998e-05 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362574+0000", 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7113999999999997e-05 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:6 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.680859691999999, 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045449999999999999, 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.071 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360257+0000", 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7400000000000002e-07 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360260+0000", 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3550000000000001e-06 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360270+0000", 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360277+0000", 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5760000000000002e-06 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360697+0000", 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042039200000000001 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360702+0000", 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7140000000000001e-06 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360711+0000", 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1209999999999996e-06 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:22 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.678915517, 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100673, 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362201+0000", 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5799999999999998e-07 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362206+0000", 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8579999999999999e-06 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362219+0000", 2024-03-16T19:30:24.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.239e-05 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362226+0000", 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2710000000000003e-06 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362281+0000", 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4756000000000002e-05 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362288+0000", 2024-03-16T19:30:24.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.075e-06 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362301+0000", 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3365e-05 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:23 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.678804247999999, 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017205200000000001, 2024-03-16T19:30:24.078 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4499999999999997e-07 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362315+0000", 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1350000000000001e-06 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362322+0000", 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9469999999999997e-06 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362328+0000", 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362366+0000", 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8160000000000001e-05 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362373+0000", 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5799999999999997e-06 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362484+0000", 2024-03-16T19:30:24.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111173 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:7 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.676260376000002, 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018177499999999999, 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364861+0000", 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3699999999999997e-06 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364875+0000", 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4059e-05 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364881+0000", 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5270000000000004e-06 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364929+0000", 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000002e-05 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3699999999999997e-06 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365037+0000", 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100819 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:8 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.676203319000003, 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025451199999999998, 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364916+0000", 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8109999999999999e-06 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364924+0000", 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1419999999999993e-06 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364930+0000", 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2260000000000002e-06 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365071+0000", 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014030100000000001 2024-03-16T19:30:24.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365075+0000", 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365167+0000", 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2196999999999998e-05 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.089 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:9 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.676177078999999, 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032082900000000001, 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.22e-07 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3980000000000002e-06 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364947+0000", 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3090000000000003e-06 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364951+0000", 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1309999999999999e-06 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365191+0000", 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023992600000000001 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365195+0000", 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.704e-06 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365260+0000", 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5139000000000001e-05 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:10 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.093 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.676148693000002, 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039415499999999998, 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:24.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364968+0000", 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364970+0000", 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4650000000000001e-06 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364977+0000", 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.443e-06 2024-03-16T19:30:24.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364981+0000", 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2320000000000003e-06 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365289+0000", 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030856900000000002 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365294+0000", 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6920000000000001e-06 2024-03-16T19:30:24.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365361+0000", 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7358000000000001e-05 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:11 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.676128192999997, 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051803199999999998, 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.097 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9299999999999999e-07 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364993+0000", 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9080000000000003e-06 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4240000000000001e-06 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365002+0000", 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7570000000000002e-06 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365395+0000", 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039237199999999999 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365401+0000", 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9380000000000003e-06 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365506+0000", 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010433999999999999 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:12 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.676106189000002, 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063430500000000005, 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9500000000000001e-07 2024-03-16T19:30:24.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365012+0000", 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7129999999999999e-06 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365015+0000", 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2629999999999999e-06 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365017+0000", 2024-03-16T19:30:24.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1749999999999999e-06 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365544+0000", 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052690099999999998 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365559+0000", 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4610000000000001e-05 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365644+0000", 2024-03-16T19:30:24.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5447999999999999e-05 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:13 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.676095691, 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00073113500000000005, 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1000000000000001e-07 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365021+0000", 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.23e-07 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365024+0000", 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7369999999999998e-06 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365026+0000", 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1289999999999999e-06 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365663+0000", 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063679600000000002 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365667+0000", 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.233e-06 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365751+0000", 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4307000000000004e-05 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.109 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.109 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:24.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:24.110 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:24.128+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:24.211 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:24.230+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:24.258 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:24.259 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:24.259 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:24.259 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:24.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:24.311 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:24.330+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:24.412 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:24.432+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:24.416 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:24.416 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:24.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:24.513 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:24.531+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:24.519 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.538+0000 7f8a227fc640 1 -- 172.21.15.172:0/1999258373 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v40) v1 ==== 113+0+0 (secure 0 0 0) 0x7f8a28057690 con 0x7f8a2c06f3d0 2024-03-16T19:30:24.519 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.1 to 1 (10000) 2024-03-16T19:30:24.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8a1006b3e0 msgr2=0x7f8a1006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8a1006b3e0 0x7f8a1006d8a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c007fd0 tx=0x7f8a1c009040 comp rx=0 tx=0).stop 2024-03-16T19:30:24.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a2c06f3d0 msgr2=0x7f8a2c1e3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a2c06f3d0 0x7f8a2c1e3e10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800c500 tx=0x7f8a2800c9c0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 shutdown_connections 2024-03-16T19:30:24.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8a1006b3e0 0x7f8a1006d8a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.522 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a2c157fc0 0x7f8a2c1e4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8a2c06f3d0 0x7f8a2c1e3e10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 --2- 172.21.15.172:0/1999258373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a2c06dc00 0x7f8a2c1f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 >> 172.21.15.172:0/1999258373 conn(0x7f8a2c066c20 msgr2=0x7f8a2c155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:24.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 shutdown_connections 2024-03-16T19:30:24.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.541+0000 7f8a34179640 1 -- 172.21.15.172:0/1999258373 wait complete. 2024-03-16T19:30:24.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:30:24.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-16T19:30:24.576 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.238554190999999, 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:30:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.238539717999998, 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.238035228999998, 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:30:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:30:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.238019833000003, 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.237561149999998, 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.237014979999998, 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:30:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.236457889, 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:30:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:30:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.235256141000001, 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:30:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.234677431999998, 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:30:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.234627905000004, 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:30:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:30:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:30:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.233667261999997, 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.229953064, 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:30:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:30:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.789552151999999, 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:30:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.223999149000001, 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.716400424, 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:24.634 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:24.635 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:24.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.650+0000 7ff5a89ac640 1 Processor -- start 2024-03-16T19:30:24.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.650+0000 7ff5a89ac640 1 -- start start 2024-03-16T19:30:24.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a89ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a4152ab0 0x7ff5a40c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a89ac640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff5a40c9af0 0x7ff5a415bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a89ac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a4152e90 con 0x7ff5a406fde0 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a89ac640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a40c85e0 con 0x7ff5a40c9af0 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a89ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40c30e0 con 0x7ff5a4152ab0 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a4152ab0 0x7ff5a40c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a1d9b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff5a40c9af0 0x7ff5a415bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a4152ab0 0x7ff5a40c8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:50854/0 (socket says 172.21.15.172:50854) 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a259c640 1 -- 172.21.15.172:0/3487028625 learned_addr learned my addr 172.21.15.172:0/3487028625 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a259c640 1 -- 172.21.15.172:0/3487028625 >> v2:172.21.15.174:3300/0 conn(0x7ff5a40c9af0 msgr2=0x7ff5a415bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a259c640 1 --2- 172.21.15.172:0/3487028625 >> v2:172.21.15.174:3300/0 conn(0x7ff5a40c9af0 0x7ff5a415bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a259c640 1 -- 172.21.15.172:0/3487028625 >> v1:172.21.15.174:6789/0 conn(0x7ff5a406fde0 legacy=0x7ff5a406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a259c640 1 -- 172.21.15.172:0/3487028625 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a40c33c0 con 0x7ff5a4152ab0 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.651+0000 7ff5a1d9b640 1 --2- 172.21.15.172:0/3487028625 >> v2:172.21.15.174:3300/0 conn(0x7ff5a40c9af0 0x7ff5a415bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.652+0000 7ff5a259c640 1 --2- 172.21.15.172:0/3487028625 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a4152ab0 0x7ff5a40c8ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009870 tx=0x7ff59c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ca2057aafa3e594 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.652+0000 7ff5a159a640 1 -- 172.21.15.172:0/3487028625 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c030840 con 0x7ff5a4152ab0 2024-03-16T19:30:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.652+0000 7ff5a159a640 1 -- 172.21.15.172:0/3487028625 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59c0309a0 con 0x7ff5a4152ab0 2024-03-16T19:30:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.652+0000 7ff5a159a640 1 -- 172.21.15.172:0/3487028625 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c030d30 con 0x7ff5a4152ab0 2024-03-16T19:30:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.653+0000 7ff5a89ac640 1 -- 172.21.15.172:0/3487028625 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a4152ab0 msgr2=0x7ff5a40c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.653+0000 7ff5a89ac640 1 --2- 172.21.15.172:0/3487028625 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a4152ab0 0x7ff5a40c8ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009870 tx=0x7ff59c02b980 comp rx=0 tx=0).stop 2024-03-16T19:30:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.653+0000 7ff5a89ac640 1 -- 172.21.15.172:0/3487028625 shutdown_connections 2024-03-16T19:30:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.653+0000 7ff5a89ac640 1 --2- 172.21.15.172:0/3487028625 >> v2:172.21.15.174:3300/0 conn(0x7ff5a40c9af0 0x7ff5a415bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.653+0000 7ff5a89ac640 1 --2- 172.21.15.172:0/3487028625 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a4152ab0 0x7ff5a40c8ac0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.653+0000 7ff5a89ac640 1 -- 172.21.15.172:0/3487028625 >> 172.21.15.172:0/3487028625 conn(0x7ff5a4070810 msgr2=0x7ff5a4070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.653+0000 7ff5a89ac640 1 -- 172.21.15.172:0/3487028625 shutdown_connections 2024-03-16T19:30:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.653+0000 7ff5a89ac640 1 -- 172.21.15.172:0/3487028625 wait complete. 2024-03-16T19:30:24.639 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:24.654+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.654+0000 7ff5a89ac640 1 Processor -- start 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.654+0000 7ff5a89ac640 1 -- start start 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a89ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a406fde0 0x7ff5a414c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a89ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a40c9af0 0x7ff5a415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a89ac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5a4152ab0 0x7ff5a406c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a89ac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a406bfe0 con 0x7ff5a4152ab0 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a89ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40b2d10 con 0x7ff5a406fde0 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a89ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40b35b0 con 0x7ff5a40c9af0 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a2d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5a4152ab0 0x7ff5a406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a1d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a40c9af0 0x7ff5a415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a406fde0 0x7ff5a414c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a2d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5a4152ab0 0x7ff5a406c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42456/0 (socket says 172.21.15.172:42456) 2024-03-16T19:30:24.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a1d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a40c9af0 0x7ff5a415acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:50866/0 (socket says 172.21.15.172:50866) 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.655+0000 7ff5a2d9d640 1 -- 172.21.15.172:0/4134331540 learned_addr learned my addr 172.21.15.172:0/4134331540 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff5a1d9b640 1 -- 172.21.15.172:0/4134331540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a406fde0 msgr2=0x7ff5a414c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff5a1d9b640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a406fde0 0x7ff5a414c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff5a1d9b640 1 -- 172.21.15.172:0/4134331540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5a4152ab0 msgr2=0x7ff5a406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff5a1d9b640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5a4152ab0 0x7ff5a406c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff5a1d9b640 1 -- 172.21.15.172:0/4134331540 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff598009400 con 0x7ff5a40c9af0 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff5a1d9b640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a40c9af0 0x7ff5a415acf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff59800e8d0 tx=0x7ff59800ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff592ffd640 1 -- 172.21.15.172:0/4134331540 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59800c940 con 0x7ff5a40c9af0 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff592ffd640 1 -- 172.21.15.172:0/4134331540 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59800caa0 con 0x7ff5a40c9af0 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c0094d0 con 0x7ff5a40c9af0 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff592ffd640 1 -- 172.21.15.172:0/4134331540 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59800ce50 con 0x7ff5a40c9af0 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.656+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a40b2f60 con 0x7ff5a40c9af0 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.657+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a406c190 con 0x7ff5a40c9af0 2024-03-16T19:30:24.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.657+0000 7ff592ffd640 1 -- 172.21.15.172:0/4134331540 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff598013470 con 0x7ff5a40c9af0 2024-03-16T19:30:24.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.658+0000 7ff592ffd640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff58406b3e0 0x7ff58406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.658+0000 7ff592ffd640 1 -- 172.21.15.172:0/4134331540 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5828+0+0 (secure 0 0 0) 0x7ff59808a160 con 0x7ff5a40c9af0 2024-03-16T19:30:24.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.658+0000 7ff5a259c640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff58406b3e0 0x7ff58406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.658+0000 7ff5a259c640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff58406b3e0 0x7ff58406d8a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff59c036000 tx=0x7ff59c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:24.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.662+0000 7ff592ffd640 1 -- 172.21.15.172:0/4134331540 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5a406c190 con 0x7ff5a40c9af0 2024-03-16T19:30:24.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.700+0000 7f2794f2c640 1 Processor -- start 2024-03-16T19:30:24.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.700+0000 7f2794f2c640 1 -- start start 2024-03-16T19:30:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f2794f2c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900b5f10 0x7f27900caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f2794f2c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f27900c7800 0x7f27900cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f2794f2c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279015b990 con 0x7f2790154e00 2024-03-16T19:30:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f2794f2c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279015b570 con 0x7f27900c7800 2024-03-16T19:30:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f2794f2c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27901551e0 con 0x7f27900b5f10 2024-03-16T19:30:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900b5f10 0x7f27900caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278ed9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f27900c7800 0x7f27900cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278e59c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f2790154e00 0x7f279006afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:58920/0 (socket says 172.21.15.172:58920) 2024-03-16T19:30:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278e59c640 1 -- 172.21.15.172:0/2832969579 learned_addr learned my addr 172.21.15.172:0/2832969579 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:24.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278d59a640 1 -- 172.21.15.172:0/2832969579 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198810867 0 0) 0x7f279015b990 con 0x7f2790154e00 2024-03-16T19:30:24.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278d59a640 1 -- 172.21.15.172:0/2832969579 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2778003520 con 0x7f2790154e00 2024-03-16T19:30:24.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278dd9b640 1 -- 172.21.15.172:0/2832969579 >> v2:172.21.15.174:3300/0 conn(0x7f27900c7800 msgr2=0x7f27900cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:24.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278dd9b640 1 --2- 172.21.15.172:0/2832969579 >> v2:172.21.15.174:3300/0 conn(0x7f27900c7800 0x7f27900cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278dd9b640 1 -- 172.21.15.172:0/2832969579 >> v1:172.21.15.174:6789/0 conn(0x7f2790154e00 legacy=0x7f279006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278dd9b640 1 -- 172.21.15.172:0/2832969579 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27900cc090 con 0x7f27900b5f10 2024-03-16T19:30:24.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.701+0000 7f278ed9d640 1 --2- 172.21.15.172:0/2832969579 >> v2:172.21.15.174:3300/0 conn(0x7f27900c7800 0x7f27900cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.702+0000 7f278dd9b640 1 --2- 172.21.15.172:0/2832969579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900b5f10 0x7f27900caed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f27840097a0 tx=0x7f278402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7a3afc4584525dc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.702+0000 7f278d59a640 1 -- 172.21.15.172:0/2832969579 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784030610 con 0x7f27900b5f10 2024-03-16T19:30:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.702+0000 7f278d59a640 1 -- 172.21.15.172:0/2832969579 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2784030770 con 0x7f27900b5f10 2024-03-16T19:30:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.702+0000 7f278d59a640 1 -- 172.21.15.172:0/2832969579 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784030b00 con 0x7f27900b5f10 2024-03-16T19:30:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.703+0000 7f2794f2c640 1 -- 172.21.15.172:0/2832969579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900b5f10 msgr2=0x7f27900caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.703+0000 7f2794f2c640 1 --2- 172.21.15.172:0/2832969579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900b5f10 0x7f27900caed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f27840097a0 tx=0x7f278402b960 comp rx=0 tx=0).stop 2024-03-16T19:30:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.703+0000 7f2794f2c640 1 -- 172.21.15.172:0/2832969579 shutdown_connections 2024-03-16T19:30:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.703+0000 7f2794f2c640 1 --2- 172.21.15.172:0/2832969579 >> v2:172.21.15.174:3300/0 conn(0x7f27900c7800 0x7f27900cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.703+0000 7f2794f2c640 1 --2- 172.21.15.172:0/2832969579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900b5f10 0x7f27900caed0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.703+0000 7f2794f2c640 1 -- 172.21.15.172:0/2832969579 >> 172.21.15.172:0/2832969579 conn(0x7f2790066c20 msgr2=0x7f2790067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:24.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.703+0000 7f2794f2c640 1 -- 172.21.15.172:0/2832969579 shutdown_connections 2024-03-16T19:30:24.685 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.703+0000 7f2794f2c640 1 -- 172.21.15.172:0/2832969579 wait complete. 2024-03-16T19:30:24.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.704+0000 7f2794f2c640 1 Processor -- start 2024-03-16T19:30:24.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.704+0000 7f2794f2c640 1 -- start start 2024-03-16T19:30:24.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f2794f2c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27900b5f10 0x7f2790146d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f2794f2c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900c7800 0x7f279014bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f2794f2c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2790154e00 0x7f2790148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f2794f2c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27901f15f0 con 0x7f27900b5f10 2024-03-16T19:30:24.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f2794f2c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27901555d0 con 0x7f2790154e00 2024-03-16T19:30:24.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f2794f2c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2790147260 con 0x7f27900c7800 2024-03-16T19:30:24.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f278e59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27900b5f10 0x7f2790146d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f278dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900c7800 0x7f279014bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f278e59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27900b5f10 0x7f2790146d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42458/0 (socket says 172.21.15.172:42458) 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f278e59c640 1 -- 172.21.15.172:0/223521906 learned_addr learned my addr 172.21.15.172:0/223521906 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f278ed9d640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2790154e00 0x7f2790148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.705+0000 7f278dd9b640 1 -- 172.21.15.172:0/223521906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2790154e00 msgr2=0x7f2790148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f278dd9b640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2790154e00 0x7f2790148b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f278dd9b640 1 -- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27900b5f10 msgr2=0x7f2790146d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f278dd9b640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27900b5f10 0x7f2790146d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f278dd9b640 1 -- 172.21.15.172:0/223521906 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788009a70 con 0x7f27900c7800 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f278e59c640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27900b5f10 0x7f2790146d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f278ed9d640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2790154e00 0x7f2790148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f278dd9b640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900c7800 0x7f279014bcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2784033000 tx=0x7f2784004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f277effd640 1 -- 172.21.15.172:0/223521906 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278403e040 con 0x7f27900c7800 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f277effd640 1 -- 172.21.15.172:0/223521906 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2784039a40 con 0x7f27900c7800 2024-03-16T19:30:24.688 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2784009400 con 0x7f27900c7800 2024-03-16T19:30:24.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f277effd640 1 -- 172.21.15.172:0/223521906 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784039df0 con 0x7f27900c7800 2024-03-16T19:30:24.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.706+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279014ed10 con 0x7f27900c7800 2024-03-16T19:30:24.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.707+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27900cd500 con 0x7f27900c7800 2024-03-16T19:30:24.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.707+0000 7f277effd640 1 -- 172.21.15.172:0/223521906 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f278404c3b0 con 0x7f27900c7800 2024-03-16T19:30:24.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.708+0000 7f277effd640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f277006b3b0 0x7f277006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:24.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.708+0000 7f277effd640 1 -- 172.21.15.172:0/223521906 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f27840abc80 con 0x7f27900c7800 2024-03-16T19:30:24.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.708+0000 7f278e59c640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f277006b3b0 0x7f277006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:24.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.708+0000 7f278e59c640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f277006b3b0 0x7f277006d870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2780005880 tx=0x7f2780009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:24.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.712+0000 7f277effd640 1 -- 172.21.15.172:0/223521906 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2784076880 con 0x7f27900c7800 2024-03-16T19:30:24.736 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:24.754+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:24.766 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:24.766 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:24.766 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:24.767 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:24.778 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:24.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.848+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7ff5a406c190 con 0x7ff5a40c9af0 2024-03-16T19:30:24.862 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:24.881+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:24.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:24.895+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f27900681c0 con 0x7f27900c7800 2024-03-16T19:30:24.925 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:24.925 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:24.933 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:24.938 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:24.957+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:25.039 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:25.057+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:25.081 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:25.081 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.744058291999998, 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.742830869999999, 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.742428164000003, 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.741819413000002, 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.741375544, 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.741232768000003, 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.741218226999997, 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.738340532000002, 2024-03-16T19:30:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.113 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:25.114 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:25.155 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:25.174+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:25.240 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:25.261+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:25.262 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:25.262 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:25.262 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:25.262 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:25.270 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:25.341 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:25.360+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:25.377 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:25.377 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:25.377 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:25.377 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:25.378 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:25.386 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:25.470 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:25.489+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.552+0000 7ff592ffd640 1 -- 172.21.15.172:0/4134331540 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v41) v1 ==== 91+0+0 (secure 0 0 0) 0x7ff5a406c190 con 0x7ff5a40c9af0 2024-03-16T19:30:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.552+0000 7f277effd640 1 -- 172.21.15.172:0/223521906 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v41) v1 ==== 113+0+0 (secure 0 0 0) 0x7f2784034090 con 0x7f27900c7800 2024-03-16T19:30:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:30:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.2 to 1 (10000) 2024-03-16T19:30:25.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff58406b3e0 msgr2=0x7ff58406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff58406b3e0 0x7ff58406d8a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff59c036000 tx=0x7ff59c008030 comp rx=0 tx=0).stop 2024-03-16T19:30:25.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a40c9af0 msgr2=0x7ff5a415acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a40c9af0 0x7ff5a415acf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff59800e8d0 tx=0x7ff59800ed90 comp rx=0 tx=0).stop 2024-03-16T19:30:25.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f277006b3b0 msgr2=0x7f277006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f277006b3b0 0x7f277006d870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2780005880 tx=0x7f2780009040 comp rx=0 tx=0).stop 2024-03-16T19:30:25.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900c7800 msgr2=0x7f279014bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900c7800 0x7f279014bcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2784033000 tx=0x7f2784004730 comp rx=0 tx=0).stop 2024-03-16T19:30:25.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 shutdown_connections 2024-03-16T19:30:25.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff5a4152ab0 0x7ff5a406c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff58406b3e0 0x7ff58406d8a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff5a40c9af0 0x7ff5a415acf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 --2- 172.21.15.172:0/4134331540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5a406fde0 0x7ff5a414c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 >> 172.21.15.172:0/4134331540 conn(0x7ff5a4070810 msgr2=0x7ff5a4074520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:25.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 shutdown_connections 2024-03-16T19:30:25.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2790154e00 0x7f2790148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f277006b3b0 0x7f277006d870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27900c7800 0x7f279014bcd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 --2- 172.21.15.172:0/223521906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27900b5f10 0x7f2790146d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 >> 172.21.15.172:0/223521906 conn(0x7f2790066c20 msgr2=0x7f27900c8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:25.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 shutdown_connections 2024-03-16T19:30:25.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7ff5a89ac640 1 -- 172.21.15.172:0/4134331540 wait complete. 2024-03-16T19:30:25.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 shutdown_connections 2024-03-16T19:30:25.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.555+0000 7f2794f2c640 1 -- 172.21.15.172:0/223521906 wait complete. 2024-03-16T19:30:25.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:25.562+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:25.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.2182855, 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.557 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:30:25.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:30:25.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.218206381999998, 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.217592258000003, 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.565 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:25.569 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.217152188999997, 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:25.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:30:25.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:30:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.216910599000002, 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:25.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:30:25.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:25.577 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:25.578 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:25.602 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:25.677 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:25.697+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:25.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.717+0000 7f63dc2da640 1 Processor -- start 2024-03-16T19:30:25.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.717+0000 7f63dc2da640 1 -- start start 2024-03-16T19:30:25.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.718+0000 7f63dc2da640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d4154e00 0x7f63d406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.718+0000 7f63dc2da640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f63d40b5f10 0x7f63d40caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.718+0000 7f63dc2da640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d415b990 con 0x7f63d406b500 2024-03-16T19:30:25.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.718+0000 7f63dc2da640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d415b570 con 0x7f63d40b5f10 2024-03-16T19:30:25.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.718+0000 7f63dc2da640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d41551e0 con 0x7f63d4154e00 2024-03-16T19:30:25.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.718+0000 7f63da076640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d4154e00 0x7f63d406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.718+0000 7f63d9875640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f63d40b5f10 0x7f63d40caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.718+0000 7f63da877640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f63d406b500 0x7f63d415df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:58922/0 (socket says 172.21.15.172:58922) 2024-03-16T19:30:25.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.718+0000 7f63da877640 1 -- 172.21.15.172:0/2815613060 learned_addr learned my addr 172.21.15.172:0/2815613060 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:25.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.719+0000 7f63d9875640 1 -- 172.21.15.172:0/2815613060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d4154e00 msgr2=0x7f63d406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:25.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.719+0000 7f63d9875640 1 --2- 172.21.15.172:0/2815613060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d4154e00 0x7f63d406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.719+0000 7f63d9875640 1 -- 172.21.15.172:0/2815613060 >> v1:172.21.15.174:6789/0 conn(0x7f63d406b500 legacy=0x7f63d415df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.719+0000 7f63d9875640 1 -- 172.21.15.172:0/2815613060 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d40cc090 con 0x7f63d40b5f10 2024-03-16T19:30:25.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.719+0000 7f63d9074640 1 -- 172.21.15.172:0/2815613060 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460182780 0 0) 0x7f63d415b990 con 0x7f63d406b500 2024-03-16T19:30:25.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.719+0000 7f63da076640 1 --2- 172.21.15.172:0/2815613060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d4154e00 0x7f63d406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.719+0000 7f63d9875640 1 --2- 172.21.15.172:0/2815613060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63d40b5f10 0x7f63d40caed0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f63c40097a0 tx=0x7f63c402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=319fc2f318f792 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.719+0000 7f63d9074640 1 -- 172.21.15.172:0/2815613060 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4030610 con 0x7f63d40b5f10 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63d9074640 1 -- 172.21.15.172:0/2815613060 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4030770 con 0x7f63d40b5f10 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63d9074640 1 -- 172.21.15.172:0/2815613060 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4030b00 con 0x7f63d40b5f10 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63dc2da640 1 -- 172.21.15.172:0/2815613060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63d40b5f10 msgr2=0x7f63d40caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63dc2da640 1 --2- 172.21.15.172:0/2815613060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63d40b5f10 0x7f63d40caed0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f63c40097a0 tx=0x7f63c402b960 comp rx=0 tx=0).stop 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63dc2da640 1 -- 172.21.15.172:0/2815613060 shutdown_connections 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63dc2da640 1 --2- 172.21.15.172:0/2815613060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63d40b5f10 0x7f63d40caed0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63dc2da640 1 --2- 172.21.15.172:0/2815613060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d4154e00 0x7f63d406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63dc2da640 1 -- 172.21.15.172:0/2815613060 >> 172.21.15.172:0/2815613060 conn(0x7f63d4066c20 msgr2=0x7f63d4067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:25.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63dc2da640 1 -- 172.21.15.172:0/2815613060 shutdown_connections 2024-03-16T19:30:25.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.720+0000 7f63dc2da640 1 -- 172.21.15.172:0/2815613060 wait complete. 2024-03-16T19:30:25.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.721+0000 7f63dc2da640 1 Processor -- start 2024-03-16T19:30:25.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63dc2da640 1 -- start start 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63dc2da640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63d406b500 0x7f63d41f0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63dc2da640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d40b5f10 0x7f63d414e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63dc2da640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63d4154e00 0x7f63d40c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63dc2da640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40c42d0 con 0x7f63d406b500 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63dc2da640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d41f4360 con 0x7f63d4154e00 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63dc2da640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d4152ba0 con 0x7f63d40b5f10 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63d9875640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d40b5f10 0x7f63d414e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63da076640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63d406b500 0x7f63d41f0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63d9875640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d40b5f10 0x7f63d414e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:50900/0 (socket says 172.21.15.172:50900) 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63da076640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63d406b500 0x7f63d41f0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42470/0 (socket says 172.21.15.172:42470) 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63d9875640 1 -- 172.21.15.172:0/2910746881 learned_addr learned my addr 172.21.15.172:0/2910746881 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.722+0000 7f63da877640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63d4154e00 0x7f63d40c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63d9875640 1 -- 172.21.15.172:0/2910746881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63d4154e00 msgr2=0x7f63d40c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63d9875640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63d4154e00 0x7f63d40c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63d9875640 1 -- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63d406b500 msgr2=0x7f63d41f0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63d9875640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63d406b500 0x7f63d41f0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63d9875640 1 -- 172.21.15.172:0/2910746881 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc009a70 con 0x7f63d40b5f10 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63da076640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63d406b500 0x7f63d41f0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63d9875640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d40b5f10 0x7f63d414e050 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f63c40098c0 tx=0x7f63c4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63caffd640 1 -- 172.21.15.172:0/2910746881 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c403e040 con 0x7f63d40b5f10 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63caffd640 1 -- 172.21.15.172:0/2910746881 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4039cb0 con 0x7f63d40b5f10 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c4009400 con 0x7f63d40b5f10 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63caffd640 1 -- 172.21.15.172:0/2910746881 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4044630 con 0x7f63d40b5f10 2024-03-16T19:30:25.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.723+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d415a790 con 0x7f63d40b5f10 2024-03-16T19:30:25.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.724+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d41f44a0 con 0x7f63d40b5f10 2024-03-16T19:30:25.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.725+0000 7f63caffd640 1 -- 172.21.15.172:0/2910746881 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f63c404d3b0 con 0x7f63d40b5f10 2024-03-16T19:30:25.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.725+0000 7f63caffd640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f63b806b3b0 0x7f63b806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.725+0000 7f63caffd640 1 -- 172.21.15.172:0/2910746881 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f63c40abf90 con 0x7f63d40b5f10 2024-03-16T19:30:25.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.725+0000 7f63da076640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f63b806b3b0 0x7f63b806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.725+0000 7f63da076640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f63b806b3b0 0x7f63b806d870 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f63cc009fd0 tx=0x7f63cc00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.734+0000 7f63caffd640 1 -- 172.21.15.172:0/2910746881 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63c40dfe10 con 0x7f63d40b5f10 2024-03-16T19:30:25.729 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:25.729 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:25.729 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:25.730 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:25.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.754+0000 7f953ecae640 1 Processor -- start 2024-03-16T19:30:25.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.754+0000 7f953ecae640 1 -- start start 2024-03-16T19:30:25.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.755+0000 7f953ecae640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9538154ed0 0x7f953815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.755+0000 7f953ecae640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95380b4270 con 0x7f95380b3630 2024-03-16T19:30:25.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.755+0000 7f953ecae640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95381552b0 con 0x7f95380b4700 2024-03-16T19:30:25.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.755+0000 7f953ecae640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953815b4b0 con 0x7f9538154ed0 2024-03-16T19:30:25.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.755+0000 7f9537fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f95380b3630 0x7f95380cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40186/0 (socket says 172.21.15.172:40186) 2024-03-16T19:30:25.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.755+0000 7f9537fff640 1 -- 172.21.15.172:0/2115675042 learned_addr learned my addr 172.21.15.172:0/2115675042 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:25.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.755+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2115675042 >> v2:172.21.15.174:3300/0 conn(0x7f9538154ed0 0x7f953815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.756+0000 7f95377fe640 1 -- 172.21.15.172:0/2115675042 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780700447 0 0) 0x7f95380b4270 con 0x7f95380b3630 2024-03-16T19:30:25.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.756+0000 7f95377fe640 1 -- 172.21.15.172:0/2115675042 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9528003520 con 0x7f95380b3630 2024-03-16T19:30:25.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.756+0000 7f953ca4a640 1 -- 172.21.15.172:0/2115675042 >> v1:172.21.15.174:6789/0 conn(0x7f95380b4700 legacy=0x7f95381654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.756+0000 7f95377fe640 1 -- 172.21.15.172:0/2115675042 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170444216 0 0) 0x7f95381552b0 con 0x7f95380b4700 2024-03-16T19:30:25.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.756+0000 7f953ca4a640 1 -- 172.21.15.172:0/2115675042 >> v1:172.21.15.172:6789/0 conn(0x7f95380b3630 legacy=0x7f95380cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.756+0000 7f953ca4a640 1 -- 172.21.15.172:0/2115675042 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953815b790 con 0x7f9538154ed0 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2115675042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9538154ed0 0x7f953815ab40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f952c0097a0 tx=0x7f952c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9de96cea98382a74 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f95377fe640 1 -- 172.21.15.172:0/2115675042 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952c030610 con 0x7f9538154ed0 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f95377fe640 1 -- 172.21.15.172:0/2115675042 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952c030770 con 0x7f9538154ed0 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f95377fe640 1 -- 172.21.15.172:0/2115675042 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952c030b00 con 0x7f9538154ed0 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f953ecae640 1 -- 172.21.15.172:0/2115675042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9538154ed0 msgr2=0x7f953815ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f953ecae640 1 --2- 172.21.15.172:0/2115675042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9538154ed0 0x7f953815ab40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f952c0097a0 tx=0x7f952c02b960 comp rx=0 tx=0).stop 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f953ecae640 1 -- 172.21.15.172:0/2115675042 shutdown_connections 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f953ecae640 1 --2- 172.21.15.172:0/2115675042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9538154ed0 0x7f953815ab40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f953ecae640 1 -- 172.21.15.172:0/2115675042 >> 172.21.15.172:0/2115675042 conn(0x7f953806bf60 msgr2=0x7f953806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:25.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f953ecae640 1 -- 172.21.15.172:0/2115675042 shutdown_connections 2024-03-16T19:30:25.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.757+0000 7f953ecae640 1 -- 172.21.15.172:0/2115675042 wait complete. 2024-03-16T19:30:25.740 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.758+0000 7f953ecae640 1 Processor -- start 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.758+0000 7f953ecae640 1 -- start start 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f953ecae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95380b3630 0x7f9538156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f953ecae640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b4700 0x7f9538155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f953ecae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9538154ed0 0x7f95380c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f953ecae640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9538156e40 con 0x7f95380b4700 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f953ecae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95381f01a0 con 0x7f9538154ed0 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f953ecae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95381f0fe0 con 0x7f95380b3630 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f9537fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b4700 0x7f9538155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f9537fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b4700 0x7f9538155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42474/0 (socket says 172.21.15.172:42474) 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f9537fff640 1 -- 172.21.15.172:0/2655940187 learned_addr learned my addr 172.21.15.172:0/2655940187 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f953d24b640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9538154ed0 0x7f95380c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.759+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95380b3630 0x7f9538156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f9537fff640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95380b3630 msgr2=0x7f9538156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f9537fff640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95380b3630 0x7f9538156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f9537fff640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9538154ed0 msgr2=0x7f95380c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f9537fff640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9538154ed0 0x7f95380c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f9537fff640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c009400 con 0x7f95380b4700 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f953d24b640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9538154ed0 0x7f95380c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f9537fff640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b4700 0x7f9538155700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f952400c460 tx=0x7f952400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524005ce0 con 0x7f95380b4700 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9524005e40 con 0x7f95380b4700 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95381f03d0 con 0x7f95380b4700 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524007290 con 0x7f95380b4700 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.760+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95381f0990 con 0x7f95380b4700 2024-03-16T19:30:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.761+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f95381f1120 con 0x7f95380b4700 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.762+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f95240073f0 con 0x7f95380b4700 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.762+0000 7f95357fa640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f951c06b3e0 0x7f951c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.763+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f95240890c0 con 0x7f95380b4700 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.763+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f951c06b3e0 0x7f951c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.763+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v41) v1 ==== 54+0+16 (secure 0 0 0) 0x7f95381f1120 con 0x7f95380b4700 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.763+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f95380c4c00 0x7f95380d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.763+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95381f1120 con 0x7f95380c4c00 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.763+0000 7f953d24b640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f95380c4c00 0x7f95380d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.763+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f951c06b3e0 0x7f951c06d8a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f952c036000 tx=0x7f952c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.763+0000 7f953d24b640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f95380c4c00 0x7f95380d0240 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.745 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:25.764+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:25.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.764+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f95381f1120 con 0x7f95380c4c00 2024-03-16T19:30:25.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.779+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9538068d20 con 0x7f95380c4c00 2024-03-16T19:30:25.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.780+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9538068d20 con 0x7f95380c4c00 2024-03-16T19:30:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:25.761 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:25.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.780+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f953815edc0 0x7f9538159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.780+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95380cbbe0 con 0x7f953815edc0 2024-03-16T19:30:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.781+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f953815edc0 0x7f9538159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.781+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f953815edc0 0x7f9538159c00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.781+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f95380cbbe0 con 0x7f953815edc0 2024-03-16T19:30:25.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.796+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95380b3f00 con 0x7f953815edc0 2024-03-16T19:30:25.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.798+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f952c03f040 con 0x7f953815edc0 2024-03-16T19:30:25.778 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:25.779 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:25.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.798+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f95381f3c20 0x7f95381f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.798+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95381f4540 con 0x7f95381f3c20 2024-03-16T19:30:25.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.798+0000 7f9537fff640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f95381f3c20 0x7f95381f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.799+0000 7f9537fff640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f95381f3c20 0x7f95381f4000 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.799+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f95381f4540 con 0x7f95381f3c20 2024-03-16T19:30:25.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.813+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95381f0e00 con 0x7f95381f3c20 2024-03-16T19:30:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.815+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f952405bbe0 con 0x7f95381f3c20 2024-03-16T19:30:25.795 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:25.796 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:25.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.815+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9538062200 0x7f95380cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.815+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95381533e0 con 0x7f9538062200 2024-03-16T19:30:25.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.815+0000 7f953d24b640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9538062200 0x7f95380cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.815+0000 7f953d24b640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9538062200 0x7f95380cc760 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.816+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f95381533e0 con 0x7f9538062200 2024-03-16T19:30:25.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.830+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95380628d0 con 0x7f9538062200 2024-03-16T19:30:25.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.833+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f95380628d0 con 0x7f9538062200 2024-03-16T19:30:25.814 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:25.814 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:25.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.833+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f95380cd1b0 0x7f953802bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.833+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9538153100 con 0x7f95380cd1b0 2024-03-16T19:30:25.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.833+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f95380cd1b0 0x7f953802bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.834+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f95380cd1b0 0x7f953802bde0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.835+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9538153100 con 0x7f95380cd1b0 2024-03-16T19:30:25.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.849+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9538153100 con 0x7f95380cd1b0 2024-03-16T19:30:25.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.852+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9538153100 con 0x7f95380cd1b0 2024-03-16T19:30:25.832 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:25.833 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:25.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.852+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f953802c830 0x7f953802ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.852+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95380cd630 con 0x7f953802c830 2024-03-16T19:30:25.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.852+0000 7f9537fff640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f953802c830 0x7f953802ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.852+0000 7f9537fff640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f953802c830 0x7f953802ccb0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.853+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f95380cd630 con 0x7f953802c830 2024-03-16T19:30:25.844 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:25.863+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:25.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.867+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95380cd630 con 0x7f953802c830 2024-03-16T19:30:25.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.869+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f95380cd630 con 0x7f953802c830 2024-03-16T19:30:25.849 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:25.850 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:25.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.869+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9538206fb0 0x7f95382093a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.869+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95380cd700 con 0x7f9538206fb0 2024-03-16T19:30:25.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.869+0000 7f953d24b640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9538206fb0 0x7f95382093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.869+0000 7f953d24b640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9538206fb0 0x7f95382093a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.870+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f95380cd700 con 0x7f9538206fb0 2024-03-16T19:30:25.852 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:25.853 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:25.853 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:25.853 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:25.853 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:25.861 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:25.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.884+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95380cd700 con 0x7f9538206fb0 2024-03-16T19:30:25.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.886+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f95380cd700 con 0x7f9538206fb0 2024-03-16T19:30:25.867 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:25.867 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:25.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.887+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f953820b5b0 0x7f953820b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:25.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.887+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f953820f330 con 0x7f953820b5b0 2024-03-16T19:30:25.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.887+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f953820b5b0 0x7f953820b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:25.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.887+0000 7f953ca4a640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f953820b5b0 0x7f953820b9f0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:25.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.888+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f953820f330 con 0x7f953820b5b0 2024-03-16T19:30:25.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.902+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f953820e1b0 con 0x7f953820b5b0 2024-03-16T19:30:25.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.903+0000 7f95357fa640 1 -- 172.21.15.172:0/2655940187 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f953820e1b0 con 0x7f953820b5b0 2024-03-16T19:30:25.884 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:25.885 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:25.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.903+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f95380c4c00 msgr2=0x7f95380d0240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.903+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f95380c4c00 0x7f95380d0240 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.903+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f953815edc0 msgr2=0x7f9538159c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.903+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f953815edc0 0x7f9538159c00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.903+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f95381f3c20 msgr2=0x7f95381f4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.903+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f95381f3c20 0x7f95381f4000 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9538062200 msgr2=0x7f95380cc760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9538062200 0x7f95380cc760 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f95380cd1b0 msgr2=0x7f953802bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f95380cd1b0 0x7f953802bde0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f953802c830 msgr2=0x7f953802ccb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f953802c830 0x7f953802ccb0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9538206fb0 msgr2=0x7f95382093a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9538206fb0 0x7f95382093a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f953820b5b0 msgr2=0x7f953820b9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f953820b5b0 0x7f953820b9f0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953d24b640 1 -- 172.21.15.172:0/2655940187 reap_dead start 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f951c06b3e0 msgr2=0x7f951c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953d24b640 1 -- 172.21.15.172:0/2655940187 reap_dead start 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f951c06b3e0 0x7f951c06d8a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f952c036000 tx=0x7f952c008030 comp rx=0 tx=0).stop 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b4700 msgr2=0x7f9538155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 --2- 172.21.15.172:0/2655940187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f95380b4700 0x7f9538155700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f952400c460 tx=0x7f952400c920 comp rx=0 tx=0).stop 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953d24b640 1 -- 172.21.15.172:0/2655940187 reap_dead start 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953d24b640 1 -- 172.21.15.172:0/2655940187 reap_dead start 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953d24b640 1 -- 172.21.15.172:0/2655940187 reap_dead start 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 shutdown_connections 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 >> 172.21.15.172:0/2655940187 conn(0x7f953806bf60 msgr2=0x7f9538153ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 shutdown_connections 2024-03-16T19:30:25.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.904+0000 7f953ecae640 1 -- 172.21.15.172:0/2655940187 wait complete. 2024-03-16T19:30:25.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:25.928+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f63d40cc710 con 0x7f63d40b5f10 2024-03-16T19:30:25.945 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:25.964+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:26.007 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:26.007 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:26.016 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:26.057 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:26.076+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:26.128 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:30:26.131 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:30:26.170 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:26.170 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:26.171 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:26.189+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:26.179 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:26.248 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:26.268+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:26.331 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:26.332 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:26.340 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:26.383 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:26.402+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:26.446 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:26.446 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:26.450 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:26.469+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:26.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:26.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.555+0000 7f63caffd640 1 -- 172.21.15.172:0/2910746881 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v42) v1 ==== 113+0+0 (secure 0 0 0) 0x7f63c4076cb0 con 0x7f63d40b5f10 2024-03-16T19:30:26.536 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.3 to 1 (10000) 2024-03-16T19:30:26.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f63b806b3b0 msgr2=0x7f63b806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:26.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f63b806b3b0 0x7f63b806d870 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f63cc009fd0 tx=0x7f63cc00b040 comp rx=0 tx=0).stop 2024-03-16T19:30:26.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d40b5f10 msgr2=0x7f63d414e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:26.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d40b5f10 0x7f63d414e050 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f63c40098c0 tx=0x7f63c4004730 comp rx=0 tx=0).stop 2024-03-16T19:30:26.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 shutdown_connections 2024-03-16T19:30:26.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63d4154e00 0x7f63d40c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:26.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f63b806b3b0 0x7f63b806d870 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:26.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63d40b5f10 0x7f63d414e050 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:26.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 --2- 172.21.15.172:0/2910746881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63d406b500 0x7f63d41f0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:26.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 >> 172.21.15.172:0/2910746881 conn(0x7f63d4066c20 msgr2=0x7f63d4153580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:26.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 shutdown_connections 2024-03-16T19:30:26.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.557+0000 7f63dc2da640 1 -- 172.21.15.172:0/2910746881 wait complete. 2024-03-16T19:30:26.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-16T19:30:26.550 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:26.569+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:26.569 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:26.569 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:26.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:26.663 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:26.682+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:26.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.693+0000 7f926bd93640 1 Processor -- start 2024-03-16T19:30:26.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.693+0000 7f926bd93640 1 -- start start 2024-03-16T19:30:26.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.694+0000 7f926bd93640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f92640c7800 0x7f9264166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.694+0000 7f926bd93640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926415b990 con 0x7f92640b5f10 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.694+0000 7f926bd93640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92640cc090 con 0x7f9264154e00 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.694+0000 7f926bd93640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926415b570 con 0x7f92640c7800 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.694+0000 7f926932e640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f92640b5f10 0x7f92640caed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40200/0 (socket says 172.21.15.172:40200) 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.694+0000 7f926932e640 1 -- 172.21.15.172:0/2526551708 learned_addr learned my addr 172.21.15.172:0/2526551708 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.694+0000 7f926a330640 1 --2- 172.21.15.172:0/2526551708 >> v2:172.21.15.174:3300/0 conn(0x7f92640c7800 0x7f9264166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.695+0000 7f9268b2d640 1 -- 172.21.15.172:0/2526551708 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362637927 0 0) 0x7f926415b990 con 0x7f92640b5f10 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.695+0000 7f9268b2d640 1 -- 172.21.15.172:0/2526551708 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f924c003520 con 0x7f92640b5f10 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.695+0000 7f9268b2d640 1 -- 172.21.15.172:0/2526551708 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332307254 0 0) 0x7f92640cc090 con 0x7f9264154e00 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.695+0000 7f9268b2d640 1 -- 172.21.15.172:0/2526551708 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926415b990 con 0x7f9264154e00 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.695+0000 7f926a330640 1 -- 172.21.15.172:0/2526551708 >> v1:172.21.15.174:6789/0 conn(0x7f9264154e00 legacy=0x7f926406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.695+0000 7f926a330640 1 -- 172.21.15.172:0/2526551708 >> v1:172.21.15.172:6789/0 conn(0x7f92640b5f10 legacy=0x7f92640caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:26.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.695+0000 7f926a330640 1 -- 172.21.15.172:0/2526551708 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926415abd0 con 0x7f92640c7800 2024-03-16T19:30:26.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.696+0000 7f926a330640 1 --2- 172.21.15.172:0/2526551708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92640c7800 0x7f9264166270 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92540097a0 tx=0x7f925402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa098a7ea8a4d853 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:26.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.696+0000 7f9268b2d640 1 -- 172.21.15.172:0/2526551708 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925402f810 con 0x7f92640c7800 2024-03-16T19:30:26.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.696+0000 7f9268b2d640 1 -- 172.21.15.172:0/2526551708 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925402f970 con 0x7f92640c7800 2024-03-16T19:30:26.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.696+0000 7f9268b2d640 1 -- 172.21.15.172:0/2526551708 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925402fd00 con 0x7f92640c7800 2024-03-16T19:30:26.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.697+0000 7f926bd93640 1 -- 172.21.15.172:0/2526551708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92640c7800 msgr2=0x7f9264166270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:26.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.697+0000 7f926bd93640 1 --2- 172.21.15.172:0/2526551708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92640c7800 0x7f9264166270 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92540097a0 tx=0x7f925402b9f0 comp rx=0 tx=0).stop 2024-03-16T19:30:26.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.697+0000 7f926bd93640 1 -- 172.21.15.172:0/2526551708 shutdown_connections 2024-03-16T19:30:26.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.697+0000 7f926bd93640 1 --2- 172.21.15.172:0/2526551708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92640c7800 0x7f9264166270 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:26.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.697+0000 7f926bd93640 1 -- 172.21.15.172:0/2526551708 >> 172.21.15.172:0/2526551708 conn(0x7f9264066c20 msgr2=0x7f9264067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:26.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.697+0000 7f926bd93640 1 -- 172.21.15.172:0/2526551708 shutdown_connections 2024-03-16T19:30:26.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.697+0000 7f926bd93640 1 -- 172.21.15.172:0/2526551708 wait complete. 2024-03-16T19:30:26.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.698+0000 7f926bd93640 1 Processor -- start 2024-03-16T19:30:26.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.698+0000 7f926bd93640 1 -- start start 2024-03-16T19:30:26.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f926bd93640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92640b5f10 0x7f92641523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:26.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f926bd93640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92640c7800 0x7f926414c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:26.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f926bd93640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9264154e00 0x7f9264159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:26.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f926bd93640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9264147a50 con 0x7f92640b5f10 2024-03-16T19:30:26.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f926bd93640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92641466a0 con 0x7f9264154e00 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f926bd93640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9264146e40 con 0x7f92640c7800 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f9269b2f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92640b5f10 0x7f92641523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f9269b2f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92640b5f10 0x7f92641523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42488/0 (socket says 172.21.15.172:42488) 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f926932e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92640c7800 0x7f926414c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f926a330640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9264154e00 0x7f9264159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f9269b2f640 1 -- 172.21.15.172:0/3402952353 learned_addr learned my addr 172.21.15.172:0/3402952353 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f9269b2f640 1 -- 172.21.15.172:0/3402952353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92640c7800 msgr2=0x7f926414c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f9269b2f640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92640c7800 0x7f926414c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f9269b2f640 1 -- 172.21.15.172:0/3402952353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9264154e00 msgr2=0x7f9264159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f9269b2f640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9264154e00 0x7f9264159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f9269b2f640 1 -- 172.21.15.172:0/3402952353 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9254009400 con 0x7f92640b5f10 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f9269b2f640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92640b5f10 0x7f92641523d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c7a0 tx=0x7f925c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f92627fc640 1 -- 172.21.15.172:0/3402952353 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c003b70 con 0x7f92640b5f10 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f92627fc640 1 -- 172.21.15.172:0/3402952353 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c003cd0 con 0x7f92640b5f10 2024-03-16T19:30:26.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.699+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926406b850 con 0x7f92640b5f10 2024-03-16T19:30:26.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.700+0000 7f92627fc640 1 -- 172.21.15.172:0/3402952353 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c00aa90 con 0x7f92640b5f10 2024-03-16T19:30:26.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.700+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9264147400 con 0x7f92640b5f10 2024-03-16T19:30:26.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.701+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92640cd940 con 0x7f92640b5f10 2024-03-16T19:30:26.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.701+0000 7f92627fc640 1 -- 172.21.15.172:0/3402952353 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f925c00abf0 con 0x7f92640b5f10 2024-03-16T19:30:26.683 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.701+0000 7f92627fc640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f925006b3b0 0x7f925006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:26.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.701+0000 7f92627fc640 1 -- 172.21.15.172:0/3402952353 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f925c0899d0 con 0x7f92640b5f10 2024-03-16T19:30:26.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.701+0000 7f926932e640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f925006b3b0 0x7f925006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:26.684 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.702+0000 7f926932e640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f925006b3b0 0x7f925006d870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9264149780 tx=0x7f9258007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:26.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.706+0000 7f92627fc640 1 -- 172.21.15.172:0/3402952353 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f925c054660 con 0x7f92640b5f10 2024-03-16T19:30:26.730 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:26.730 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:26.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:26.752 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:26.771+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:26.853 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:26.872+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:26.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:26.896+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f92640681c0 con 0x7f92640b5f10 2024-03-16T19:30:26.887 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:26.887 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:26.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:26.953 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:26.972+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:27.002 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:27.002 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:27.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:27.055 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:27.074+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:27.118 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:27.118 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:27.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:27.155 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:27.174+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:27.256 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:27.275+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:27.273 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:27.273 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:27.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:27.365 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:27.384+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:27.437 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:27.478 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:27.497+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:27.539 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.558+0000 7f92627fc640 1 -- 172.21.15.172:0/3402952353 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v43) v1 ==== 113+0+0 (secure 0 0 0) 0x7f925c057d20 con 0x7f92640b5f10 2024-03-16T19:30:27.539 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.4 to 1 (10000) 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f925006b3b0 msgr2=0x7f925006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f925006b3b0 0x7f925006d870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9264149780 tx=0x7f9258007d50 comp rx=0 tx=0).stop 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92640b5f10 msgr2=0x7f92641523d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92640b5f10 0x7f92641523d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c7a0 tx=0x7f925c00cc60 comp rx=0 tx=0).stop 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 shutdown_connections 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9264154e00 0x7f9264159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f925006b3b0 0x7f925006d870 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92640c7800 0x7f926414c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 --2- 172.21.15.172:0/3402952353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f92640b5f10 0x7f92641523d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 >> 172.21.15.172:0/3402952353 conn(0x7f9264066c20 msgr2=0x7f92640ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 shutdown_connections 2024-03-16T19:30:27.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.560+0000 7f926bd93640 1 -- 172.21.15.172:0/3402952353 wait complete. 2024-03-16T19:30:27.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:30:27.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-16T19:30:27.589 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:27.589 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:27.589 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:27.608+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:27.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:27.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.675+0000 7efd8d72c640 1 Processor -- start 2024-03-16T19:30:27.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.675+0000 7efd8d72c640 1 -- start start 2024-03-16T19:30:27.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.676+0000 7efd8d72c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd880b0de0 0x7efd8815c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.676+0000 7efd8d72c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880c9cf0 con 0x7efd88152a20 2024-03-16T19:30:27.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.676+0000 7efd8d72c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880c9f90 con 0x7efd880ce4e0 2024-03-16T19:30:27.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.676+0000 7efd8d72c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd880ca250 con 0x7efd880b0de0 2024-03-16T19:30:27.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.676+0000 7efd7ffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd880b0de0 0x7efd8815c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.676+0000 7efd86ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7efd88152a20 0x7efd8806f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40208/0 (socket says 172.21.15.172:40208) 2024-03-16T19:30:27.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.676+0000 7efd7ffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efd880b0de0 0x7efd8815c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42500/0 (socket says 172.21.15.172:42500) 2024-03-16T19:30:27.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.676+0000 7efd86ffd640 1 -- 172.21.15.172:0/2965116883 learned_addr learned my addr 172.21.15.172:0/2965116883 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:27.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.677+0000 7efd867fc640 1 -- 172.21.15.172:0/2965116883 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33284270 0 0) 0x7efd880c9f90 con 0x7efd880ce4e0 2024-03-16T19:30:27.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.677+0000 7efd7ffff640 1 -- 172.21.15.172:0/2965116883 >> v1:172.21.15.174:6789/0 conn(0x7efd880ce4e0 legacy=0x7efd88162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.677+0000 7efd7ffff640 1 -- 172.21.15.172:0/2965116883 >> v1:172.21.15.172:6789/0 conn(0x7efd88152a20 legacy=0x7efd8806f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.677+0000 7efd7ffff640 1 -- 172.21.15.172:0/2965116883 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd880ca530 con 0x7efd880b0de0 2024-03-16T19:30:27.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.678+0000 7efd7ffff640 1 --2- 172.21.15.172:0/2965116883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 0x7efd8815c560 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efd700097a0 tx=0x7efd7002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bb73f032860da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:27.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.678+0000 7efd867fc640 1 -- 172.21.15.172:0/2965116883 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd70030610 con 0x7efd880b0de0 2024-03-16T19:30:27.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.678+0000 7efd867fc640 1 -- 172.21.15.172:0/2965116883 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd70030770 con 0x7efd880b0de0 2024-03-16T19:30:27.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.678+0000 7efd867fc640 1 -- 172.21.15.172:0/2965116883 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd70030b00 con 0x7efd880b0de0 2024-03-16T19:30:27.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.679+0000 7efd8d72c640 1 -- 172.21.15.172:0/2965116883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 msgr2=0x7efd8815c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.679+0000 7efd8d72c640 1 --2- 172.21.15.172:0/2965116883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 0x7efd8815c560 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efd700097a0 tx=0x7efd7002b960 comp rx=0 tx=0).stop 2024-03-16T19:30:27.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.679+0000 7efd8d72c640 1 -- 172.21.15.172:0/2965116883 shutdown_connections 2024-03-16T19:30:27.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.679+0000 7efd8d72c640 1 --2- 172.21.15.172:0/2965116883 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 0x7efd8815c560 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.679+0000 7efd8d72c640 1 -- 172.21.15.172:0/2965116883 >> 172.21.15.172:0/2965116883 conn(0x7efd88066e00 msgr2=0x7efd880671e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:27.661 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:27.679+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:27.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.679+0000 7efd8d72c640 1 -- 172.21.15.172:0/2965116883 shutdown_connections 2024-03-16T19:30:27.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.679+0000 7efd8d72c640 1 -- 172.21.15.172:0/2965116883 wait complete. 2024-03-16T19:30:27.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.680+0000 7efd8d72c640 1 Processor -- start 2024-03-16T19:30:27.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.680+0000 7efd8d72c640 1 -- start start 2024-03-16T19:30:27.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.680+0000 7efd8d72c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 0x7efd881eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.680+0000 7efd8d72c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd880ce4e0 0x7efd881f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.680+0000 7efd8d72c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd88152a20 0x7efd880c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd8d72c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd881f1cc0 con 0x7efd880b0de0 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd8d72c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd881ef1d0 con 0x7efd880ce4e0 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd8d72c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd88153260 con 0x7efd88152a20 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd86ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 0x7efd881eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd86ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 0x7efd881eec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42502/0 (socket says 172.21.15.172:42502) 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd86ffd640 1 -- 172.21.15.172:0/69621017 learned_addr learned my addr 172.21.15.172:0/69621017 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd7ffff640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd880ce4e0 0x7efd881f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd877fe640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd88152a20 0x7efd880c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd86ffd640 1 -- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd88152a20 msgr2=0x7efd880c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd86ffd640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd88152a20 0x7efd880c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd86ffd640 1 -- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd880ce4e0 msgr2=0x7efd881f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd86ffd640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd880ce4e0 0x7efd881f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd86ffd640 1 -- 172.21.15.172:0/69621017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd70009400 con 0x7efd880b0de0 2024-03-16T19:30:27.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd877fe640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd88152a20 0x7efd880c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:27.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd7ffff640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd880ce4e0 0x7efd881f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:27.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd86ffd640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 0x7efd881eec90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efd7800c4a0 tx=0x7efd7800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:27.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd7f7fe640 1 -- 172.21.15.172:0/69621017 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78005ce0 con 0x7efd880b0de0 2024-03-16T19:30:27.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd7f7fe640 1 -- 172.21.15.172:0/69621017 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd78005e40 con 0x7efd880b0de0 2024-03-16T19:30:27.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd881f1920 con 0x7efd880b0de0 2024-03-16T19:30:27.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd7f7fe640 1 -- 172.21.15.172:0/69621017 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78007270 con 0x7efd880b0de0 2024-03-16T19:30:27.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.681+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd88153500 con 0x7efd880b0de0 2024-03-16T19:30:27.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.683+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd881f1780 con 0x7efd880b0de0 2024-03-16T19:30:27.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.683+0000 7efd7f7fe640 1 -- 172.21.15.172:0/69621017 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7efd780073d0 con 0x7efd880b0de0 2024-03-16T19:30:27.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.683+0000 7efd7f7fe640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efd6806b3e0 0x7efd6806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.684+0000 7efd7f7fe640 1 -- 172.21.15.172:0/69621017 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5828+0+0 (secure 0 0 0) 0x7efd78089130 con 0x7efd880b0de0 2024-03-16T19:30:27.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.684+0000 7efd7ffff640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efd6806b3e0 0x7efd6806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.684+0000 7efd7ffff640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efd6806b3e0 0x7efd6806d8a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efd70036000 tx=0x7efd7003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:27.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.689+0000 7efd7f7fe640 1 -- 172.21.15.172:0/69621017 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd881f1780 con 0x7efd880b0de0 2024-03-16T19:30:27.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.715+0000 7f2b9e9ac640 1 Processor -- start 2024-03-16T19:30:27.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.715+0000 7f2b9e9ac640 1 -- start start 2024-03-16T19:30:27.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b9e9ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b9806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b9e9ac640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2b980b5f10 0x7f2b980caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b9e9ac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9815b990 con 0x7f2b9806b500 2024-03-16T19:30:27.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b9e9ac640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9815b570 con 0x7f2b980b5f10 2024-03-16T19:30:27.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b9e9ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b981551e0 con 0x7f2b98154e00 2024-03-16T19:30:27.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b977fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2b980b5f10 0x7f2b980caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b97fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b9806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b977fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2b980b5f10 0x7f2b980caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:58390/0 (socket says 172.21.15.172:58390) 2024-03-16T19:30:27.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b9cf49640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f2b9806b500 0x7f2b9815df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:58964/0 (socket says 172.21.15.172:58964) 2024-03-16T19:30:27.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b97fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b9806afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:50938/0 (socket says 172.21.15.172:50938) 2024-03-16T19:30:27.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b977fe640 1 -- 172.21.15.172:0/2345161087 learned_addr learned my addr 172.21.15.172:0/2345161087 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:27.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b977fe640 1 -- 172.21.15.172:0/2345161087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 msgr2=0x7f2b9806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:27.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b977fe640 1 --2- 172.21.15.172:0/2345161087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b9806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b977fe640 1 -- 172.21.15.172:0/2345161087 >> v1:172.21.15.174:6789/0 conn(0x7f2b9806b500 legacy=0x7f2b9815df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b977fe640 1 -- 172.21.15.172:0/2345161087 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b980cc090 con 0x7f2b980b5f10 2024-03-16T19:30:27.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b96ffd640 1 -- 172.21.15.172:0/2345161087 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019601710 0 0) 0x7f2b9815b990 con 0x7f2b9806b500 2024-03-16T19:30:27.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.716+0000 7f2b97fff640 1 --2- 172.21.15.172:0/2345161087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b9806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:27.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.717+0000 7f2b977fe640 1 --2- 172.21.15.172:0/2345161087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b980b5f10 0x7f2b980caed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c009870 tx=0x7f2b8c02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad078dc761bed0d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:27.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.717+0000 7f2b96ffd640 1 -- 172.21.15.172:0/2345161087 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c02f840 con 0x7f2b980b5f10 2024-03-16T19:30:27.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.717+0000 7f2b96ffd640 1 -- 172.21.15.172:0/2345161087 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8c02f9a0 con 0x7f2b980b5f10 2024-03-16T19:30:27.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.717+0000 7f2b96ffd640 1 -- 172.21.15.172:0/2345161087 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c02fd30 con 0x7f2b980b5f10 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.718+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/2345161087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b980b5f10 msgr2=0x7f2b980caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.718+0000 7f2b9e9ac640 1 --2- 172.21.15.172:0/2345161087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b980b5f10 0x7f2b980caed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c009870 tx=0x7f2b8c02ba20 comp rx=0 tx=0).stop 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.718+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/2345161087 shutdown_connections 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.718+0000 7f2b9e9ac640 1 --2- 172.21.15.172:0/2345161087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b980b5f10 0x7f2b980caed0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.718+0000 7f2b9e9ac640 1 --2- 172.21.15.172:0/2345161087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b9806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.718+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/2345161087 >> 172.21.15.172:0/2345161087 conn(0x7f2b98066c20 msgr2=0x7f2b98067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.718+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/2345161087 shutdown_connections 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.718+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/2345161087 wait complete. 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.719+0000 7f2b9e9ac640 1 Processor -- start 2024-03-16T19:30:27.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.719+0000 7f2b9e9ac640 1 -- start start 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9e9ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b9806b500 0x7f2b981f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9e9ac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980b5f10 0x7f2b98159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9e9ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b98158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9e9ac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b981f10d0 con 0x7f2b980b5f10 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9e9ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9814d760 con 0x7f2b9806b500 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9e9ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9814da00 con 0x7f2b98154e00 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980b5f10 0x7f2b98159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b97fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b9806b500 0x7f2b981f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980b5f10 0x7f2b98159a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42504/0 (socket says 172.21.15.172:42504) 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b977fe640 1 -- 172.21.15.172:0/1020212330 learned_addr learned my addr 172.21.15.172:0/1020212330 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9cf49640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b98158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9cf49640 1 -- 172.21.15.172:0/1020212330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b9806b500 msgr2=0x7f2b981f0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9cf49640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b9806b500 0x7f2b981f0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9cf49640 1 -- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980b5f10 msgr2=0x7f2b98159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9cf49640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980b5f10 0x7f2b98159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b9cf49640 1 -- 172.21.15.172:0/1020212330 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88009400 con 0x7f2b98154e00 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b97fff640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b9806b500 0x7f2b981f0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.720+0000 7f2b977fe640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980b5f10 0x7f2b98159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.721+0000 7f2b9cf49640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b98158e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b9806b9b0 tx=0x7f2b9000cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.721+0000 7f2b94ff9640 1 -- 172.21.15.172:0/1020212330 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90004d60 con 0x7f2b98154e00 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.721+0000 7f2b94ff9640 1 -- 172.21.15.172:0/1020212330 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9000a640 con 0x7f2b98154e00 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.721+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c0094d0 con 0x7f2b98154e00 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.721+0000 7f2b94ff9640 1 -- 172.21.15.172:0/1020212330 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9000a9f0 con 0x7f2b98154e00 2024-03-16T19:30:27.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.721+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9814dfc0 con 0x7f2b98154e00 2024-03-16T19:30:27.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.722+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b980cc710 con 0x7f2b98154e00 2024-03-16T19:30:27.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.723+0000 7f2b94ff9640 1 -- 172.21.15.172:0/1020212330 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f2b9000ab50 con 0x7f2b98154e00 2024-03-16T19:30:27.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.723+0000 7f2b94ff9640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2b8406b3b0 0x7f2b8406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.723+0000 7f2b94ff9640 1 -- 172.21.15.172:0/1020212330 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f2b90089360 con 0x7f2b98154e00 2024-03-16T19:30:27.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.723+0000 7f2b97fff640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2b8406b3b0 0x7f2b8406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.724+0000 7f2b97fff640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2b8406b3b0 0x7f2b8406d870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009fd0 tx=0x7f2b88009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:27.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.732+0000 7f2b94ff9640 1 -- 172.21.15.172:0/1020212330 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b90053ff0 con 0x7f2b98154e00 2024-03-16T19:30:27.751 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:27.751 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:27.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:27.761 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:27.779+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:27.861 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:27.880+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:27.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.881+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7efd881f1780 con 0x7efd880b0de0 2024-03-16T19:30:27.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:27.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:27.925+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f2b980681c0 con 0x7f2b98154e00 2024-03-16T19:30:27.910 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:27.910 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:27.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:27.962 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:27.981+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:27.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.016+0000 7f4755df4640 1 Processor -- start 2024-03-16T19:30:27.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.016+0000 7f4755df4640 1 -- start start 2024-03-16T19:30:27.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f4755df4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f47500c95a0 0x7f4750158630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:27.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f4755df4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47500c9110 con 0x7f47501525a0 2024-03-16T19:30:27.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f4755df4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47500c8150 con 0x7f475006aed0 2024-03-16T19:30:27.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f4755df4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4750158b70 con 0x7f47500c95a0 2024-03-16T19:30:27.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474f7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f47501525a0 0x7f4750071be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40218/0 (socket says 172.21.15.172:40218) 2024-03-16T19:30:27.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474f7fe640 1 -- 172.21.15.172:0/2006377558 learned_addr learned my addr 172.21.15.172:0/2006377558 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:27.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474ffff640 1 --2- 172.21.15.172:0/2006377558 >> v2:172.21.15.174:3300/0 conn(0x7f47500c95a0 0x7f4750158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:27.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474e7fc640 1 -- 172.21.15.172:0/2006377558 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234127397 0 0) 0x7f47500c8150 con 0x7f475006aed0 2024-03-16T19:30:28.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474e7fc640 1 -- 172.21.15.172:0/2006377558 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4738003520 con 0x7f475006aed0 2024-03-16T19:30:28.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474e7fc640 1 -- 172.21.15.172:0/2006377558 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060330648 0 0) 0x7f47500c9110 con 0x7f47501525a0 2024-03-16T19:30:28.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474e7fc640 1 -- 172.21.15.172:0/2006377558 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47500c8150 con 0x7f47501525a0 2024-03-16T19:30:28.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474ffff640 1 -- 172.21.15.172:0/2006377558 >> v1:172.21.15.174:6789/0 conn(0x7f475006aed0 legacy=0x7f4750160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474ffff640 1 -- 172.21.15.172:0/2006377558 >> v1:172.21.15.172:6789/0 conn(0x7f47501525a0 legacy=0x7f4750071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.017+0000 7f474ffff640 1 -- 172.21.15.172:0/2006377558 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475006b3c0 con 0x7f47500c95a0 2024-03-16T19:30:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.018+0000 7f474ffff640 1 --2- 172.21.15.172:0/2006377558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47500c95a0 0x7f4750158630 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4750158cb0 tx=0x7f474802bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79525d343782935e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.018+0000 7f474e7fc640 1 -- 172.21.15.172:0/2006377558 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474802cd80 con 0x7f47500c95a0 2024-03-16T19:30:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.018+0000 7f474e7fc640 1 -- 172.21.15.172:0/2006377558 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4748034440 con 0x7f47500c95a0 2024-03-16T19:30:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.018+0000 7f474e7fc640 1 -- 172.21.15.172:0/2006377558 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47480347d0 con 0x7f47500c95a0 2024-03-16T19:30:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.019+0000 7f4755df4640 1 -- 172.21.15.172:0/2006377558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47500c95a0 msgr2=0x7f4750158630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.019+0000 7f4755df4640 1 --2- 172.21.15.172:0/2006377558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47500c95a0 0x7f4750158630 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4750158cb0 tx=0x7f474802bfd0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.019+0000 7f4755df4640 1 -- 172.21.15.172:0/2006377558 shutdown_connections 2024-03-16T19:30:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.019+0000 7f4755df4640 1 --2- 172.21.15.172:0/2006377558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47500c95a0 0x7f4750158630 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.019+0000 7f4755df4640 1 -- 172.21.15.172:0/2006377558 >> 172.21.15.172:0/2006377558 conn(0x7f4750072b40 msgr2=0x7f4750072f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:28.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.019+0000 7f4755df4640 1 -- 172.21.15.172:0/2006377558 shutdown_connections 2024-03-16T19:30:28.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.019+0000 7f4755df4640 1 -- 172.21.15.172:0/2006377558 wait complete. 2024-03-16T19:30:28.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.020+0000 7f4755df4640 1 Processor -- start 2024-03-16T19:30:28.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.020+0000 7f4755df4640 1 -- start start 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f4755df4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f475006aed0 0x7f47501569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f4755df4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47500c95a0 0x7f47501417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f4755df4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47501525a0 0x7f47500cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f4755df4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47500cc990 con 0x7f47500c95a0 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f4755df4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4750156ef0 con 0x7f475006aed0 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f4755df4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47501ef060 con 0x7f47501525a0 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f474effd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47500c95a0 0x7f47501417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f474effd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47500c95a0 0x7f47501417e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42520/0 (socket says 172.21.15.172:42520) 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f474effd640 1 -- 172.21.15.172:0/3915709086 learned_addr learned my addr 172.21.15.172:0/3915709086 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f474ffff640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47501525a0 0x7f47500cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f474f7fe640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f475006aed0 0x7f47501569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f474effd640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47501525a0 msgr2=0x7f47500cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f474effd640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47501525a0 0x7f47500cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f474effd640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f475006aed0 msgr2=0x7f47501569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.021+0000 7f474effd640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f475006aed0 0x7f47501569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.022+0000 7f474effd640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4748009a70 con 0x7f47500c95a0 2024-03-16T19:30:28.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.022+0000 7f474ffff640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47501525a0 0x7f47500cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:28.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.022+0000 7f474effd640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47500c95a0 0x7f47501417e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f474000c430 tx=0x7f474000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.022+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47400052d0 con 0x7f47500c95a0 2024-03-16T19:30:28.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.022+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4740005450 con 0x7f47500c95a0 2024-03-16T19:30:28.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.022+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47501ee190 con 0x7f47500c95a0 2024-03-16T19:30:28.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.022+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740011890 con 0x7f47500c95a0 2024-03-16T19:30:28.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.022+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47501ee730 con 0x7f47500c95a0 2024-03-16T19:30:28.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.023+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f47500688d0 con 0x7f47500c95a0 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.023+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f47400119f0 con 0x7f47500c95a0 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.024+0000 7f4737fff640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f473c06b3e0 0x7f473c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.024+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f47400891c0 con 0x7f47500c95a0 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.024+0000 7f474f7fe640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f473c06b3e0 0x7f473c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.024+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v43) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4740090070 con 0x7f47500c95a0 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.024+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4750158300 0x7f47501ee950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.024+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475006caa0 con 0x7f4750158300 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.024+0000 7f474f7fe640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f473c06b3e0 0x7f473c06d8a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4744000f60 tx=0x7f4744009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.024+0000 7f474ffff640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4750158300 0x7f47501ee950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.025+0000 7f474ffff640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4750158300 0x7f47501ee950 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.025+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f47480345b0 con 0x7f4750158300 2024-03-16T19:30:28.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.040+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f47500c8da0 con 0x7f4750158300 2024-03-16T19:30:28.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.042+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4748034830 con 0x7f4750158300 2024-03-16T19:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:28.024 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:28.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.042+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f47500cb010 0x7f47500cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.042+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475006d080 con 0x7f47500cb010 2024-03-16T19:30:28.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.042+0000 7f474f7fe640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f47500cb010 0x7f47500cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.042+0000 7f474f7fe640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f47500cb010 0x7f47500cb3f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.043+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4744001470 con 0x7f47500cb010 2024-03-16T19:30:28.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.058+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f47501eee90 con 0x7f47500cb010 2024-03-16T19:30:28.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.060+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f47501eee90 con 0x7f47500cb010 2024-03-16T19:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:28.042 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:28.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.060+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f47501f1bc0 0x7f47501f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.060+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47501f24e0 con 0x7f47501f1bc0 2024-03-16T19:30:28.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.060+0000 7f474effd640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f47501f1bc0 0x7f47501f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.061+0000 7f474effd640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f47501f1bc0 0x7f47501f1fa0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.061+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f47501f24e0 con 0x7f47501f1bc0 2024-03-16T19:30:28.057 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.075+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f475014c3c0 con 0x7f47501f1bc0 2024-03-16T19:30:28.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.077+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f475014c3c0 con 0x7f47501f1bc0 2024-03-16T19:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:28.060 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:28.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.078+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4750062200 0x7f475014e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.078+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750158f90 con 0x7f4750062200 2024-03-16T19:30:28.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.078+0000 7f474ffff640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4750062200 0x7f475014e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.078+0000 7f474ffff640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4750062200 0x7f475014e7a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.079+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4750158f90 con 0x7f4750062200 2024-03-16T19:30:28.063 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:28.082+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:28.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.093+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f47500628b0 con 0x7f4750062200 2024-03-16T19:30:28.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.094+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f47500628b0 con 0x7f4750062200 2024-03-16T19:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:28.076 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:28.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.095+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f475014f1f0 0x7f475002bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.095+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475014f670 con 0x7f475014f1f0 2024-03-16T19:30:28.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.095+0000 7f474f7fe640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f475014f1f0 0x7f475002bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.095+0000 7f474f7fe640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f475014f1f0 0x7f475002bde0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.096+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f475014f670 con 0x7f475014f1f0 2024-03-16T19:30:28.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:28.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.111+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f475014f670 con 0x7f475014f1f0 2024-03-16T19:30:28.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.113+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f475014f670 con 0x7f475014f1f0 2024-03-16T19:30:28.095 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:28.095 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:28.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.113+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f475002c830 0x7f475002cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.113+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475002d1d0 con 0x7f475002c830 2024-03-16T19:30:28.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.113+0000 7f474effd640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f475002c830 0x7f475002cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.113+0000 7f474effd640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f475002c830 0x7f475002cc90 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.114+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f475002d1d0 con 0x7f475002c830 2024-03-16T19:30:28.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.128+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f475002d1d0 con 0x7f475002c830 2024-03-16T19:30:28.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.130+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f475002d1d0 con 0x7f475002c830 2024-03-16T19:30:28.112 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:28.112 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:28.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.130+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4750204fa0 0x7f4750207390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.130+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475020a1e0 con 0x7f4750204fa0 2024-03-16T19:30:28.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.130+0000 7f474ffff640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4750204fa0 0x7f4750207390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.131+0000 7f474ffff640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4750204fa0 0x7f4750207390 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.131+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f475020a1e0 con 0x7f4750204fa0 2024-03-16T19:30:28.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.146+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4750207ba0 con 0x7f4750204fa0 2024-03-16T19:30:28.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.148+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4750207ba0 con 0x7f4750204fa0 2024-03-16T19:30:28.129 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:28.130 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:28.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.148+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4750208280 0x7f47502086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.148+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475020d5f0 con 0x7f4750208280 2024-03-16T19:30:28.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.148+0000 7f474f7fe640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4750208280 0x7f47502086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.148+0000 7f474f7fe640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4750208280 0x7f47502086c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.149+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f475020d5f0 con 0x7f4750208280 2024-03-16T19:30:28.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.163+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4750208c00 con 0x7f4750208280 2024-03-16T19:30:28.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4737fff640 1 -- 172.21.15.172:0/3915709086 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4750208c00 con 0x7f4750208280 2024-03-16T19:30:28.146 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4750158300 msgr2=0x7f47501ee950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f4750158300 0x7f47501ee950 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f47500cb010 msgr2=0x7f47500cb3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f47500cb010 0x7f47500cb3f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f47501f1bc0 msgr2=0x7f47501f1fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f47501f1bc0 0x7f47501f1fa0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4750062200 msgr2=0x7f475014e7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f4750062200 0x7f475014e7a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f475014f1f0 msgr2=0x7f475002bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f475014f1f0 0x7f475002bde0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f475002c830 msgr2=0x7f475002cc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f475002c830 0x7f475002cc90 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4750204fa0 msgr2=0x7f4750207390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f4750204fa0 0x7f4750207390 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4750208280 msgr2=0x7f47502086c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4750208280 0x7f47502086c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f474ffff640 1 -- 172.21.15.172:0/3915709086 reap_dead start 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f473c06b3e0 msgr2=0x7f473c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f473c06b3e0 0x7f473c06d8a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4744000f60 tx=0x7f4744009040 comp rx=0 tx=0).stop 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f474ffff640 1 -- 172.21.15.172:0/3915709086 reap_dead start 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47500c95a0 msgr2=0x7f47501417e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 --2- 172.21.15.172:0/3915709086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47500c95a0 0x7f47501417e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f474000c430 tx=0x7f474000c8f0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f474ffff640 1 -- 172.21.15.172:0/3915709086 reap_dead start 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f474ffff640 1 -- 172.21.15.172:0/3915709086 reap_dead start 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f474ffff640 1 -- 172.21.15.172:0/3915709086 reap_dead start 2024-03-16T19:30:28.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 shutdown_connections 2024-03-16T19:30:28.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.165+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 >> 172.21.15.172:0/3915709086 conn(0x7f4750072b40 msgr2=0x7f47500c7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:28.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.166+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 shutdown_connections 2024-03-16T19:30:28.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.166+0000 7f4755df4640 1 -- 172.21.15.172:0/3915709086 wait complete. 2024-03-16T19:30:28.164 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:28.183+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:28.202 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.864311903999997, 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.864297430999997, 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:28.210 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.863792942000003, 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:28.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:30:28.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:30:28.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.863777546000001, 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:30:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.863318863000003, 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:30:28.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.862772692999997, 2024-03-16T19:30:28.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:30:28.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:28.227 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.862215601999999, 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:30:28.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.231 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.861013853999999, 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:30:28.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:30:28.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:30:28.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.860435144999997, 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.860385618000002, 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.859424975000003, 2024-03-16T19:30:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:30:28.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:30:28.245 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.855710776999999, 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:30:28.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:30:28.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.415309864999998, 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.250 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:30:28.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.849756862, 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.254 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.342158136999998, 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:28.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:28.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:30:28.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:30:28.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.262 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:28.262 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:28.262 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:28.263 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:28.265 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:28.284+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:28.367 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:28.386+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:28.369 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:28.369 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:28.378 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:28.468 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:28.487+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:28.485 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:28.485 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:28.485 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:28.485 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:28.485 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:28.493 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:28.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.565+0000 7efd7f7fe640 1 -- 172.21.15.172:0/69621017 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v44) v1 ==== 101+0+0 (secure 0 0 0) 0x7efd881f1780 con 0x7efd880b0de0 2024-03-16T19:30:28.547 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:30:28.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efd6806b3e0 msgr2=0x7efd6806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efd6806b3e0 0x7efd6806d8a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efd70036000 tx=0x7efd7003b040 comp rx=0 tx=0).stop 2024-03-16T19:30:28.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 msgr2=0x7efd881eec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 0x7efd881eec90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efd7800c4a0 tx=0x7efd7800c960 comp rx=0 tx=0).stop 2024-03-16T19:30:28.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 shutdown_connections 2024-03-16T19:30:28.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efd6806b3e0 0x7efd6806d8a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd88152a20 0x7efd880c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd880ce4e0 0x7efd881f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 --2- 172.21.15.172:0/69621017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd880b0de0 0x7efd881eec90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.567+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 >> 172.21.15.172:0/69621017 conn(0x7efd88066e00 msgr2=0x7efd880cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:28.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.568+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 shutdown_connections 2024-03-16T19:30:28.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.568+0000 7efd8d72c640 1 -- 172.21.15.172:0/69621017 wait complete. 2024-03-16T19:30:28.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.574+0000 7f2b94ff9640 1 -- 172.21.15.172:0/1020212330 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v44) v1 ==== 113+0+0 (secure 0 0 0) 0x7f2b900576b0 con 0x7f2b98154e00 2024-03-16T19:30:28.555 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.5 to 1 (10000) 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2b8406b3b0 msgr2=0x7f2b8406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2b8406b3b0 0x7f2b8406d870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009fd0 tx=0x7f2b88009190 comp rx=0 tx=0).stop 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 msgr2=0x7f2b98158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b98158e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b9806b9b0 tx=0x7f2b9000cb60 comp rx=0 tx=0).stop 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 shutdown_connections 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2b8406b3b0 0x7f2b8406d870 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b98154e00 0x7f2b98158e50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b980b5f10 0x7f2b98159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 --2- 172.21.15.172:0/1020212330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b9806b500 0x7f2b981f0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 >> 172.21.15.172:0/1020212330 conn(0x7f2b98066c20 msgr2=0x7f2b980cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:28.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 shutdown_connections 2024-03-16T19:30:28.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.576+0000 7f2b9e9ac640 1 -- 172.21.15.172:0/1020212330 wait complete. 2024-03-16T19:30:28.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-16T19:30:28.570 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:28.588+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:28.604 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:28.604 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.266740538999997, 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:28.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:28.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:28.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.265513116999998, 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:28.610 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.613 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.265110411000002, 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:28.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:28.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:28.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.264501660000001, 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.264057790999999, 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:28.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.263915015000002, 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:28.626 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:28.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:28.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:28.629 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.263900474000003, 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:28.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:28.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:28.633 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.261022779000001, 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:28.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:28.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:28.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:28.637 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.637 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:28.637 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.637 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:28.637 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:28.637 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:28.638 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:28.682 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:28.701+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:28.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.758+0000 7f2e165f4640 1 Processor -- start 2024-03-16T19:30:28.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.758+0000 7f2e165f4640 1 -- start start 2024-03-16T19:30:28.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.759+0000 7f2e165f4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2e1006f3d0 0x7f2e1006e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.759+0000 7f2e165f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006dc00 0x7f2e1006ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.759+0000 7f2e165f4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e101583a0 con 0x7f2e10157fc0 2024-03-16T19:30:28.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.759+0000 7f2e165f4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1006c620 con 0x7f2e1006f3d0 2024-03-16T19:30:28.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.759+0000 7f2e165f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1015ed70 con 0x7f2e1006dc00 2024-03-16T19:30:28.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.759+0000 7f2e0ffff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f2e10157fc0 0x7f2e100c8ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39286/0 (socket says 172.21.15.172:39286) 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.759+0000 7f2e0ffff640 1 -- 172.21.15.172:0/2014738547 learned_addr learned my addr 172.21.15.172:0/2014738547 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.759+0000 7f2e0f7fe640 1 --2- 172.21.15.172:0/2014738547 >> v2:172.21.15.174:3300/0 conn(0x7f2e1006f3d0 0x7f2e1006e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.759+0000 7f2e14b91640 1 --2- 172.21.15.172:0/2014738547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006dc00 0x7f2e1006ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.760+0000 7f2e0effd640 1 -- 172.21.15.172:0/2014738547 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140955796 0 0) 0x7f2e101583a0 con 0x7f2e10157fc0 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.760+0000 7f2e0effd640 1 -- 172.21.15.172:0/2014738547 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df8003520 con 0x7f2e10157fc0 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.760+0000 7f2e14b91640 1 -- 172.21.15.172:0/2014738547 >> v2:172.21.15.174:3300/0 conn(0x7f2e1006f3d0 msgr2=0x7f2e1006e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.760+0000 7f2e14b91640 1 --2- 172.21.15.172:0/2014738547 >> v2:172.21.15.174:3300/0 conn(0x7f2e1006f3d0 0x7f2e1006e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.760+0000 7f2e14b91640 1 -- 172.21.15.172:0/2014738547 >> v1:172.21.15.172:6789/0 conn(0x7f2e10157fc0 legacy=0x7f2e100c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.760+0000 7f2e14b91640 1 -- 172.21.15.172:0/2014738547 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1006c340 con 0x7f2e1006dc00 2024-03-16T19:30:28.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.760+0000 7f2e0f7fe640 1 --2- 172.21.15.172:0/2014738547 >> v2:172.21.15.174:3300/0 conn(0x7f2e1006f3d0 0x7f2e1006e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:28.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e14b91640 1 --2- 172.21.15.172:0/2014738547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006dc00 0x7f2e1006ebd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e08009d90 tx=0x7f2e0802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=472fb423fcf9ce06 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e0effd640 1 -- 172.21.15.172:0/2014738547 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0802cbc0 con 0x7f2e1006dc00 2024-03-16T19:30:28.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e0effd640 1 -- 172.21.15.172:0/2014738547 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0802cd20 con 0x7f2e1006dc00 2024-03-16T19:30:28.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e0effd640 1 -- 172.21.15.172:0/2014738547 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08034640 con 0x7f2e1006dc00 2024-03-16T19:30:28.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e165f4640 1 -- 172.21.15.172:0/2014738547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006dc00 msgr2=0x7f2e1006ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e165f4640 1 --2- 172.21.15.172:0/2014738547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006dc00 0x7f2e1006ebd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e08009d90 tx=0x7f2e0802be10 comp rx=0 tx=0).stop 2024-03-16T19:30:28.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e165f4640 1 -- 172.21.15.172:0/2014738547 shutdown_connections 2024-03-16T19:30:28.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e165f4640 1 --2- 172.21.15.172:0/2014738547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006dc00 0x7f2e1006ebd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e165f4640 1 --2- 172.21.15.172:0/2014738547 >> v2:172.21.15.174:3300/0 conn(0x7f2e1006f3d0 0x7f2e1006e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e165f4640 1 -- 172.21.15.172:0/2014738547 >> 172.21.15.172:0/2014738547 conn(0x7f2e10066c20 msgr2=0x7f2e10067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:28.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e165f4640 1 -- 172.21.15.172:0/2014738547 shutdown_connections 2024-03-16T19:30:28.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.761+0000 7f2e165f4640 1 -- 172.21.15.172:0/2014738547 wait complete. 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.762+0000 7f2e165f4640 1 Processor -- start 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e165f4640 1 -- start start 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e165f4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1006dc00 0x7f2e101f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e165f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006f3d0 0x7f2e10159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e165f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10157fc0 0x7f2e10154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e165f4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1015a340 con 0x7f2e1006dc00 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e165f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e101e7680 con 0x7f2e10157fc0 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e165f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e10154cc0 con 0x7f2e1006f3d0 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e0ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1006dc00 0x7f2e101f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e0ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1006dc00 0x7f2e101f0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47516/0 (socket says 172.21.15.172:47516) 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e0ffff640 1 -- 172.21.15.172:0/3555053622 learned_addr learned my addr 172.21.15.172:0/3555053622 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e14b91640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10157fc0 0x7f2e10154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.763+0000 7f2e0f7fe640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006f3d0 0x7f2e10159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e0ffff640 1 -- 172.21.15.172:0/3555053622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006f3d0 msgr2=0x7f2e10159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e0ffff640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006f3d0 0x7f2e10159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e0ffff640 1 -- 172.21.15.172:0/3555053622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10157fc0 msgr2=0x7f2e10154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e0ffff640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10157fc0 0x7f2e10154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e0ffff640 1 -- 172.21.15.172:0/3555053622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e04009400 con 0x7f2e1006dc00 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e0ffff640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1006dc00 0x7f2e101f0160 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000c430 tx=0x7f2e0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e0cff9640 1 -- 172.21.15.172:0/3555053622 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00005ce0 con 0x7f2e1006dc00 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e0cff9640 1 -- 172.21.15.172:0/3555053622 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e00005e40 con 0x7f2e1006dc00 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e08009a70 con 0x7f2e1006dc00 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e0cff9640 1 -- 172.21.15.172:0/3555053622 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e000072f0 con 0x7f2e1006dc00 2024-03-16T19:30:28.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.764+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e10158d60 con 0x7f2e1006dc00 2024-03-16T19:30:28.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.765+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e100681c0 con 0x7f2e1006dc00 2024-03-16T19:30:28.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.765+0000 7f2e0cff9640 1 -- 172.21.15.172:0/3555053622 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f2e00007450 con 0x7f2e1006dc00 2024-03-16T19:30:28.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.766+0000 7f2e0cff9640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2dfc06b3b0 0x7f2dfc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:28.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.766+0000 7f2e0cff9640 1 -- 172.21.15.172:0/3555053622 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f2e000892e0 con 0x7f2e1006dc00 2024-03-16T19:30:28.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.766+0000 7f2e0f7fe640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2dfc06b3b0 0x7f2dfc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:28.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.766+0000 7f2e0f7fe640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2dfc06b3b0 0x7f2dfc06d870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2e1015a9c0 tx=0x7f2e04009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:28.749 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:28.750 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:28.750 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:28.750 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:28.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.770+0000 7f2e0cff9640 1 -- 172.21.15.172:0/3555053622 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e00053ee0 con 0x7f2e1006dc00 2024-03-16T19:30:28.758 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:28.770 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:28.788+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:28.870 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:28.889+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:28.905 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:28.905 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:28.913 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:28.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:28.953+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f2e101e7aa0 con 0x7f2e1006dc00 2024-03-16T19:30:28.971 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:28.990+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:29.021 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:29.021 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:29.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:29.021 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:29.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.021 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:29.021 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:29.021 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.682481566, 2024-03-16T19:30:29.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:30:29.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:30:29.025 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.682402447999998, 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:30:29.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:30:29.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:30:29.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.681788324000003, 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.031 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:30:29.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.681348255000003, 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.681106665000001, 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.040 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:30:29.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:30:29.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:30:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:30:29.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:30:29.045 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.045 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.045 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.045 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.045 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.045 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:29.046 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:29.090 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:29.109+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:29.153 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:29.154 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:29.154 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:29.154 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:29.162 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:29.173 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:29.192+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:29.274 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:29.293+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:29.310 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:29.310 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:29.310 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:29.310 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:29.310 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:29.318 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:29.375 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:29.394+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:29.428 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:29.428 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.089146393, 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:29.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:29.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.431 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.431 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:30:29.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:30:29.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.431 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.431 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:30:29.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:29.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.08912222, 2024-03-16T19:30:29.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:30:29.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.089116392999998, 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.442 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.445 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.089112481000001, 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.089106499000003, 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.089102832999998, 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.089092172999997, 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:30:29.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.464 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.089088283999999, 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.468 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.085571592999997, 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:30:29.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.085501158, 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:30:29.472 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:30:29.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:30:29.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.085118811000001, 2024-03-16T19:30:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:30:29.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:30:29.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:30:29.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.083352527999999, 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:30:29.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:30:29.484 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.081539606, 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:30:29.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:29.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.080683856999997, 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:30:29.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.074823090999999, 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:29.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:29.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.642271882999999, 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:30:29.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:30:29.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:30:29.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 48.074778176999999, 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.502 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:30:29.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:30:29.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:30:29.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 46.567165926000001, 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:30:29.507 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:30:29.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:30:29.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.070302700999999, 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0697667849999997, 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:30:29.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:30:29.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:30:29.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:30:29.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.521 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.521 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.521 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:29.522 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:29.523 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:29.541+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:29.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.567+0000 7f2e0cff9640 1 -- 172.21.15.172:0/3555053622 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v45) v1 ==== 113+0+0 (secure 0 0 0) 0x7f2e00017090 con 0x7f2e1006dc00 2024-03-16T19:30:29.548 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.6 to 1 (10000) 2024-03-16T19:30:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2dfc06b3b0 msgr2=0x7f2dfc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2dfc06b3b0 0x7f2dfc06d870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2e1015a9c0 tx=0x7f2e04009190 comp rx=0 tx=0).stop 2024-03-16T19:30:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1006dc00 msgr2=0x7f2e101f0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1006dc00 0x7f2e101f0160 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000c430 tx=0x7f2e0000c8f0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 shutdown_connections 2024-03-16T19:30:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10157fc0 0x7f2e10154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2dfc06b3b0 0x7f2dfc06d870 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e1006f3d0 0x7f2e10159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 --2- 172.21.15.172:0/3555053622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e1006dc00 0x7f2e101f0160 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 >> 172.21.15.172:0/3555053622 conn(0x7f2e10066c20 msgr2=0x7f2e101572a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 shutdown_connections 2024-03-16T19:30:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.570+0000 7f2e165f4640 1 -- 172.21.15.172:0/3555053622 wait complete. 2024-03-16T19:30:29.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-16T19:30:29.623 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:29.642+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:29.673 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:29.673 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:29.673 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:29.673 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:29.682 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:29.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.732+0000 7f8146f3a640 1 Processor -- start 2024-03-16T19:30:29.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.732+0000 7f8146f3a640 1 -- start start 2024-03-16T19:30:29.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.732+0000 7f8146f3a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154a90 0x7f81400c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.732+0000 7f8146f3a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8140154e70 0x7f814006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.732+0000 7f8146f3a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f814015df50 con 0x7f81400cc280 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.732+0000 7f8146f3a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f814015e1f0 con 0x7f8140154e70 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.732+0000 7f8146f3a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814015e4d0 con 0x7f8140154a90 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f8144cd6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154a90 0x7f81400c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f813ffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8140154e70 0x7f814006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f81454d7640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f81400cc280 0x7f81400cae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:40990/0 (socket says 172.21.15.172:40990) 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f8144cd6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154a90 0x7f81400c50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:57236/0 (socket says 172.21.15.172:57236) 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f81454d7640 1 -- 172.21.15.172:0/3394821182 learned_addr learned my addr 172.21.15.172:0/3394821182 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f813ffff640 1 -- 172.21.15.172:0/3394821182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154a90 msgr2=0x7f81400c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:29.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f813ffff640 1 --2- 172.21.15.172:0/3394821182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154a90 0x7f81400c50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f813ffff640 1 -- 172.21.15.172:0/3394821182 >> v1:172.21.15.174:6789/0 conn(0x7f81400cc280 legacy=0x7f81400cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:29.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f813ffff640 1 -- 172.21.15.172:0/3394821182 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814015e7b0 con 0x7f8140154e70 2024-03-16T19:30:29.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f813f7fe640 1 -- 172.21.15.172:0/3394821182 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327366115 0 0) 0x7f814015df50 con 0x7f81400cc280 2024-03-16T19:30:29.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.733+0000 7f8144cd6640 1 --2- 172.21.15.172:0/3394821182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154a90 0x7f81400c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.734+0000 7f813ffff640 1 --2- 172.21.15.172:0/3394821182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8140154e70 0x7f814006b8c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f81280097a0 tx=0x7f812802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab3e1607c53af182 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.734+0000 7f813f7fe640 1 -- 172.21.15.172:0/3394821182 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128030610 con 0x7f8140154e70 2024-03-16T19:30:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.734+0000 7f813f7fe640 1 -- 172.21.15.172:0/3394821182 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8128030770 con 0x7f8140154e70 2024-03-16T19:30:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.734+0000 7f813f7fe640 1 -- 172.21.15.172:0/3394821182 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128030b00 con 0x7f8140154e70 2024-03-16T19:30:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.734+0000 7f8146f3a640 1 -- 172.21.15.172:0/3394821182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8140154e70 msgr2=0x7f814006b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.734+0000 7f8146f3a640 1 --2- 172.21.15.172:0/3394821182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8140154e70 0x7f814006b8c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f81280097a0 tx=0x7f812802b960 comp rx=0 tx=0).stop 2024-03-16T19:30:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.735+0000 7f8146f3a640 1 -- 172.21.15.172:0/3394821182 shutdown_connections 2024-03-16T19:30:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.735+0000 7f8146f3a640 1 --2- 172.21.15.172:0/3394821182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8140154e70 0x7f814006b8c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.735+0000 7f8146f3a640 1 --2- 172.21.15.172:0/3394821182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154a90 0x7f81400c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.735+0000 7f8146f3a640 1 -- 172.21.15.172:0/3394821182 >> 172.21.15.172:0/3394821182 conn(0x7f8140067250 msgr2=0x7f81400d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.735+0000 7f8146f3a640 1 -- 172.21.15.172:0/3394821182 shutdown_connections 2024-03-16T19:30:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.735+0000 7f8146f3a640 1 -- 172.21.15.172:0/3394821182 wait complete. 2024-03-16T19:30:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.736+0000 7f8146f3a640 1 Processor -- start 2024-03-16T19:30:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.736+0000 7f8146f3a640 1 -- start start 2024-03-16T19:30:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.736+0000 7f8146f3a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81400cc280 0x7f81401f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:29.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8146f3a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8140154a90 0x7f814015cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:29.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8146f3a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154e70 0x7f8140145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:29.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8146f3a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814014f8b0 con 0x7f8140154a90 2024-03-16T19:30:29.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8146f3a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814014eca0 con 0x7f81400cc280 2024-03-16T19:30:29.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8146f3a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814014ef80 con 0x7f8140154e70 2024-03-16T19:30:29.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8144cd6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81400cc280 0x7f81401f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:29.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8140154a90 0x7f814015cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:29.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f81454d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154e70 0x7f8140145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:29.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8144cd6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81400cc280 0x7f81401f0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:41064/0 (socket says 172.21.15.172:41064) 2024-03-16T19:30:29.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8144cd6640 1 -- 172.21.15.172:0/3715997725 learned_addr learned my addr 172.21.15.172:0/3715997725 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:29.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813ffff640 1 -- 172.21.15.172:0/3715997725 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154e70 msgr2=0x7f8140145d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:29.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813ffff640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154e70 0x7f8140145d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813ffff640 1 -- 172.21.15.172:0/3715997725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81400cc280 msgr2=0x7f81401f0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:29.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813ffff640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81400cc280 0x7f81401f0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:29.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813ffff640 1 -- 172.21.15.172:0/3715997725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8138009a70 con 0x7f8140154a90 2024-03-16T19:30:29.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813ffff640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8140154a90 0x7f814015cb00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8128033000 tx=0x7f81280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:29.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813d7fa640 1 -- 172.21.15.172:0/3715997725 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812803e070 con 0x7f8140154a90 2024-03-16T19:30:29.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813d7fa640 1 -- 172.21.15.172:0/3715997725 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8128030e30 con 0x7f8140154a90 2024-03-16T19:30:29.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8128009400 con 0x7f8140154a90 2024-03-16T19:30:29.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f813d7fa640 1 -- 172.21.15.172:0/3715997725 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128039c50 con 0x7f8140154a90 2024-03-16T19:30:29.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.737+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814014f540 con 0x7f8140154a90 2024-03-16T19:30:29.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.738+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81400681c0 con 0x7f8140154a90 2024-03-16T19:30:29.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.739+0000 7f813d7fa640 1 -- 172.21.15.172:0/3715997725 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f81280423f0 con 0x7f8140154a90 2024-03-16T19:30:29.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.739+0000 7f813d7fa640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f812006b3e0 0x7f812006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:29.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.739+0000 7f813d7fa640 1 -- 172.21.15.172:0/3715997725 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f81280ab620 con 0x7f8140154a90 2024-03-16T19:30:29.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.739+0000 7f8144cd6640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f812006b3e0 0x7f812006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:29.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.740+0000 7f8144cd6640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f812006b3e0 0x7f812006d8a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f813800f600 tx=0x7f813800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:29.728 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.743+0000 7f813d7fa640 1 -- 172.21.15.172:0/3715997725 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81280761a0 con 0x7f8140154a90 2024-03-16T19:30:29.728 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:29.747+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:29.830 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:29.848+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:29.832 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:29.832 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:29.832 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:29.832 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:29.832 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:29.841 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:29.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:29.927+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f814014f0c0 con 0x7f8140154a90 2024-03-16T19:30:29.931 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:29.950+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:29.949 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:29.949 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:29.949 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 73.738082874, 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:30:29.950 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:30:29.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:30:29.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.954 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:29.955 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 73.361387327000003, 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.958 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.959 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.585410882999994, 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:30:29.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.962 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:30:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:30:29.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.585326469999998, 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:30:29.965 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:30:29.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:30:29.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.969 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.969 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.969 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:30:29.969 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:30:29.969 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.969 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.970 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.970 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:30:29.970 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:30:29.970 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.970 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.971 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:29.971 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:30:29.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:30:29.971 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.971 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.971 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.972 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:30:29.972 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:30:29.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.972 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.972 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.972 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:30:29.973 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:30:29.973 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.973 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.973 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:30:29.973 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:30:29.974 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.974 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.974 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.974 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:30:29.974 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:30:29.975 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.975 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.975 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.975 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.975 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.975 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:29.975 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.585319114000001, 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:30:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:30:29.978 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:29.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:29.981 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:29.982 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.392779922000003, 2024-03-16T19:30:29.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:30:29.982 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.982 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.982 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:29.982 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:29.983 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:29.983 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:30:29.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.983 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:29.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:29.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.985 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.985 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:30:29.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:30:29.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.985 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.985 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:30:29.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:30:29.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:30:29.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:30:29.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:30:29.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:30:29.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:30:29.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:30:29.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:30:29.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:30:29.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.990 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.990 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:30:29.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:30:29.991 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.991 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.991 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.991 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.392774114999995, 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:30:29.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:30:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:30:29.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:29.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:29.997 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.392655981999994, 2024-03-16T19:30:29.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:30:29.998 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:29.998 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:29.998 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:29.998 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:29.998 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:29.998 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:30:29.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.998 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:30:29.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:30:30.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:30.003 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.392560785000001, 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:30:30.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:30:30.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:30:30.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:30:30.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.392556773999999, 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.009 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.014 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.973624900000004, 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:30.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:30:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.973575530999994, 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.020 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:30:30.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.025 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.973496698999995, 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.026 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:30:30.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:30:30.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:30:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.973493488000003, 2024-03-16T19:30:30.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:30:30.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:30:30.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:30:30.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:30:30.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:30:30.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:30:30.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:30:30.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:30:30.037 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.973490534999996, 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.038 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.043 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.044 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:30.044 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.973375560999997, 2024-03-16T19:30:30.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:30:30.044 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.044 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.044 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.045 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.045 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.045 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:30:30.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.045 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:30.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:30:30.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:30:30.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:30:30.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:30:30.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:30:30.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:30:30.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:30:30.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:30:30.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:30:30.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:30:30.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:30:30.053 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.053 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.053 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.973372209000004, 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:30:30.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.973369093000002, 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.059 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:30.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.973308623999998, 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:30.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:30:30.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:30:30.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:30:30.069 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.364628573999994, 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:30.070 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:30.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:30:30.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:30:30.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:30:30.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:30.075 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:30.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:30.077 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:30.096+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:30.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:30.177 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:30.195+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:30.188 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:30.189 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:30.189 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:30.189 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:30.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:30.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.279+0000 7f102d39a640 1 Processor -- start 2024-03-16T19:30:30.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.279+0000 7f102d39a640 1 -- start start 2024-03-16T19:30:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.280+0000 7f102d39a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1028154a70 0x7f102806c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.280+0000 7f102d39a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1028154e50 0x7f102806b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.280+0000 7f102d39a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102815b900 con 0x7f10280c5190 2024-03-16T19:30:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.280+0000 7f102d39a640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10280cad10 con 0x7f1028154a70 2024-03-16T19:30:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.280+0000 7f102d39a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102815ad80 con 0x7f1028154e50 2024-03-16T19:30:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.280+0000 7f1026ffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1028154a70 0x7f102806c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.280+0000 7f1026ffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1028154a70 0x7f102806c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47540/0 (socket says 172.21.15.172:47540) 2024-03-16T19:30:30.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.280+0000 7f1026ffd640 1 -- 172.21.15.172:0/4156477796 learned_addr learned my addr 172.21.15.172:0/4156477796 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:30.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.280+0000 7f101ffff640 1 --2- 172.21.15.172:0/4156477796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1028154e50 0x7f102806b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.281+0000 7f1026ffd640 1 -- 172.21.15.172:0/4156477796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1028154e50 msgr2=0x7f102806b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:30.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.281+0000 7f1026ffd640 1 --2- 172.21.15.172:0/4156477796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1028154e50 0x7f102806b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.281+0000 7f1026ffd640 1 -- 172.21.15.172:0/4156477796 >> v1:172.21.15.174:6789/0 conn(0x7f10280c5190 legacy=0x7f10280cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.281+0000 7f1026ffd640 1 -- 172.21.15.172:0/4156477796 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102815b060 con 0x7f1028154a70 2024-03-16T19:30:30.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.281+0000 7f101ffff640 1 --2- 172.21.15.172:0/4156477796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1028154e50 0x7f102806b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.281+0000 7f1026ffd640 1 --2- 172.21.15.172:0/4156477796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1028154a70 0x7f102806c490 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1018009870 tx=0x7f101802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e056385cfb345f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f10267fc640 1 -- 172.21.15.172:0/4156477796 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018031640 con 0x7f1028154a70 2024-03-16T19:30:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f10267fc640 1 -- 172.21.15.172:0/4156477796 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10180317a0 con 0x7f1028154a70 2024-03-16T19:30:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f10267fc640 1 -- 172.21.15.172:0/4156477796 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018031b30 con 0x7f1028154a70 2024-03-16T19:30:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f102d39a640 1 -- 172.21.15.172:0/4156477796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1028154a70 msgr2=0x7f102806c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f102d39a640 1 --2- 172.21.15.172:0/4156477796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1028154a70 0x7f102806c490 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1018009870 tx=0x7f101802b8f0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f102d39a640 1 -- 172.21.15.172:0/4156477796 shutdown_connections 2024-03-16T19:30:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f102d39a640 1 --2- 172.21.15.172:0/4156477796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1028154e50 0x7f102806b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f102d39a640 1 --2- 172.21.15.172:0/4156477796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1028154a70 0x7f102806c490 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f102d39a640 1 -- 172.21.15.172:0/4156477796 >> 172.21.15.172:0/4156477796 conn(0x7f1028067250 msgr2=0x7f10280d01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:30.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f102d39a640 1 -- 172.21.15.172:0/4156477796 shutdown_connections 2024-03-16T19:30:30.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.282+0000 7f102d39a640 1 -- 172.21.15.172:0/4156477796 wait complete. 2024-03-16T19:30:30.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.283+0000 7f102d39a640 1 Processor -- start 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f102d39a640 1 -- start start 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f102d39a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10280c5190 0x7f102815bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f102d39a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1028154a70 0x7f102814e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f102d39a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1028154e50 0x7f1028150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f102d39a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10280cb210 con 0x7f1028154a70 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f102d39a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102814c440 con 0x7f1028154e50 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f102d39a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10281efbb0 con 0x7f10280c5190 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f101ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1028154a70 0x7f102814e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f1026ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10280c5190 0x7f102815bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f1026ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10280c5190 0x7f102815bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:57264/0 (socket says 172.21.15.172:57264) 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f1026ffd640 1 -- 172.21.15.172:0/22701493 learned_addr learned my addr 172.21.15.172:0/22701493 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.284+0000 7f10277fe640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1028154e50 0x7f1028150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f1026ffd640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1028154e50 msgr2=0x7f1028150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f1026ffd640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1028154e50 0x7f1028150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f1026ffd640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1028154a70 msgr2=0x7f102814e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f1026ffd640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1028154a70 0x7f102814e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f1026ffd640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1010009400 con 0x7f10280c5190 2024-03-16T19:30:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f1026ffd640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10280c5190 0x7f102815bb00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1018000c00 tx=0x7f10180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101803a9f0 con 0x7f10280c5190 2024-03-16T19:30:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101803ab90 con 0x7f10280c5190 2024-03-16T19:30:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10180094d0 con 0x7f10280c5190 2024-03-16T19:30:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10180443f0 con 0x7f10280c5190 2024-03-16T19:30:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.285+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10281f00a0 con 0x7f10280c5190 2024-03-16T19:30:30.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.286+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f10280681c0 con 0x7f10280c5190 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.286+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f101802f050 con 0x7f10280c5190 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.287+0000 7f101f7fe640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f100806b3b0 0x7f100806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.287+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f10180abb30 con 0x7f10280c5190 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.287+0000 7f101ffff640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f100806b3b0 0x7f100806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.287+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v45) v1 ==== 54+0+16 (secure 0 0 0) 0x7f10180b0240 con 0x7f10280c5190 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.287+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f102815c3c0 0x7f1028152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.287+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f102815a940 con 0x7f102815c3c0 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.287+0000 7f101ffff640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f100806b3b0 0x7f100806d870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10280cc580 tx=0x7f1010009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.287+0000 7f10277fe640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f102815c3c0 0x7f1028152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.288+0000 7f10277fe640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f102815c3c0 0x7f1028152cc0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.288+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1020001b20 con 0x7f102815c3c0 2024-03-16T19:30:30.277 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:30.296+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:30.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.303+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f10280ce1e0 con 0x7f102815c3c0 2024-03-16T19:30:30.286 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.305+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f10280ce1e0 con 0x7f102815c3c0 2024-03-16T19:30:30.286 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:30.287 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:30.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.305+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f10281f0890 0x7f10280cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.305+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f102815cb80 con 0x7f10281f0890 2024-03-16T19:30:30.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.305+0000 7f1026ffd640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f10281f0890 0x7f10280cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.305+0000 7f1026ffd640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f10281f0890 0x7f10280cb630 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.306+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f102815cb80 con 0x7f10281f0890 2024-03-16T19:30:30.302 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.321+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f10280cbdc0 con 0x7f10281f0890 2024-03-16T19:30:30.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.322+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f10280cbdc0 con 0x7f10281f0890 2024-03-16T19:30:30.304 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:30.305 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:30.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.323+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f10281f39f0 0x7f1028061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.323+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f102806c9d0 con 0x7f10281f39f0 2024-03-16T19:30:30.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.323+0000 7f101ffff640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f10281f39f0 0x7f1028061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.323+0000 7f101ffff640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f10281f39f0 0x7f1028061930 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.324+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1010011ed0 con 0x7f10281f39f0 2024-03-16T19:30:30.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.338+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f102815cd80 con 0x7f10281f39f0 2024-03-16T19:30:30.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.340+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f102815cd80 con 0x7f10281f39f0 2024-03-16T19:30:30.321 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:30.322 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:30.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.340+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f10280c5680 0x7f10280c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.340+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10280cbf90 con 0x7f10280c5680 2024-03-16T19:30:30.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.340+0000 7f10277fe640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f10280c5680 0x7f10280c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.340+0000 7f10277fe640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f10280c5680 0x7f10280c5a80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.341+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f10280cbf90 con 0x7f10280c5680 2024-03-16T19:30:30.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.358+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f10280c5fc0 con 0x7f10280c5680 2024-03-16T19:30:30.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.359+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f10280c5fc0 con 0x7f10280c5680 2024-03-16T19:30:30.341 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:30.341 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:30.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.359+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f102802bde0 0x7f102802c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.359+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10280c6640 con 0x7f102802bde0 2024-03-16T19:30:30.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.360+0000 7f1026ffd640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f102802bde0 0x7f102802c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.360+0000 7f1026ffd640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f102802bde0 0x7f102802c200 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.361+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f10280c6640 con 0x7f102802bde0 2024-03-16T19:30:30.351 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:30.352 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:30.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.375+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f10280c6640 con 0x7f102802bde0 2024-03-16T19:30:30.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.377+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f10280c6640 con 0x7f102802bde0 2024-03-16T19:30:30.358 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:30.360 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:30.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.377+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f102802cc50 0x7f10282064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.377+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f102802d150 con 0x7f102802cc50 2024-03-16T19:30:30.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.377+0000 7f101ffff640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f102802cc50 0x7f10282064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.378+0000 7f101ffff640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f102802cc50 0x7f10282064b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.378+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f102802d150 con 0x7f102802cc50 2024-03-16T19:30:30.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:30.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.392+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f102802d0d0 con 0x7f102802cc50 2024-03-16T19:30:30.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.394+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f102802d0d0 con 0x7f102802cc50 2024-03-16T19:30:30.376 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:30.377 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:30.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.394+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1028206f00 0x7f10282092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.394+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f102820c1d0 con 0x7f1028206f00 2024-03-16T19:30:30.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.395+0000 7f10277fe640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1028206f00 0x7f10282092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.395+0000 7f10277fe640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1028206f00 0x7f10282092f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.396+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f102820c1d0 con 0x7f1028206f00 2024-03-16T19:30:30.378 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:30.397+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:30.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.410+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f102820c1d0 con 0x7f1028206f00 2024-03-16T19:30:30.393 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.411+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f102820c1d0 con 0x7f1028206f00 2024-03-16T19:30:30.393 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:30.393 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:30.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.412+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f102820b530 0x7f102820b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.412+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f102820f5e0 con 0x7f102820b530 2024-03-16T19:30:30.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.412+0000 7f1026ffd640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f102820b530 0x7f102820b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.412+0000 7f1026ffd640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f102820b530 0x7f102820b970 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.413+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f102820f5e0 con 0x7f102820b530 2024-03-16T19:30:30.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.427+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f102820e460 con 0x7f102820b530 2024-03-16T19:30:30.410 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f101f7fe640 1 -- 172.21.15.172:0/22701493 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f102820e460 con 0x7f102820b530 2024-03-16T19:30:30.411 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:30.414 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:30.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f102815c3c0 msgr2=0x7f1028152cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f102815c3c0 0x7f1028152cc0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f10281f0890 msgr2=0x7f10280cb630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f10281f0890 0x7f10280cb630 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f10281f39f0 msgr2=0x7f1028061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f10281f39f0 0x7f1028061930 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f10280c5680 msgr2=0x7f10280c5a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f10280c5680 0x7f10280c5a80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f102802bde0 msgr2=0x7f102802c200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f10277fe640 1 -- 172.21.15.172:0/22701493 reap_dead start 2024-03-16T19:30:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f102802bde0 0x7f102802c200 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f10277fe640 1 -- 172.21.15.172:0/22701493 reap_dead start 2024-03-16T19:30:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f102802cc50 msgr2=0x7f10282064b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f102802cc50 0x7f10282064b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1028206f00 msgr2=0x7f10282092f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1028206f00 0x7f10282092f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f102820b530 msgr2=0x7f102820b970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f102820b530 0x7f102820b970 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f100806b3b0 msgr2=0x7f100806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f100806b3b0 0x7f100806d870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10280cc580 tx=0x7f1010009190 comp rx=0 tx=0).stop 2024-03-16T19:30:30.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10280c5190 msgr2=0x7f102815bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f10277fe640 1 -- 172.21.15.172:0/22701493 reap_dead start 2024-03-16T19:30:30.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.429+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10280c5190 0x7f102815bb00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1018000c00 tx=0x7f10180046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.430+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 shutdown_connections 2024-03-16T19:30:30.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.430+0000 7f102d39a640 1 --2- 172.21.15.172:0/22701493 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10280c5190 0x7f102815bb00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.430+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 >> 172.21.15.172:0/22701493 conn(0x7f1028067250 msgr2=0x7f1028151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:30.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.430+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 shutdown_connections 2024-03-16T19:30:30.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.430+0000 7f102d39a640 1 -- 172.21.15.172:0/22701493 wait complete. 2024-03-16T19:30:30.479 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:30.498+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:30.482 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:30.482 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:30.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:30.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.574+0000 7f813d7fa640 1 -- 172.21.15.172:0/3715997725 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v46) v1 ==== 113+0+0 (secure 0 0 0) 0x7f814014f0c0 con 0x7f8140154a90 2024-03-16T19:30:30.555 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.7 to 1 (10000) 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f812006b3e0 msgr2=0x7f812006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f812006b3e0 0x7f812006d8a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f813800f600 tx=0x7f813800b040 comp rx=0 tx=0).stop 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8140154a90 msgr2=0x7f814015cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8140154a90 0x7f814015cb00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8128033000 tx=0x7f81280046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 shutdown_connections 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f812006b3e0 0x7f812006d8a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8140154e70 0x7f8140145d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8140154a90 0x7f814015cb00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 --2- 172.21.15.172:0/3715997725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81400cc280 0x7f81401f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 >> 172.21.15.172:0/3715997725 conn(0x7f8140067250 msgr2=0x7f81400c50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 shutdown_connections 2024-03-16T19:30:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.576+0000 7f8146f3a640 1 -- 172.21.15.172:0/3715997725 wait complete. 2024-03-16T19:30:30.562 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:30:30.571 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-16T19:30:30.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:30:30.580 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:30.599+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:30.603 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:30.603 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:30.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:30.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.686+0000 7fa1c1f4b640 1 Processor -- start 2024-03-16T19:30:30.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.686+0000 7fa1c1f4b640 1 -- start start 2024-03-16T19:30:30.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.687+0000 7fa1c1f4b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c93a0 0x7fa1bc06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.687+0000 7fa1c1f4b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc072610 con 0x7fa1bc0c79d0 2024-03-16T19:30:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.687+0000 7fa1c1f4b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc0729c0 con 0x7fa1bc152a20 2024-03-16T19:30:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.687+0000 7fa1c1f4b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc0c9890 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.687+0000 7fa1bbfff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa1bc0c79d0 0x7fa1bc06b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39288/0 (socket says 172.21.15.172:39288) 2024-03-16T19:30:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.687+0000 7fa1bbfff640 1 -- 172.21.15.172:0/1350544961 learned_addr learned my addr 172.21.15.172:0/1350544961 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.687+0000 7fa1baffd640 1 --2- 172.21.15.172:0/1350544961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c93a0 0x7fa1bc06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.688+0000 7fa1ba7fc640 1 -- 172.21.15.172:0/1350544961 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271355723 0 0) 0x7fa1bc072610 con 0x7fa1bc0c79d0 2024-03-16T19:30:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.688+0000 7fa1ba7fc640 1 -- 172.21.15.172:0/1350544961 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a4003520 con 0x7fa1bc0c79d0 2024-03-16T19:30:30.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.688+0000 7fa1ba7fc640 1 -- 172.21.15.172:0/1350544961 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802236832 0 0) 0x7fa1bc0729c0 con 0x7fa1bc152a20 2024-03-16T19:30:30.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.688+0000 7fa1ba7fc640 1 -- 172.21.15.172:0/1350544961 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1bc072610 con 0x7fa1bc152a20 2024-03-16T19:30:30.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.688+0000 7fa1baffd640 1 -- 172.21.15.172:0/1350544961 >> v1:172.21.15.174:6789/0 conn(0x7fa1bc152a20 legacy=0x7fa1bc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.688+0000 7fa1baffd640 1 -- 172.21.15.172:0/1350544961 >> v1:172.21.15.172:6789/0 conn(0x7fa1bc0c79d0 legacy=0x7fa1bc06b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.688+0000 7fa1baffd640 1 -- 172.21.15.172:0/1350544961 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc072ca0 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.689+0000 7fa1baffd640 1 --2- 172.21.15.172:0/1350544961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c93a0 0x7fa1bc06c730 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00097d0 tx=0x7fa1b002b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9fd6096ebaf0336 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.689+0000 7fa1ba7fc640 1 -- 172.21.15.172:0/1350544961 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0030890 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.689+0000 7fa1ba7fc640 1 -- 172.21.15.172:0/1350544961 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b00309f0 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.689+0000 7fa1ba7fc640 1 -- 172.21.15.172:0/1350544961 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0030d80 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.689+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/1350544961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c93a0 msgr2=0x7fa1bc06c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.689+0000 7fa1c1f4b640 1 --2- 172.21.15.172:0/1350544961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c93a0 0x7fa1bc06c730 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00097d0 tx=0x7fa1b002b9d0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.690+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/1350544961 shutdown_connections 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.690+0000 7fa1c1f4b640 1 --2- 172.21.15.172:0/1350544961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c93a0 0x7fa1bc06c730 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.690+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/1350544961 >> 172.21.15.172:0/1350544961 conn(0x7fa1bc06db70 msgr2=0x7fa1bc06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:30.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.690+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/1350544961 shutdown_connections 2024-03-16T19:30:30.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.690+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/1350544961 wait complete. 2024-03-16T19:30:30.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.691+0000 7fa1c1f4b640 1 Processor -- start 2024-03-16T19:30:30.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.691+0000 7fa1c1f4b640 1 -- start start 2024-03-16T19:30:30.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1c1f4b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c79d0 0x7fa1bc1482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1c1f4b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1bc0c93a0 0x7fa1bc1e51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1c1f4b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1bc152a20 0x7fa1bc1ee120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1c1f4b640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc1489d0 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1c1f4b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc14a150 con 0x7fa1bc152a20 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1c1f4b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc14a710 con 0x7fa1bc0c79d0 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1baffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1bc0c93a0 0x7fa1bc1e51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1bbfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1bc152a20 0x7fa1bc1ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1baffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1bc0c93a0 0x7fa1bc1e51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47560/0 (socket says 172.21.15.172:47560) 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1bb7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c79d0 0x7fa1bc1482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1baffd640 1 -- 172.21.15.172:0/2327067056 learned_addr learned my addr 172.21.15.172:0/2327067056 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1baffd640 1 -- 172.21.15.172:0/2327067056 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c79d0 msgr2=0x7fa1bc1482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1baffd640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c79d0 0x7fa1bc1482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1baffd640 1 -- 172.21.15.172:0/2327067056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1bc152a20 msgr2=0x7fa1bc1ee120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1baffd640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1bc152a20 0x7fa1bc1ee120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1baffd640 1 -- 172.21.15.172:0/2327067056 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b0009400 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1bb7fe640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c79d0 0x7fa1bc1482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1baffd640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1bc0c93a0 0x7fa1bc1e51a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa1b002bfe0 tx=0x7fa1b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.692+0000 7fa1a3fff640 1 -- 172.21.15.172:0/2327067056 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b003e040 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.693+0000 7fa1a3fff640 1 -- 172.21.15.172:0/2327067056 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b0039c10 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.693+0000 7fa1a3fff640 1 -- 172.21.15.172:0/2327067056 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0043440 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.693+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1bc152f80 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.693+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc149d90 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.694+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1bc075a70 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.695+0000 7fa1a3fff640 1 -- 172.21.15.172:0/2327067056 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa1b00423e0 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.695+0000 7fa1a3fff640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1a806b3b0 0x7fa1a806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.695+0000 7fa1a3fff640 1 -- 172.21.15.172:0/2327067056 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fa1b00ac1b0 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.695+0000 7fa1bb7fe640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1a806b3b0 0x7fa1a806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.696+0000 7fa1bb7fe640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1a806b3b0 0x7fa1a806d870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac007fd0 tx=0x7fa1ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.681 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:30.700+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:30.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.700+0000 7fa1a3fff640 1 -- 172.21.15.172:0/2327067056 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1b0076e40 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.727+0000 7fb89059a640 1 Processor -- start 2024-03-16T19:30:30.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.727+0000 7fb89059a640 1 -- start start 2024-03-16T19:30:30.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.728+0000 7fb89059a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8880cb560 0x7fb8881661b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.728+0000 7fb89059a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8880cc720 con 0x7fb88806bb10 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.728+0000 7fb89059a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8880cc9c0 con 0x7fb88806bef0 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.728+0000 7fb89059a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88815de30 con 0x7fb8880cb560 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.728+0000 7fb88e336640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb88806bb10 0x7fb8880cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39304/0 (socket says 172.21.15.172:39304) 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.728+0000 7fb88e336640 1 -- 172.21.15.172:0/378034852 learned_addr learned my addr 172.21.15.172:0/378034852 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.728+0000 7fb88eb37640 1 --2- 172.21.15.172:0/378034852 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8880cb560 0x7fb8881661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.729+0000 7fb88d334640 1 -- 172.21.15.172:0/378034852 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058217329 0 0) 0x7fb8880cc720 con 0x7fb88806bb10 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.729+0000 7fb88d334640 1 -- 172.21.15.172:0/378034852 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb874003520 con 0x7fb88806bb10 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.729+0000 7fb88eb37640 1 -- 172.21.15.172:0/378034852 >> v1:172.21.15.174:6789/0 conn(0x7fb88806bef0 legacy=0x7fb888154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.729+0000 7fb88eb37640 1 -- 172.21.15.172:0/378034852 >> v1:172.21.15.172:6789/0 conn(0x7fb88806bb10 legacy=0x7fb8880cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.729+0000 7fb88eb37640 1 -- 172.21.15.172:0/378034852 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88815e110 con 0x7fb8880cb560 2024-03-16T19:30:30.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.730+0000 7fb88eb37640 1 --2- 172.21.15.172:0/378034852 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8880cb560 0x7fb8881661b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb880009dc0 tx=0x7fb88002bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b8bc7dcddaf30f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.730+0000 7fb88d334640 1 -- 172.21.15.172:0/378034852 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88002cd60 con 0x7fb8880cb560 2024-03-16T19:30:30.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.730+0000 7fb88d334640 1 -- 172.21.15.172:0/378034852 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb880034440 con 0x7fb8880cb560 2024-03-16T19:30:30.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.730+0000 7fb88d334640 1 -- 172.21.15.172:0/378034852 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8800347d0 con 0x7fb8880cb560 2024-03-16T19:30:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.731+0000 7fb89059a640 1 -- 172.21.15.172:0/378034852 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8880cb560 msgr2=0x7fb8881661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.731+0000 7fb89059a640 1 --2- 172.21.15.172:0/378034852 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8880cb560 0x7fb8881661b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb880009dc0 tx=0x7fb88002bfb0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.731+0000 7fb89059a640 1 -- 172.21.15.172:0/378034852 shutdown_connections 2024-03-16T19:30:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.731+0000 7fb89059a640 1 --2- 172.21.15.172:0/378034852 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8880cb560 0x7fb8881661b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.731+0000 7fb89059a640 1 -- 172.21.15.172:0/378034852 >> 172.21.15.172:0/378034852 conn(0x7fb888067250 msgr2=0x7fb8880d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.731+0000 7fb89059a640 1 -- 172.21.15.172:0/378034852 shutdown_connections 2024-03-16T19:30:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.731+0000 7fb89059a640 1 -- 172.21.15.172:0/378034852 wait complete. 2024-03-16T19:30:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.732+0000 7fb89059a640 1 Processor -- start 2024-03-16T19:30:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.732+0000 7fb89059a640 1 -- start start 2024-03-16T19:30:30.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb89059a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88806bb10 0x7fb888147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb89059a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88806bef0 0x7fb888141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb89059a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8880cb560 0x7fb888141fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb89059a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880b4e40 con 0x7fb8880cb560 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb89059a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb888150380 con 0x7fb88806bb10 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb89059a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88814f840 con 0x7fb88806bef0 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88eb37640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8880cb560 0x7fb888141fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88db35640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88806bef0 0x7fb888141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88e336640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88806bb10 0x7fb888147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88db35640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88806bef0 0x7fb888141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:57300/0 (socket says 172.21.15.172:57300) 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88eb37640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8880cb560 0x7fb888141fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47564/0 (socket says 172.21.15.172:47564) 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88db35640 1 -- 172.21.15.172:0/2372193154 learned_addr learned my addr 172.21.15.172:0/2372193154 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88e336640 1 -- 172.21.15.172:0/2372193154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88806bef0 msgr2=0x7fb888141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88e336640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88806bef0 0x7fb888141aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88e336640 1 -- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8880cb560 msgr2=0x7fb888141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88e336640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8880cb560 0x7fb888141fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88e336640 1 -- 172.21.15.172:0/2372193154 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb880009a70 con 0x7fb88806bb10 2024-03-16T19:30:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88eb37640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8880cb560 0x7fb888141fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:30.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.733+0000 7fb88e336640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88806bb10 0x7fb888147290 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb88400c3c0 tx=0x7fb884001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.734+0000 7fb87effd640 1 -- 172.21.15.172:0/2372193154 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb884005ce0 con 0x7fb88806bb10 2024-03-16T19:30:30.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.734+0000 7fb87effd640 1 -- 172.21.15.172:0/2372193154 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb884005e40 con 0x7fb88806bb10 2024-03-16T19:30:30.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.734+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88814fae0 con 0x7fb88806bb10 2024-03-16T19:30:30.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.734+0000 7fb87effd640 1 -- 172.21.15.172:0/2372193154 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8840072d0 con 0x7fb88806bb10 2024-03-16T19:30:30.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.734+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8881500a0 con 0x7fb88806bb10 2024-03-16T19:30:30.718 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.735+0000 7fb87effd640 1 -- 172.21.15.172:0/2372193154 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb884007430 con 0x7fb88806bb10 2024-03-16T19:30:30.718 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.735+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb888147f50 con 0x7fb88806bb10 2024-03-16T19:30:30.718 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.735+0000 7fb87effd640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3e0 0x7fb86c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:30.718 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.735+0000 7fb87effd640 1 -- 172.21.15.172:0/2372193154 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fb884089180 con 0x7fb88806bb10 2024-03-16T19:30:30.718 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.735+0000 7fb88db35640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3e0 0x7fb86c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:30.718 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.736+0000 7fb88db35640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3e0 0x7fb86c06d8a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb8880c6b50 tx=0x7fb878009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:30.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.744+0000 7fb87effd640 1 -- 172.21.15.172:0/2372193154 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb888147f50 con 0x7fb88806bb10 2024-03-16T19:30:30.767 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:30.767 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:30.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:30.781 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:30.800+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:30.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.891+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fa1bc14a850 con 0x7fa1bc0c93a0 2024-03-16T19:30:30.882 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:30.901+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:30.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.920+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb88815e6d0 con 0x7fb86c06b3e0 2024-03-16T19:30:30.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.933+0000 7fb87effd640 1 -- 172.21.15.172:0/2372193154 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251867 (secure 0 0 0) 0x7fb87800f820 con 0x7fb86c06b3e0 2024-03-16T19:30:30.917 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:30:30.918 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3e0 msgr2=0x7fb86c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3e0 0x7fb86c06d8a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb8880c6b50 tx=0x7fb878009040 comp rx=0 tx=0).stop 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88806bb10 msgr2=0x7fb888147290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88806bb10 0x7fb888147290 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb88400c3c0 tx=0x7fb884001e70 comp rx=0 tx=0).stop 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 shutdown_connections 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb8880cb560 0x7fb888141fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3e0 0x7fb86c06d8a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88806bef0 0x7fb888141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 --2- 172.21.15.172:0/2372193154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88806bb10 0x7fb888147290 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 >> 172.21.15.172:0/2372193154 conn(0x7fb888067250 msgr2=0x7fb888152b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 shutdown_connections 2024-03-16T19:30:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:30.936+0000 7fb89059a640 1 -- 172.21.15.172:0/2372193154 wait complete. 2024-03-16T19:30:30.928 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-03-16T19:30:30.577908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597828,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":216,"num_write_kb":2441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164,"ondisk_log_size":164,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221188,"kb_used_data":4992,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753532,"statfs":{"total":773094113280,"available":772867616768,"internally_reserved":0,"allocated":5111808,"data_stored":2834128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.024646"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938344+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:30:27.938344+0000","last_peered":"2024-03-16T19:30:27.938344+0000","last_clean":"2024-03-16T19:30:27.938344+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:30:27.938344+0000","last_undegraded":"2024-03-16T19:30:27.938344+0000","last_fullsized":"2024-03-16T19:30:27.938344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664523+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:30:27.664523+0000","last_peered":"2024-03-16T19:30:27.664523+0000","last_clean":"2024-03-16T19:30:27.664523+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:30:27.664523+0000","last_undegraded":"2024-03-16T19:30:27.664523+0000","last_fullsized":"2024-03-16T19:30:27.664523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":49,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564569+0000","last_change":"2024-03-16T19:29:44.412537+0000","last_active":"2024-03-16T19:30:27.564569+0000","last_peered":"2024-03-16T19:30:27.564569+0000","last_clean":"2024-03-16T19:30:27.564569+0000","last_became_active":"2024-03-16T19:29:44.412236+0000","last_became_peered":"2024-03-16T19:29:44.412236+0000","last_unstale":"2024-03-16T19:30:27.564569+0000","last_undegraded":"2024-03-16T19:30:27.564569+0000","last_fullsized":"2024-03-16T19:30:27.564569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:50:57.180910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562949+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:30:27.562949+0000","last_peered":"2024-03-16T19:30:27.562949+0000","last_clean":"2024-03-16T19:30:27.562949+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:30:27.562949+0000","last_undegraded":"2024-03-16T19:30:27.562949+0000","last_fullsized":"2024-03-16T19:30:27.562949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938310+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:30:27.938310+0000","last_peered":"2024-03-16T19:30:27.938310+0000","last_clean":"2024-03-16T19:30:27.938310+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:30:27.938310+0000","last_undegraded":"2024-03-16T19:30:27.938310+0000","last_fullsized":"2024-03-16T19:30:27.938310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562831+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:30:27.562831+0000","last_peered":"2024-03-16T19:30:27.562831+0000","last_clean":"2024-03-16T19:30:27.562831+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:30:27.562831+0000","last_undegraded":"2024-03-16T19:30:27.562831+0000","last_fullsized":"2024-03-16T19:30:27.562831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937835+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:30:27.937835+0000","last_peered":"2024-03-16T19:30:27.937835+0000","last_clean":"2024-03-16T19:30:27.937835+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:30:27.937835+0000","last_undegraded":"2024-03-16T19:30:27.937835+0000","last_fullsized":"2024-03-16T19:30:27.937835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938054+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:30:27.938054+0000","last_peered":"2024-03-16T19:30:27.938054+0000","last_clean":"2024-03-16T19:30:27.938054+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:30:27.938054+0000","last_undegraded":"2024-03-16T19:30:27.938054+0000","last_fullsized":"2024-03-16T19:30:27.938054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938441+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:30:27.938441+0000","last_peered":"2024-03-16T19:30:27.938441+0000","last_clean":"2024-03-16T19:30:27.938441+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:30:27.938441+0000","last_undegraded":"2024-03-16T19:30:27.938441+0000","last_fullsized":"2024-03-16T19:30:27.938441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938573+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:30:27.938573+0000","last_peered":"2024-03-16T19:30:27.938573+0000","last_clean":"2024-03-16T19:30:27.938573+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:30:27.938573+0000","last_undegraded":"2024-03-16T19:30:27.938573+0000","last_fullsized":"2024-03-16T19:30:27.938573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938366+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:30:27.938366+0000","last_peered":"2024-03-16T19:30:27.938366+0000","last_clean":"2024-03-16T19:30:27.938366+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:30:27.938366+0000","last_undegraded":"2024-03-16T19:30:27.938366+0000","last_fullsized":"2024-03-16T19:30:27.938366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563010+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:30:27.563010+0000","last_peered":"2024-03-16T19:30:27.563010+0000","last_clean":"2024-03-16T19:30:27.563010+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:30:27.563010+0000","last_undegraded":"2024-03-16T19:30:27.563010+0000","last_fullsized":"2024-03-16T19:30:27.563010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938617+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:30:27.938617+0000","last_peered":"2024-03-16T19:30:27.938617+0000","last_clean":"2024-03-16T19:30:27.938617+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:30:27.938617+0000","last_undegraded":"2024-03-16T19:30:27.938617+0000","last_fullsized":"2024-03-16T19:30:27.938617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":50,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.665283+0000","last_change":"2024-03-16T19:29:44.413594+0000","last_active":"2024-03-16T19:30:27.665283+0000","last_peered":"2024-03-16T19:30:27.665283+0000","last_clean":"2024-03-16T19:30:27.665283+0000","last_became_active":"2024-03-16T19:29:44.413235+0000","last_became_peered":"2024-03-16T19:29:44.413235+0000","last_unstale":"2024-03-16T19:30:27.665283+0000","last_undegraded":"2024-03-16T19:30:27.665283+0000","last_fullsized":"2024-03-16T19:30:27.665283+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:35.778808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664388+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:30:27.664388+0000","last_peered":"2024-03-16T19:30:27.664388+0000","last_clean":"2024-03-16T19:30:27.664388+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:30:27.664388+0000","last_undegraded":"2024-03-16T19:30:27.664388+0000","last_fullsized":"2024-03-16T19:30:27.664388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562300+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:30:27.562300+0000","last_peered":"2024-03-16T19:30:27.562300+0000","last_clean":"2024-03-16T19:30:27.562300+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:30:27.562300+0000","last_undegraded":"2024-03-16T19:30:27.562300+0000","last_fullsized":"2024-03-16T19:30:27.562300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562830+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:30:27.562830+0000","last_peered":"2024-03-16T19:30:27.562830+0000","last_clean":"2024-03-16T19:30:27.562830+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:30:27.562830+0000","last_undegraded":"2024-03-16T19:30:27.562830+0000","last_fullsized":"2024-03-16T19:30:27.562830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564600+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:30:27.564600+0000","last_peered":"2024-03-16T19:30:27.564600+0000","last_clean":"2024-03-16T19:30:27.564600+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:30:27.564600+0000","last_undegraded":"2024-03-16T19:30:27.564600+0000","last_fullsized":"2024-03-16T19:30:27.564600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938163+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:30:27.938163+0000","last_peered":"2024-03-16T19:30:27.938163+0000","last_clean":"2024-03-16T19:30:27.938163+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:30:27.938163+0000","last_undegraded":"2024-03-16T19:30:27.938163+0000","last_fullsized":"2024-03-16T19:30:27.938163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938295+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:30:27.938295+0000","last_peered":"2024-03-16T19:30:27.938295+0000","last_clean":"2024-03-16T19:30:27.938295+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:30:27.938295+0000","last_undegraded":"2024-03-16T19:30:27.938295+0000","last_fullsized":"2024-03-16T19:30:27.938295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938230+0000","last_change":"2024-03-16T19:29:44.414141+0000","last_active":"2024-03-16T19:30:27.938230+0000","last_peered":"2024-03-16T19:30:27.938230+0000","last_clean":"2024-03-16T19:30:27.938230+0000","last_became_active":"2024-03-16T19:29:44.413873+0000","last_became_peered":"2024-03-16T19:29:44.413873+0000","last_unstale":"2024-03-16T19:30:27.938230+0000","last_undegraded":"2024-03-16T19:30:27.938230+0000","last_fullsized":"2024-03-16T19:30:27.938230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:03:03.291206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564391+0000","last_change":"2024-03-16T19:30:07.591770+0000","last_active":"2024-03-16T19:30:27.564391+0000","last_peered":"2024-03-16T19:30:27.564391+0000","last_clean":"2024-03-16T19:30:27.564391+0000","last_became_active":"2024-03-16T19:30:07.591482+0000","last_became_peered":"2024-03-16T19:30:07.591482+0000","last_unstale":"2024-03-16T19:30:27.564391+0000","last_undegraded":"2024-03-16T19:30:27.564391+0000","last_fullsized":"2024-03-16T19:30:27.564391+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:02.003669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938106+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:30:27.938106+0000","last_peered":"2024-03-16T19:30:27.938106+0000","last_clean":"2024-03-16T19:30:27.938106+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:30:27.938106+0000","last_undegraded":"2024-03-16T19:30:27.938106+0000","last_fullsized":"2024-03-16T19:30:27.938106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562360+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:30:27.562360+0000","last_peered":"2024-03-16T19:30:27.562360+0000","last_clean":"2024-03-16T19:30:27.562360+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:30:27.562360+0000","last_undegraded":"2024-03-16T19:30:27.562360+0000","last_fullsized":"2024-03-16T19:30:27.562360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938946+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:30:27.938946+0000","last_peered":"2024-03-16T19:30:27.938946+0000","last_clean":"2024-03-16T19:30:27.938946+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:30:27.938946+0000","last_undegraded":"2024-03-16T19:30:27.938946+0000","last_fullsized":"2024-03-16T19:30:27.938946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938614+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:30:27.938614+0000","last_peered":"2024-03-16T19:30:27.938614+0000","last_clean":"2024-03-16T19:30:27.938614+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:30:27.938614+0000","last_undegraded":"2024-03-16T19:30:27.938614+0000","last_fullsized":"2024-03-16T19:30:27.938614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664932+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:30:27.664932+0000","last_peered":"2024-03-16T19:30:27.664932+0000","last_clean":"2024-03-16T19:30:27.664932+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:30:27.664932+0000","last_undegraded":"2024-03-16T19:30:27.664932+0000","last_fullsized":"2024-03-16T19:30:27.664932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938755+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:30:27.938755+0000","last_peered":"2024-03-16T19:30:27.938755+0000","last_clean":"2024-03-16T19:30:27.938755+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:30:27.938755+0000","last_undegraded":"2024-03-16T19:30:27.938755+0000","last_fullsized":"2024-03-16T19:30:27.938755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":50,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938527+0000","last_change":"2024-03-16T19:29:44.412475+0000","last_active":"2024-03-16T19:30:27.938527+0000","last_peered":"2024-03-16T19:30:27.938527+0000","last_clean":"2024-03-16T19:30:27.938527+0000","last_became_active":"2024-03-16T19:29:44.412176+0000","last_became_peered":"2024-03-16T19:29:44.412176+0000","last_unstale":"2024-03-16T19:30:27.938527+0000","last_undegraded":"2024-03-16T19:30:27.938527+0000","last_fullsized":"2024-03-16T19:30:27.938527+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:29:34.284744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562925+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:30:27.562925+0000","last_peered":"2024-03-16T19:30:27.562925+0000","last_clean":"2024-03-16T19:30:27.562925+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:30:27.562925+0000","last_undegraded":"2024-03-16T19:30:27.562925+0000","last_fullsized":"2024-03-16T19:30:27.562925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564355+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:30:27.564355+0000","last_peered":"2024-03-16T19:30:27.564355+0000","last_clean":"2024-03-16T19:30:27.564355+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:30:27.564355+0000","last_undegraded":"2024-03-16T19:30:27.564355+0000","last_fullsized":"2024-03-16T19:30:27.564355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937963+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:30:27.937963+0000","last_peered":"2024-03-16T19:30:27.937963+0000","last_clean":"2024-03-16T19:30:27.937963+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:30:27.937963+0000","last_undegraded":"2024-03-16T19:30:27.937963+0000","last_fullsized":"2024-03-16T19:30:27.937963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562211+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:30:27.562211+0000","last_peered":"2024-03-16T19:30:27.562211+0000","last_clean":"2024-03-16T19:30:27.562211+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:30:27.562211+0000","last_undegraded":"2024-03-16T19:30:27.562211+0000","last_fullsized":"2024-03-16T19:30:27.562211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938779+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:30:27.938779+0000","last_peered":"2024-03-16T19:30:27.938779+0000","last_clean":"2024-03-16T19:30:27.938779+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:30:27.938779+0000","last_undegraded":"2024-03-16T19:30:27.938779+0000","last_fullsized":"2024-03-16T19:30:27.938779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562654+0000","last_change":"2024-03-16T19:29:44.412541+0000","last_active":"2024-03-16T19:30:27.562654+0000","last_peered":"2024-03-16T19:30:27.562654+0000","last_clean":"2024-03-16T19:30:27.562654+0000","last_became_active":"2024-03-16T19:29:44.412194+0000","last_became_peered":"2024-03-16T19:29:44.412194+0000","last_unstale":"2024-03-16T19:30:27.562654+0000","last_undegraded":"2024-03-16T19:30:27.562654+0000","last_fullsized":"2024-03-16T19:30:27.562654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:51:50.130592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938168+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:30:27.938168+0000","last_peered":"2024-03-16T19:30:27.938168+0000","last_clean":"2024-03-16T19:30:27.938168+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:30:27.938168+0000","last_undegraded":"2024-03-16T19:30:27.938168+0000","last_fullsized":"2024-03-16T19:30:27.938168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938200+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:30:27.938200+0000","last_peered":"2024-03-16T19:30:27.938200+0000","last_clean":"2024-03-16T19:30:27.938200+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:30:27.938200+0000","last_undegraded":"2024-03-16T19:30:27.938200+0000","last_fullsized":"2024-03-16T19:30:27.938200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938116+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:30:27.938116+0000","last_peered":"2024-03-16T19:30:27.938116+0000","last_clean":"2024-03-16T19:30:27.938116+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:30:27.938116+0000","last_undegraded":"2024-03-16T19:30:27.938116+0000","last_fullsized":"2024-03-16T19:30:27.938116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562399+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:30:27.562399+0000","last_peered":"2024-03-16T19:30:27.562399+0000","last_clean":"2024-03-16T19:30:27.562399+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:30:27.562399+0000","last_undegraded":"2024-03-16T19:30:27.562399+0000","last_fullsized":"2024-03-16T19:30:27.562399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664292+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:30:27.664292+0000","last_peered":"2024-03-16T19:30:27.664292+0000","last_clean":"2024-03-16T19:30:27.664292+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:30:27.664292+0000","last_undegraded":"2024-03-16T19:30:27.664292+0000","last_fullsized":"2024-03-16T19:30:27.664292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938657+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:30:27.938657+0000","last_peered":"2024-03-16T19:30:27.938657+0000","last_clean":"2024-03-16T19:30:27.938657+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:30:27.938657+0000","last_undegraded":"2024-03-16T19:30:27.938657+0000","last_fullsized":"2024-03-16T19:30:27.938657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562691+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:30:27.562691+0000","last_peered":"2024-03-16T19:30:27.562691+0000","last_clean":"2024-03-16T19:30:27.562691+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:30:27.562691+0000","last_undegraded":"2024-03-16T19:30:27.562691+0000","last_fullsized":"2024-03-16T19:30:27.562691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562776+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:30:27.562776+0000","last_peered":"2024-03-16T19:30:27.562776+0000","last_clean":"2024-03-16T19:30:27.562776+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:30:27.562776+0000","last_undegraded":"2024-03-16T19:30:27.562776+0000","last_fullsized":"2024-03-16T19:30:27.562776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562734+0000","last_change":"2024-03-16T19:29:44.413558+0000","last_active":"2024-03-16T19:30:27.562734+0000","last_peered":"2024-03-16T19:30:27.562734+0000","last_clean":"2024-03-16T19:30:27.562734+0000","last_became_active":"2024-03-16T19:29:44.413310+0000","last_became_peered":"2024-03-16T19:29:44.413310+0000","last_unstale":"2024-03-16T19:30:27.562734+0000","last_undegraded":"2024-03-16T19:30:27.562734+0000","last_fullsized":"2024-03-16T19:30:27.562734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:25:53.846575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":60,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.665015+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:30:27.665015+0000","last_peered":"2024-03-16T19:30:27.665015+0000","last_clean":"2024-03-16T19:30:27.665015+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:30:27.665015+0000","last_undegraded":"2024-03-16T19:30:27.665015+0000","last_fullsized":"2024-03-16T19:30:27.665015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563032+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:30:27.563032+0000","last_peered":"2024-03-16T19:30:27.563032+0000","last_clean":"2024-03-16T19:30:27.563032+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:30:27.563032+0000","last_undegraded":"2024-03-16T19:30:27.563032+0000","last_fullsized":"2024-03-16T19:30:27.563032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"36'11","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938392+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:30:27.938392+0000","last_peered":"2024-03-16T19:30:27.938392+0000","last_clean":"2024-03-16T19:30:27.938392+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:30:27.938392+0000","last_undegraded":"2024-03-16T19:30:27.938392+0000","last_fullsized":"2024-03-16T19:30:27.938392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6662,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938074+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:30:27.938074+0000","last_peered":"2024-03-16T19:30:27.938074+0000","last_clean":"2024-03-16T19:30:27.938074+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:30:27.938074+0000","last_undegraded":"2024-03-16T19:30:27.938074+0000","last_fullsized":"2024-03-16T19:30:27.938074+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":64,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562743+0000","last_change":"2024-03-16T19:29:44.412731+0000","last_active":"2024-03-16T19:30:27.562743+0000","last_peered":"2024-03-16T19:30:27.562743+0000","last_clean":"2024-03-16T19:30:27.562743+0000","last_became_active":"2024-03-16T19:29:44.412174+0000","last_became_peered":"2024-03-16T19:29:44.412174+0000","last_unstale":"2024-03-16T19:30:27.562743+0000","last_undegraded":"2024-03-16T19:30:27.562743+0000","last_fullsized":"2024-03-16T19:30:27.562743+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:57.962644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937941+0000","last_change":"2024-03-16T19:29:44.412524+0000","last_active":"2024-03-16T19:30:27.937941+0000","last_peered":"2024-03-16T19:30:27.937941+0000","last_clean":"2024-03-16T19:30:27.937941+0000","last_became_active":"2024-03-16T19:29:44.412200+0000","last_became_peered":"2024-03-16T19:29:44.412200+0000","last_unstale":"2024-03-16T19:30:27.937941+0000","last_undegraded":"2024-03-16T19:30:27.937941+0000","last_fullsized":"2024-03-16T19:30:27.937941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:52.636929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563008+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:30:27.563008+0000","last_peered":"2024-03-16T19:30:27.563008+0000","last_clean":"2024-03-16T19:30:27.563008+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:30:27.563008+0000","last_undegraded":"2024-03-16T19:30:27.563008+0000","last_fullsized":"2024-03-16T19:30:27.563008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938400+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:30:27.938400+0000","last_peered":"2024-03-16T19:30:27.938400+0000","last_clean":"2024-03-16T19:30:27.938400+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:30:27.938400+0000","last_undegraded":"2024-03-16T19:30:27.938400+0000","last_fullsized":"2024-03-16T19:30:27.938400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938049+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:30:27.938049+0000","last_peered":"2024-03-16T19:30:27.938049+0000","last_clean":"2024-03-16T19:30:27.938049+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:30:27.938049+0000","last_undegraded":"2024-03-16T19:30:27.938049+0000","last_fullsized":"2024-03-16T19:30:27.938049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938035+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:30:27.938035+0000","last_peered":"2024-03-16T19:30:27.938035+0000","last_clean":"2024-03-16T19:30:27.938035+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:30:27.938035+0000","last_undegraded":"2024-03-16T19:30:27.938035+0000","last_fullsized":"2024-03-16T19:30:27.938035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":64,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564350+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:30:27.564350+0000","last_peered":"2024-03-16T19:30:27.564350+0000","last_clean":"2024-03-16T19:30:27.564350+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:30:27.564350+0000","last_undegraded":"2024-03-16T19:30:27.564350+0000","last_fullsized":"2024-03-16T19:30:27.564350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937897+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:30:27.937897+0000","last_peered":"2024-03-16T19:30:27.937897+0000","last_clean":"2024-03-16T19:30:27.937897+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:30:27.937897+0000","last_undegraded":"2024-03-16T19:30:27.937897+0000","last_fullsized":"2024-03-16T19:30:27.937897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":59,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562545+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:30:27.562545+0000","last_peered":"2024-03-16T19:30:27.562545+0000","last_clean":"2024-03-16T19:30:27.562545+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:30:27.562545+0000","last_undegraded":"2024-03-16T19:30:27.562545+0000","last_fullsized":"2024-03-16T19:30:27.562545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938264+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:30:27.938264+0000","last_peered":"2024-03-16T19:30:27.938264+0000","last_clean":"2024-03-16T19:30:27.938264+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:30:27.938264+0000","last_undegraded":"2024-03-16T19:30:27.938264+0000","last_fullsized":"2024-03-16T19:30:27.938264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564661+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:30:27.564661+0000","last_peered":"2024-03-16T19:30:27.564661+0000","last_clean":"2024-03-16T19:30:27.564661+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:30:27.564661+0000","last_undegraded":"2024-03-16T19:30:27.564661+0000","last_fullsized":"2024-03-16T19:30:27.564661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938492+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:30:27.938492+0000","last_peered":"2024-03-16T19:30:27.938492+0000","last_clean":"2024-03-16T19:30:27.938492+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:30:27.938492+0000","last_undegraded":"2024-03-16T19:30:27.938492+0000","last_fullsized":"2024-03-16T19:30:27.938492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.665114+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:30:27.665114+0000","last_peered":"2024-03-16T19:30:27.665114+0000","last_clean":"2024-03-16T19:30:27.665114+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:30:27.665114+0000","last_undegraded":"2024-03-16T19:30:27.665114+0000","last_fullsized":"2024-03-16T19:30:27.665114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563118+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:30:27.563118+0000","last_peered":"2024-03-16T19:30:27.563118+0000","last_clean":"2024-03-16T19:30:27.563118+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:30:27.563118+0000","last_undegraded":"2024-03-16T19:30:27.563118+0000","last_fullsized":"2024-03-16T19:30:27.563118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564437+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:30:27.564437+0000","last_peered":"2024-03-16T19:30:27.564437+0000","last_clean":"2024-03-16T19:30:27.564437+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:30:27.564437+0000","last_undegraded":"2024-03-16T19:30:27.564437+0000","last_fullsized":"2024-03-16T19:30:27.564437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938018+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:30:27.938018+0000","last_peered":"2024-03-16T19:30:27.938018+0000","last_clean":"2024-03-16T19:30:27.938018+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:30:27.938018+0000","last_undegraded":"2024-03-16T19:30:27.938018+0000","last_fullsized":"2024-03-16T19:30:27.938018+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":214,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937928+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:30:27.937928+0000","last_peered":"2024-03-16T19:30:27.937928+0000","last_clean":"2024-03-16T19:30:27.937928+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:30:27.937928+0000","last_undegraded":"2024-03-16T19:30:27.937928+0000","last_fullsized":"2024-03-16T19:30:27.937928+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562537+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:30:27.562537+0000","last_peered":"2024-03-16T19:30:27.562537+0000","last_clean":"2024-03-16T19:30:27.562537+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:30:27.562537+0000","last_undegraded":"2024-03-16T19:30:27.562537+0000","last_fullsized":"2024-03-16T19:30:27.562537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938707+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:30:27.938707+0000","last_peered":"2024-03-16T19:30:27.938707+0000","last_clean":"2024-03-16T19:30:27.938707+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:30:27.938707+0000","last_undegraded":"2024-03-16T19:30:27.938707+0000","last_fullsized":"2024-03-16T19:30:27.938707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937811+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:30:27.937811+0000","last_peered":"2024-03-16T19:30:27.937811+0000","last_clean":"2024-03-16T19:30:27.937811+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:30:27.937811+0000","last_undegraded":"2024-03-16T19:30:27.937811+0000","last_fullsized":"2024-03-16T19:30:27.937811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562250+0000","last_change":"2024-03-16T19:29:44.412505+0000","last_active":"2024-03-16T19:30:27.562250+0000","last_peered":"2024-03-16T19:30:27.562250+0000","last_clean":"2024-03-16T19:30:27.562250+0000","last_became_active":"2024-03-16T19:29:44.412221+0000","last_became_peered":"2024-03-16T19:29:44.412221+0000","last_unstale":"2024-03-16T19:30:27.562250+0000","last_undegraded":"2024-03-16T19:30:27.562250+0000","last_fullsized":"2024-03-16T19:30:27.562250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:14.739638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564454+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:30:27.564454+0000","last_peered":"2024-03-16T19:30:27.564454+0000","last_clean":"2024-03-16T19:30:27.564454+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:30:27.564454+0000","last_undegraded":"2024-03-16T19:30:27.564454+0000","last_fullsized":"2024-03-16T19:30:27.564454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664372+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:30:27.664372+0000","last_peered":"2024-03-16T19:30:27.664372+0000","last_clean":"2024-03-16T19:30:27.664372+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:30:27.664372+0000","last_undegraded":"2024-03-16T19:30:27.664372+0000","last_fullsized":"2024-03-16T19:30:27.664372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563096+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:30:27.563096+0000","last_peered":"2024-03-16T19:30:27.563096+0000","last_clean":"2024-03-16T19:30:27.563096+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:30:27.563096+0000","last_undegraded":"2024-03-16T19:30:27.563096+0000","last_fullsized":"2024-03-16T19:30:27.563096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562837+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:30:27.562837+0000","last_peered":"2024-03-16T19:30:27.562837+0000","last_clean":"2024-03-16T19:30:27.562837+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:30:27.562837+0000","last_undegraded":"2024-03-16T19:30:27.562837+0000","last_fullsized":"2024-03-16T19:30:27.562837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664837+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:30:27.664837+0000","last_peered":"2024-03-16T19:30:27.664837+0000","last_clean":"2024-03-16T19:30:27.664837+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:30:27.664837+0000","last_undegraded":"2024-03-16T19:30:27.664837+0000","last_fullsized":"2024-03-16T19:30:27.664837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562956+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:30:27.562956+0000","last_peered":"2024-03-16T19:30:27.562956+0000","last_clean":"2024-03-16T19:30:27.562956+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:30:27.562956+0000","last_undegraded":"2024-03-16T19:30:27.562956+0000","last_fullsized":"2024-03-16T19:30:27.562956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938244+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:30:27.938244+0000","last_peered":"2024-03-16T19:30:27.938244+0000","last_clean":"2024-03-16T19:30:27.938244+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:30:27.938244+0000","last_undegraded":"2024-03-16T19:30:27.938244+0000","last_fullsized":"2024-03-16T19:30:27.938244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664748+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:30:27.664748+0000","last_peered":"2024-03-16T19:30:27.664748+0000","last_clean":"2024-03-16T19:30:27.664748+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:30:27.664748+0000","last_undegraded":"2024-03-16T19:30:27.664748+0000","last_fullsized":"2024-03-16T19:30:27.664748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"40'4","reported_seq":61,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664623+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:30:27.664623+0000","last_peered":"2024-03-16T19:30:27.664623+0000","last_clean":"2024-03-16T19:30:27.664623+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:30:27.664623+0000","last_undegraded":"2024-03-16T19:30:27.664623+0000","last_fullsized":"2024-03-16T19:30:27.664623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938880+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:30:27.938880+0000","last_peered":"2024-03-16T19:30:27.938880+0000","last_clean":"2024-03-16T19:30:27.938880+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:30:27.938880+0000","last_undegraded":"2024-03-16T19:30:27.938880+0000","last_fullsized":"2024-03-16T19:30:27.938880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":59,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938666+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:30:27.938666+0000","last_peered":"2024-03-16T19:30:27.938666+0000","last_clean":"2024-03-16T19:30:27.938666+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:30:27.938666+0000","last_undegraded":"2024-03-16T19:30:27.938666+0000","last_fullsized":"2024-03-16T19:30:27.938666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":50,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664420+0000","last_change":"2024-03-16T19:29:44.413626+0000","last_active":"2024-03-16T19:30:27.664420+0000","last_peered":"2024-03-16T19:30:27.664420+0000","last_clean":"2024-03-16T19:30:27.664420+0000","last_became_active":"2024-03-16T19:29:44.413174+0000","last_became_peered":"2024-03-16T19:29:44.413174+0000","last_unstale":"2024-03-16T19:30:27.664420+0000","last_undegraded":"2024-03-16T19:30:27.664420+0000","last_fullsized":"2024-03-16T19:30:27.664420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:10:39.043849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937804+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:30:27.937804+0000","last_peered":"2024-03-16T19:30:27.937804+0000","last_clean":"2024-03-16T19:30:27.937804+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:30:27.937804+0000","last_undegraded":"2024-03-16T19:30:27.937804+0000","last_fullsized":"2024-03-16T19:30:27.937804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562259+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:30:27.562259+0000","last_peered":"2024-03-16T19:30:27.562259+0000","last_clean":"2024-03-16T19:30:27.562259+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:30:27.562259+0000","last_undegraded":"2024-03-16T19:30:27.562259+0000","last_fullsized":"2024-03-16T19:30:27.562259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664460+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:30:27.664460+0000","last_peered":"2024-03-16T19:30:27.664460+0000","last_clean":"2024-03-16T19:30:27.664460+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:30:27.664460+0000","last_undegraded":"2024-03-16T19:30:27.664460+0000","last_fullsized":"2024-03-16T19:30:27.664460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564267+0000","last_change":"2024-03-16T19:30:07.477317+0000","last_active":"2024-03-16T19:30:27.564267+0000","last_peered":"2024-03-16T19:30:27.564267+0000","last_clean":"2024-03-16T19:30:27.564267+0000","last_became_active":"2024-03-16T19:30:07.477037+0000","last_became_peered":"2024-03-16T19:30:07.477037+0000","last_unstale":"2024-03-16T19:30:27.564267+0000","last_undegraded":"2024-03-16T19:30:27.564267+0000","last_fullsized":"2024-03-16T19:30:27.564267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:59:02.992861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.665436+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:30:27.665436+0000","last_peered":"2024-03-16T19:30:27.665436+0000","last_clean":"2024-03-16T19:30:27.665436+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:30:27.665436+0000","last_undegraded":"2024-03-16T19:30:27.665436+0000","last_fullsized":"2024-03-16T19:30:27.665436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938697+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:30:27.938697+0000","last_peered":"2024-03-16T19:30:27.938697+0000","last_clean":"2024-03-16T19:30:27.938697+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:30:27.938697+0000","last_undegraded":"2024-03-16T19:30:27.938697+0000","last_fullsized":"2024-03-16T19:30:27.938697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":60,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938423+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:30:27.938423+0000","last_peered":"2024-03-16T19:30:27.938423+0000","last_clean":"2024-03-16T19:30:27.938423+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:30:27.938423+0000","last_undegraded":"2024-03-16T19:30:27.938423+0000","last_fullsized":"2024-03-16T19:30:27.938423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938115+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:30:27.938115+0000","last_peered":"2024-03-16T19:30:27.938115+0000","last_clean":"2024-03-16T19:30:27.938115+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:30:27.938115+0000","last_undegraded":"2024-03-16T19:30:27.938115+0000","last_fullsized":"2024-03-16T19:30:27.938115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562627+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:30:27.562627+0000","last_peered":"2024-03-16T19:30:27.562627+0000","last_clean":"2024-03-16T19:30:27.562627+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:30:27.562627+0000","last_undegraded":"2024-03-16T19:30:27.562627+0000","last_fullsized":"2024-03-16T19:30:27.562627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":60,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938027+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:30:27.938027+0000","last_peered":"2024-03-16T19:30:27.938027+0000","last_clean":"2024-03-16T19:30:27.938027+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:30:27.938027+0000","last_undegraded":"2024-03-16T19:30:27.938027+0000","last_fullsized":"2024-03-16T19:30:27.938027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563032+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:30:27.563032+0000","last_peered":"2024-03-16T19:30:27.563032+0000","last_clean":"2024-03-16T19:30:27.563032+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:30:27.563032+0000","last_undegraded":"2024-03-16T19:30:27.563032+0000","last_fullsized":"2024-03-16T19:30:27.563032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":60,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938317+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:30:27.938317+0000","last_peered":"2024-03-16T19:30:27.938317+0000","last_clean":"2024-03-16T19:30:27.938317+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:30:27.938317+0000","last_undegraded":"2024-03-16T19:30:27.938317+0000","last_fullsized":"2024-03-16T19:30:27.938317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":31,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664407+0000","last_change":"2024-03-16T19:30:07.684215+0000","last_active":"2024-03-16T19:30:27.664407+0000","last_peered":"2024-03-16T19:30:27.664407+0000","last_clean":"2024-03-16T19:30:27.664407+0000","last_became_active":"2024-03-16T19:30:07.683959+0000","last_became_peered":"2024-03-16T19:30:07.683959+0000","last_unstale":"2024-03-16T19:30:27.664407+0000","last_undegraded":"2024-03-16T19:30:27.664407+0000","last_fullsized":"2024-03-16T19:30:27.664407+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:23:17.567774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938953+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:30:27.938953+0000","last_peered":"2024-03-16T19:30:27.938953+0000","last_clean":"2024-03-16T19:30:27.938953+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:30:27.938953+0000","last_undegraded":"2024-03-16T19:30:27.938953+0000","last_fullsized":"2024-03-16T19:30:27.938953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938884+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:30:27.938884+0000","last_peered":"2024-03-16T19:30:27.938884+0000","last_clean":"2024-03-16T19:30:27.938884+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:30:27.938884+0000","last_undegraded":"2024-03-16T19:30:27.938884+0000","last_fullsized":"2024-03-16T19:30:27.938884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":64,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938536+0000","last_change":"2024-03-16T19:29:44.414207+0000","last_active":"2024-03-16T19:30:27.938536+0000","last_peered":"2024-03-16T19:30:27.938536+0000","last_clean":"2024-03-16T19:30:27.938536+0000","last_became_active":"2024-03-16T19:29:44.413852+0000","last_became_peered":"2024-03-16T19:29:44.413852+0000","last_unstale":"2024-03-16T19:30:27.938536+0000","last_undegraded":"2024-03-16T19:30:27.938536+0000","last_fullsized":"2024-03-16T19:30:27.938536+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:25.285467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7460,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":14920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1073152,"data_stored":792741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":491520,"data_stored":208453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":491520,"data_stored":207917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":487424,"data_stored":201917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":503808,"data_stored":214489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":487424,"data_stored":207827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":495616,"data_stored":207987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28084,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343756,"statfs":{"total":96636764160,"available":96608006144,"internally_reserved":0,"allocated":1081344,"data_stored":792797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":6662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":6662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:30:30.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:30:30.933 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:30.933 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:30.938 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:31.022 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:31.041+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.054+0000 7f9977519640 1 Processor -- start 2024-03-16T19:30:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.054+0000 7f9977519640 1 -- start start 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9977519640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9970154cb0 0x7f997006dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9977519640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99700cbba0 con 0x7f99700c8ba0 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9977519640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99700c91f0 con 0x7f997006fe10 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9977519640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997006c180 con 0x7f9970154cb0 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9975ab6640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f99700c8ba0 0x7f99700cabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39316/0 (socket says 172.21.15.172:39316) 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9975ab6640 1 -- 172.21.15.172:0/3953344993 learned_addr learned my addr 172.21.15.172:0/3953344993 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f99752b5640 1 --2- 172.21.15.172:0/3953344993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9970154cb0 0x7f997006dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9967fff640 1 -- 172.21.15.172:0/3953344993 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150994564 0 0) 0x7f99700cbba0 con 0x7f99700c8ba0 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9967fff640 1 -- 172.21.15.172:0/3953344993 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f995c003520 con 0x7f99700c8ba0 2024-03-16T19:30:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9967fff640 1 -- 172.21.15.172:0/3953344993 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651007274 0 0) 0x7f99700c91f0 con 0x7f997006fe10 2024-03-16T19:30:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f9967fff640 1 -- 172.21.15.172:0/3953344993 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99700cbba0 con 0x7f997006fe10 2024-03-16T19:30:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f99752b5640 1 -- 172.21.15.172:0/3953344993 >> v1:172.21.15.174:6789/0 conn(0x7f997006fe10 legacy=0x7f997015de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f99752b5640 1 -- 172.21.15.172:0/3953344993 >> v1:172.21.15.172:6789/0 conn(0x7f99700c8ba0 legacy=0x7f99700cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.055+0000 7f99752b5640 1 -- 172.21.15.172:0/3953344993 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997006c460 con 0x7f9970154cb0 2024-03-16T19:30:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.056+0000 7f99752b5640 1 --2- 172.21.15.172:0/3953344993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9970154cb0 0x7f997006dc20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f99680097d0 tx=0x7f996802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c528fe40f80fb43 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.056+0000 7f9967fff640 1 -- 172.21.15.172:0/3953344993 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9968031770 con 0x7f9970154cb0 2024-03-16T19:30:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.056+0000 7f9967fff640 1 -- 172.21.15.172:0/3953344993 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99680318d0 con 0x7f9970154cb0 2024-03-16T19:30:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.056+0000 7f9967fff640 1 -- 172.21.15.172:0/3953344993 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9968031c60 con 0x7f9970154cb0 2024-03-16T19:30:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.057+0000 7f9977519640 1 -- 172.21.15.172:0/3953344993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9970154cb0 msgr2=0x7f997006dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.057+0000 7f9977519640 1 --2- 172.21.15.172:0/3953344993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9970154cb0 0x7f997006dc20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f99680097d0 tx=0x7f996802b810 comp rx=0 tx=0).stop 2024-03-16T19:30:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.057+0000 7f9977519640 1 -- 172.21.15.172:0/3953344993 shutdown_connections 2024-03-16T19:30:31.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.057+0000 7f9977519640 1 --2- 172.21.15.172:0/3953344993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9970154cb0 0x7f997006dc20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.057+0000 7f9977519640 1 -- 172.21.15.172:0/3953344993 >> 172.21.15.172:0/3953344993 conn(0x7f9970070860 msgr2=0x7f9970070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:31.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.057+0000 7f9977519640 1 -- 172.21.15.172:0/3953344993 shutdown_connections 2024-03-16T19:30:31.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.057+0000 7f9977519640 1 -- 172.21.15.172:0/3953344993 wait complete. 2024-03-16T19:30:31.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.058+0000 7f9977519640 1 Processor -- start 2024-03-16T19:30:31.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.058+0000 7f9977519640 1 -- start start 2024-03-16T19:30:31.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9977519640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f997006fe10 0x7f99701f02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:31.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9977519640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99700c8ba0 0x7f99700cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9977519640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9970154cb0 0x7f997015d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9977519640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99701f0830 con 0x7f99700c8ba0 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9977519640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99700cefc0 con 0x7f9970154cb0 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9977519640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99701f3990 con 0x7f997006fe10 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9974ab4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99700c8ba0 0x7f99700cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9974ab4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99700c8ba0 0x7f99700cdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47580/0 (socket says 172.21.15.172:47580) 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9974ab4640 1 -- 172.21.15.172:0/443308243 learned_addr learned my addr 172.21.15.172:0/443308243 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f99752b5640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f997006fe10 0x7f99701f02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9975ab6640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9970154cb0 0x7f997015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9974ab4640 1 -- 172.21.15.172:0/443308243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f997006fe10 msgr2=0x7f99701f02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9974ab4640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f997006fe10 0x7f99701f02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9974ab4640 1 -- 172.21.15.172:0/443308243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9970154cb0 msgr2=0x7f997015d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9974ab4640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9970154cb0 0x7f997015d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.059+0000 7f9974ab4640 1 -- 172.21.15.172:0/443308243 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9968009400 con 0x7f99700c8ba0 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.060+0000 7f9974ab4640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99700c8ba0 0x7f99700cdce0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f996000c400 tx=0x7f996000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.060+0000 7f9965ffb640 1 -- 172.21.15.172:0/443308243 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99600052d0 con 0x7f99700c8ba0 2024-03-16T19:30:31.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.060+0000 7f9965ffb640 1 -- 172.21.15.172:0/443308243 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9960005430 con 0x7f99700c8ba0 2024-03-16T19:30:31.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.060+0000 7f9965ffb640 1 -- 172.21.15.172:0/443308243 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960003810 con 0x7f99700c8ba0 2024-03-16T19:30:31.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.060+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99700ce220 con 0x7f99700c8ba0 2024-03-16T19:30:31.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.060+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99700cece0 con 0x7f99700c8ba0 2024-03-16T19:30:31.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.061+0000 7f9965ffb640 1 -- 172.21.15.172:0/443308243 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9960003970 con 0x7f99700c8ba0 2024-03-16T19:30:31.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.061+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997006f770 con 0x7f99700c8ba0 2024-03-16T19:30:31.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.062+0000 7f9965ffb640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f995406b3b0 0x7f995406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:31.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.062+0000 7f9965ffb640 1 -- 172.21.15.172:0/443308243 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f99600891d0 con 0x7f99700c8ba0 2024-03-16T19:30:31.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.062+0000 7f99752b5640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f995406b3b0 0x7f995406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:31.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.062+0000 7f99752b5640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f995406b3b0 0x7f995406d870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9968036000 tx=0x7f996803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:31.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.066+0000 7f9965ffb640 1 -- 172.21.15.172:0/443308243 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9960053ef0 con 0x7f99700c8ba0 2024-03-16T19:30:31.085 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:31.103+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:31.097 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:31.097 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:31.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:31.185 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:31.204+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:31.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.253+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f99700681c0 con 0x7f995406b3b0 2024-03-16T19:30:31.241 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.259+0000 7f9965ffb640 1 -- 172.21.15.172:0/443308243 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251867 (secure 0 0 0) 0x7f99700681c0 con 0x7f995406b3b0 2024-03-16T19:30:31.243 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:30:31.244 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.263+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f995406b3b0 msgr2=0x7f995406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.263+0000 7f9977519640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f995406b3b0 0x7f995406d870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9968036000 tx=0x7f996803d040 comp rx=0 tx=0).stop 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.263+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99700c8ba0 msgr2=0x7f99700cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.263+0000 7f9977519640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99700c8ba0 0x7f99700cdce0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f996000c400 tx=0x7f996000c8c0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.263+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 shutdown_connections 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.263+0000 7f9977519640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9970154cb0 0x7f997015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.263+0000 7f9977519640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99700c8ba0 0x7f99700cdce0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.263+0000 7f9977519640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f995406b3b0 0x7f995406d870 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.264+0000 7f9977519640 1 --2- 172.21.15.172:0/443308243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f997006fe10 0x7f99701f02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.264+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 >> 172.21.15.172:0/443308243 conn(0x7f9970070860 msgr2=0x7f997015bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.264+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 shutdown_connections 2024-03-16T19:30:31.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.264+0000 7f9977519640 1 -- 172.21.15.172:0/443308243 wait complete. 2024-03-16T19:30:31.260 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-03-16T19:30:30.577908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597828,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":216,"num_write_kb":2441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164,"ondisk_log_size":164,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221188,"kb_used_data":4992,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753532,"statfs":{"total":773094113280,"available":772867616768,"internally_reserved":0,"allocated":5111808,"data_stored":2834128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.024646"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938344+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:30:27.938344+0000","last_peered":"2024-03-16T19:30:27.938344+0000","last_clean":"2024-03-16T19:30:27.938344+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:30:27.938344+0000","last_undegraded":"2024-03-16T19:30:27.938344+0000","last_fullsized":"2024-03-16T19:30:27.938344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664523+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:30:27.664523+0000","last_peered":"2024-03-16T19:30:27.664523+0000","last_clean":"2024-03-16T19:30:27.664523+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:30:27.664523+0000","last_undegraded":"2024-03-16T19:30:27.664523+0000","last_fullsized":"2024-03-16T19:30:27.664523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":49,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564569+0000","last_change":"2024-03-16T19:29:44.412537+0000","last_active":"2024-03-16T19:30:27.564569+0000","last_peered":"2024-03-16T19:30:27.564569+0000","last_clean":"2024-03-16T19:30:27.564569+0000","last_became_active":"2024-03-16T19:29:44.412236+0000","last_became_peered":"2024-03-16T19:29:44.412236+0000","last_unstale":"2024-03-16T19:30:27.564569+0000","last_undegraded":"2024-03-16T19:30:27.564569+0000","last_fullsized":"2024-03-16T19:30:27.564569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:50:57.180910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562949+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:30:27.562949+0000","last_peered":"2024-03-16T19:30:27.562949+0000","last_clean":"2024-03-16T19:30:27.562949+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:30:27.562949+0000","last_undegraded":"2024-03-16T19:30:27.562949+0000","last_fullsized":"2024-03-16T19:30:27.562949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938310+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:30:27.938310+0000","last_peered":"2024-03-16T19:30:27.938310+0000","last_clean":"2024-03-16T19:30:27.938310+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:30:27.938310+0000","last_undegraded":"2024-03-16T19:30:27.938310+0000","last_fullsized":"2024-03-16T19:30:27.938310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562831+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:30:27.562831+0000","last_peered":"2024-03-16T19:30:27.562831+0000","last_clean":"2024-03-16T19:30:27.562831+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:30:27.562831+0000","last_undegraded":"2024-03-16T19:30:27.562831+0000","last_fullsized":"2024-03-16T19:30:27.562831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937835+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:30:27.937835+0000","last_peered":"2024-03-16T19:30:27.937835+0000","last_clean":"2024-03-16T19:30:27.937835+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:30:27.937835+0000","last_undegraded":"2024-03-16T19:30:27.937835+0000","last_fullsized":"2024-03-16T19:30:27.937835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938054+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:30:27.938054+0000","last_peered":"2024-03-16T19:30:27.938054+0000","last_clean":"2024-03-16T19:30:27.938054+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:30:27.938054+0000","last_undegraded":"2024-03-16T19:30:27.938054+0000","last_fullsized":"2024-03-16T19:30:27.938054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938441+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:30:27.938441+0000","last_peered":"2024-03-16T19:30:27.938441+0000","last_clean":"2024-03-16T19:30:27.938441+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:30:27.938441+0000","last_undegraded":"2024-03-16T19:30:27.938441+0000","last_fullsized":"2024-03-16T19:30:27.938441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938573+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:30:27.938573+0000","last_peered":"2024-03-16T19:30:27.938573+0000","last_clean":"2024-03-16T19:30:27.938573+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:30:27.938573+0000","last_undegraded":"2024-03-16T19:30:27.938573+0000","last_fullsized":"2024-03-16T19:30:27.938573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938366+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:30:27.938366+0000","last_peered":"2024-03-16T19:30:27.938366+0000","last_clean":"2024-03-16T19:30:27.938366+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:30:27.938366+0000","last_undegraded":"2024-03-16T19:30:27.938366+0000","last_fullsized":"2024-03-16T19:30:27.938366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563010+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:30:27.563010+0000","last_peered":"2024-03-16T19:30:27.563010+0000","last_clean":"2024-03-16T19:30:27.563010+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:30:27.563010+0000","last_undegraded":"2024-03-16T19:30:27.563010+0000","last_fullsized":"2024-03-16T19:30:27.563010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938617+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:30:27.938617+0000","last_peered":"2024-03-16T19:30:27.938617+0000","last_clean":"2024-03-16T19:30:27.938617+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:30:27.938617+0000","last_undegraded":"2024-03-16T19:30:27.938617+0000","last_fullsized":"2024-03-16T19:30:27.938617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":50,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.665283+0000","last_change":"2024-03-16T19:29:44.413594+0000","last_active":"2024-03-16T19:30:27.665283+0000","last_peered":"2024-03-16T19:30:27.665283+0000","last_clean":"2024-03-16T19:30:27.665283+0000","last_became_active":"2024-03-16T19:29:44.413235+0000","last_became_peered":"2024-03-16T19:29:44.413235+0000","last_unstale":"2024-03-16T19:30:27.665283+0000","last_undegraded":"2024-03-16T19:30:27.665283+0000","last_fullsized":"2024-03-16T19:30:27.665283+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:35.778808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664388+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:30:27.664388+0000","last_peered":"2024-03-16T19:30:27.664388+0000","last_clean":"2024-03-16T19:30:27.664388+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:30:27.664388+0000","last_undegraded":"2024-03-16T19:30:27.664388+0000","last_fullsized":"2024-03-16T19:30:27.664388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562300+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:30:27.562300+0000","last_peered":"2024-03-16T19:30:27.562300+0000","last_clean":"2024-03-16T19:30:27.562300+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:30:27.562300+0000","last_undegraded":"2024-03-16T19:30:27.562300+0000","last_fullsized":"2024-03-16T19:30:27.562300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562830+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:30:27.562830+0000","last_peered":"2024-03-16T19:30:27.562830+0000","last_clean":"2024-03-16T19:30:27.562830+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:30:27.562830+0000","last_undegraded":"2024-03-16T19:30:27.562830+0000","last_fullsized":"2024-03-16T19:30:27.562830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564600+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:30:27.564600+0000","last_peered":"2024-03-16T19:30:27.564600+0000","last_clean":"2024-03-16T19:30:27.564600+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:30:27.564600+0000","last_undegraded":"2024-03-16T19:30:27.564600+0000","last_fullsized":"2024-03-16T19:30:27.564600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938163+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:30:27.938163+0000","last_peered":"2024-03-16T19:30:27.938163+0000","last_clean":"2024-03-16T19:30:27.938163+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:30:27.938163+0000","last_undegraded":"2024-03-16T19:30:27.938163+0000","last_fullsized":"2024-03-16T19:30:27.938163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938295+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:30:27.938295+0000","last_peered":"2024-03-16T19:30:27.938295+0000","last_clean":"2024-03-16T19:30:27.938295+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:30:27.938295+0000","last_undegraded":"2024-03-16T19:30:27.938295+0000","last_fullsized":"2024-03-16T19:30:27.938295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938230+0000","last_change":"2024-03-16T19:29:44.414141+0000","last_active":"2024-03-16T19:30:27.938230+0000","last_peered":"2024-03-16T19:30:27.938230+0000","last_clean":"2024-03-16T19:30:27.938230+0000","last_became_active":"2024-03-16T19:29:44.413873+0000","last_became_peered":"2024-03-16T19:29:44.413873+0000","last_unstale":"2024-03-16T19:30:27.938230+0000","last_undegraded":"2024-03-16T19:30:27.938230+0000","last_fullsized":"2024-03-16T19:30:27.938230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:03:03.291206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564391+0000","last_change":"2024-03-16T19:30:07.591770+0000","last_active":"2024-03-16T19:30:27.564391+0000","last_peered":"2024-03-16T19:30:27.564391+0000","last_clean":"2024-03-16T19:30:27.564391+0000","last_became_active":"2024-03-16T19:30:07.591482+0000","last_became_peered":"2024-03-16T19:30:07.591482+0000","last_unstale":"2024-03-16T19:30:27.564391+0000","last_undegraded":"2024-03-16T19:30:27.564391+0000","last_fullsized":"2024-03-16T19:30:27.564391+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:02.003669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938106+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:30:27.938106+0000","last_peered":"2024-03-16T19:30:27.938106+0000","last_clean":"2024-03-16T19:30:27.938106+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:30:27.938106+0000","last_undegraded":"2024-03-16T19:30:27.938106+0000","last_fullsized":"2024-03-16T19:30:27.938106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562360+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:30:27.562360+0000","last_peered":"2024-03-16T19:30:27.562360+0000","last_clean":"2024-03-16T19:30:27.562360+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:30:27.562360+0000","last_undegraded":"2024-03-16T19:30:27.562360+0000","last_fullsized":"2024-03-16T19:30:27.562360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938946+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:30:27.938946+0000","last_peered":"2024-03-16T19:30:27.938946+0000","last_clean":"2024-03-16T19:30:27.938946+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:30:27.938946+0000","last_undegraded":"2024-03-16T19:30:27.938946+0000","last_fullsized":"2024-03-16T19:30:27.938946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938614+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:30:27.938614+0000","last_peered":"2024-03-16T19:30:27.938614+0000","last_clean":"2024-03-16T19:30:27.938614+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:30:27.938614+0000","last_undegraded":"2024-03-16T19:30:27.938614+0000","last_fullsized":"2024-03-16T19:30:27.938614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664932+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:30:27.664932+0000","last_peered":"2024-03-16T19:30:27.664932+0000","last_clean":"2024-03-16T19:30:27.664932+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:30:27.664932+0000","last_undegraded":"2024-03-16T19:30:27.664932+0000","last_fullsized":"2024-03-16T19:30:27.664932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938755+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:30:27.938755+0000","last_peered":"2024-03-16T19:30:27.938755+0000","last_clean":"2024-03-16T19:30:27.938755+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:30:27.938755+0000","last_undegraded":"2024-03-16T19:30:27.938755+0000","last_fullsized":"2024-03-16T19:30:27.938755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":50,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938527+0000","last_change":"2024-03-16T19:29:44.412475+0000","last_active":"2024-03-16T19:30:27.938527+0000","last_peered":"2024-03-16T19:30:27.938527+0000","last_clean":"2024-03-16T19:30:27.938527+0000","last_became_active":"2024-03-16T19:29:44.412176+0000","last_became_peered":"2024-03-16T19:29:44.412176+0000","last_unstale":"2024-03-16T19:30:27.938527+0000","last_undegraded":"2024-03-16T19:30:27.938527+0000","last_fullsized":"2024-03-16T19:30:27.938527+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:29:34.284744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562925+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:30:27.562925+0000","last_peered":"2024-03-16T19:30:27.562925+0000","last_clean":"2024-03-16T19:30:27.562925+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:30:27.562925+0000","last_undegraded":"2024-03-16T19:30:27.562925+0000","last_fullsized":"2024-03-16T19:30:27.562925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564355+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:30:27.564355+0000","last_peered":"2024-03-16T19:30:27.564355+0000","last_clean":"2024-03-16T19:30:27.564355+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:30:27.564355+0000","last_undegraded":"2024-03-16T19:30:27.564355+0000","last_fullsized":"2024-03-16T19:30:27.564355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937963+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:30:27.937963+0000","last_peered":"2024-03-16T19:30:27.937963+0000","last_clean":"2024-03-16T19:30:27.937963+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:30:27.937963+0000","last_undegraded":"2024-03-16T19:30:27.937963+0000","last_fullsized":"2024-03-16T19:30:27.937963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562211+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:30:27.562211+0000","last_peered":"2024-03-16T19:30:27.562211+0000","last_clean":"2024-03-16T19:30:27.562211+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:30:27.562211+0000","last_undegraded":"2024-03-16T19:30:27.562211+0000","last_fullsized":"2024-03-16T19:30:27.562211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938779+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:30:27.938779+0000","last_peered":"2024-03-16T19:30:27.938779+0000","last_clean":"2024-03-16T19:30:27.938779+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:30:27.938779+0000","last_undegraded":"2024-03-16T19:30:27.938779+0000","last_fullsized":"2024-03-16T19:30:27.938779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562654+0000","last_change":"2024-03-16T19:29:44.412541+0000","last_active":"2024-03-16T19:30:27.562654+0000","last_peered":"2024-03-16T19:30:27.562654+0000","last_clean":"2024-03-16T19:30:27.562654+0000","last_became_active":"2024-03-16T19:29:44.412194+0000","last_became_peered":"2024-03-16T19:29:44.412194+0000","last_unstale":"2024-03-16T19:30:27.562654+0000","last_undegraded":"2024-03-16T19:30:27.562654+0000","last_fullsized":"2024-03-16T19:30:27.562654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:51:50.130592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938168+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:30:27.938168+0000","last_peered":"2024-03-16T19:30:27.938168+0000","last_clean":"2024-03-16T19:30:27.938168+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:30:27.938168+0000","last_undegraded":"2024-03-16T19:30:27.938168+0000","last_fullsized":"2024-03-16T19:30:27.938168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938200+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:30:27.938200+0000","last_peered":"2024-03-16T19:30:27.938200+0000","last_clean":"2024-03-16T19:30:27.938200+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:30:27.938200+0000","last_undegraded":"2024-03-16T19:30:27.938200+0000","last_fullsized":"2024-03-16T19:30:27.938200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938116+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:30:27.938116+0000","last_peered":"2024-03-16T19:30:27.938116+0000","last_clean":"2024-03-16T19:30:27.938116+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:30:27.938116+0000","last_undegraded":"2024-03-16T19:30:27.938116+0000","last_fullsized":"2024-03-16T19:30:27.938116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562399+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:30:27.562399+0000","last_peered":"2024-03-16T19:30:27.562399+0000","last_clean":"2024-03-16T19:30:27.562399+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:30:27.562399+0000","last_undegraded":"2024-03-16T19:30:27.562399+0000","last_fullsized":"2024-03-16T19:30:27.562399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664292+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:30:27.664292+0000","last_peered":"2024-03-16T19:30:27.664292+0000","last_clean":"2024-03-16T19:30:27.664292+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:30:27.664292+0000","last_undegraded":"2024-03-16T19:30:27.664292+0000","last_fullsized":"2024-03-16T19:30:27.664292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938657+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:30:27.938657+0000","last_peered":"2024-03-16T19:30:27.938657+0000","last_clean":"2024-03-16T19:30:27.938657+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:30:27.938657+0000","last_undegraded":"2024-03-16T19:30:27.938657+0000","last_fullsized":"2024-03-16T19:30:27.938657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562691+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:30:27.562691+0000","last_peered":"2024-03-16T19:30:27.562691+0000","last_clean":"2024-03-16T19:30:27.562691+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:30:27.562691+0000","last_undegraded":"2024-03-16T19:30:27.562691+0000","last_fullsized":"2024-03-16T19:30:27.562691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562776+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:30:27.562776+0000","last_peered":"2024-03-16T19:30:27.562776+0000","last_clean":"2024-03-16T19:30:27.562776+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:30:27.562776+0000","last_undegraded":"2024-03-16T19:30:27.562776+0000","last_fullsized":"2024-03-16T19:30:27.562776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562734+0000","last_change":"2024-03-16T19:29:44.413558+0000","last_active":"2024-03-16T19:30:27.562734+0000","last_peered":"2024-03-16T19:30:27.562734+0000","last_clean":"2024-03-16T19:30:27.562734+0000","last_became_active":"2024-03-16T19:29:44.413310+0000","last_became_peered":"2024-03-16T19:29:44.413310+0000","last_unstale":"2024-03-16T19:30:27.562734+0000","last_undegraded":"2024-03-16T19:30:27.562734+0000","last_fullsized":"2024-03-16T19:30:27.562734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:25:53.846575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":60,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.665015+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:30:27.665015+0000","last_peered":"2024-03-16T19:30:27.665015+0000","last_clean":"2024-03-16T19:30:27.665015+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:30:27.665015+0000","last_undegraded":"2024-03-16T19:30:27.665015+0000","last_fullsized":"2024-03-16T19:30:27.665015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563032+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:30:27.563032+0000","last_peered":"2024-03-16T19:30:27.563032+0000","last_clean":"2024-03-16T19:30:27.563032+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:30:27.563032+0000","last_undegraded":"2024-03-16T19:30:27.563032+0000","last_fullsized":"2024-03-16T19:30:27.563032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"36'11","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938392+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:30:27.938392+0000","last_peered":"2024-03-16T19:30:27.938392+0000","last_clean":"2024-03-16T19:30:27.938392+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:30:27.938392+0000","last_undegraded":"2024-03-16T19:30:27.938392+0000","last_fullsized":"2024-03-16T19:30:27.938392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6662,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938074+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:30:27.938074+0000","last_peered":"2024-03-16T19:30:27.938074+0000","last_clean":"2024-03-16T19:30:27.938074+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:30:27.938074+0000","last_undegraded":"2024-03-16T19:30:27.938074+0000","last_fullsized":"2024-03-16T19:30:27.938074+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":64,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562743+0000","last_change":"2024-03-16T19:29:44.412731+0000","last_active":"2024-03-16T19:30:27.562743+0000","last_peered":"2024-03-16T19:30:27.562743+0000","last_clean":"2024-03-16T19:30:27.562743+0000","last_became_active":"2024-03-16T19:29:44.412174+0000","last_became_peered":"2024-03-16T19:29:44.412174+0000","last_unstale":"2024-03-16T19:30:27.562743+0000","last_undegraded":"2024-03-16T19:30:27.562743+0000","last_fullsized":"2024-03-16T19:30:27.562743+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:57.962644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937941+0000","last_change":"2024-03-16T19:29:44.412524+0000","last_active":"2024-03-16T19:30:27.937941+0000","last_peered":"2024-03-16T19:30:27.937941+0000","last_clean":"2024-03-16T19:30:27.937941+0000","last_became_active":"2024-03-16T19:29:44.412200+0000","last_became_peered":"2024-03-16T19:29:44.412200+0000","last_unstale":"2024-03-16T19:30:27.937941+0000","last_undegraded":"2024-03-16T19:30:27.937941+0000","last_fullsized":"2024-03-16T19:30:27.937941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:52.636929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563008+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:30:27.563008+0000","last_peered":"2024-03-16T19:30:27.563008+0000","last_clean":"2024-03-16T19:30:27.563008+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:30:27.563008+0000","last_undegraded":"2024-03-16T19:30:27.563008+0000","last_fullsized":"2024-03-16T19:30:27.563008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938400+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:30:27.938400+0000","last_peered":"2024-03-16T19:30:27.938400+0000","last_clean":"2024-03-16T19:30:27.938400+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:30:27.938400+0000","last_undegraded":"2024-03-16T19:30:27.938400+0000","last_fullsized":"2024-03-16T19:30:27.938400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938049+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:30:27.938049+0000","last_peered":"2024-03-16T19:30:27.938049+0000","last_clean":"2024-03-16T19:30:27.938049+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:30:27.938049+0000","last_undegraded":"2024-03-16T19:30:27.938049+0000","last_fullsized":"2024-03-16T19:30:27.938049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938035+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:30:27.938035+0000","last_peered":"2024-03-16T19:30:27.938035+0000","last_clean":"2024-03-16T19:30:27.938035+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:30:27.938035+0000","last_undegraded":"2024-03-16T19:30:27.938035+0000","last_fullsized":"2024-03-16T19:30:27.938035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":64,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564350+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:30:27.564350+0000","last_peered":"2024-03-16T19:30:27.564350+0000","last_clean":"2024-03-16T19:30:27.564350+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:30:27.564350+0000","last_undegraded":"2024-03-16T19:30:27.564350+0000","last_fullsized":"2024-03-16T19:30:27.564350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937897+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:30:27.937897+0000","last_peered":"2024-03-16T19:30:27.937897+0000","last_clean":"2024-03-16T19:30:27.937897+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:30:27.937897+0000","last_undegraded":"2024-03-16T19:30:27.937897+0000","last_fullsized":"2024-03-16T19:30:27.937897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":59,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562545+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:30:27.562545+0000","last_peered":"2024-03-16T19:30:27.562545+0000","last_clean":"2024-03-16T19:30:27.562545+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:30:27.562545+0000","last_undegraded":"2024-03-16T19:30:27.562545+0000","last_fullsized":"2024-03-16T19:30:27.562545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938264+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:30:27.938264+0000","last_peered":"2024-03-16T19:30:27.938264+0000","last_clean":"2024-03-16T19:30:27.938264+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:30:27.938264+0000","last_undegraded":"2024-03-16T19:30:27.938264+0000","last_fullsized":"2024-03-16T19:30:27.938264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564661+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:30:27.564661+0000","last_peered":"2024-03-16T19:30:27.564661+0000","last_clean":"2024-03-16T19:30:27.564661+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:30:27.564661+0000","last_undegraded":"2024-03-16T19:30:27.564661+0000","last_fullsized":"2024-03-16T19:30:27.564661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938492+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:30:27.938492+0000","last_peered":"2024-03-16T19:30:27.938492+0000","last_clean":"2024-03-16T19:30:27.938492+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:30:27.938492+0000","last_undegraded":"2024-03-16T19:30:27.938492+0000","last_fullsized":"2024-03-16T19:30:27.938492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.665114+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:30:27.665114+0000","last_peered":"2024-03-16T19:30:27.665114+0000","last_clean":"2024-03-16T19:30:27.665114+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:30:27.665114+0000","last_undegraded":"2024-03-16T19:30:27.665114+0000","last_fullsized":"2024-03-16T19:30:27.665114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563118+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:30:27.563118+0000","last_peered":"2024-03-16T19:30:27.563118+0000","last_clean":"2024-03-16T19:30:27.563118+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:30:27.563118+0000","last_undegraded":"2024-03-16T19:30:27.563118+0000","last_fullsized":"2024-03-16T19:30:27.563118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564437+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:30:27.564437+0000","last_peered":"2024-03-16T19:30:27.564437+0000","last_clean":"2024-03-16T19:30:27.564437+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:30:27.564437+0000","last_undegraded":"2024-03-16T19:30:27.564437+0000","last_fullsized":"2024-03-16T19:30:27.564437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938018+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:30:27.938018+0000","last_peered":"2024-03-16T19:30:27.938018+0000","last_clean":"2024-03-16T19:30:27.938018+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:30:27.938018+0000","last_undegraded":"2024-03-16T19:30:27.938018+0000","last_fullsized":"2024-03-16T19:30:27.938018+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":214,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937928+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:30:27.937928+0000","last_peered":"2024-03-16T19:30:27.937928+0000","last_clean":"2024-03-16T19:30:27.937928+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:30:27.937928+0000","last_undegraded":"2024-03-16T19:30:27.937928+0000","last_fullsized":"2024-03-16T19:30:27.937928+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562537+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:30:27.562537+0000","last_peered":"2024-03-16T19:30:27.562537+0000","last_clean":"2024-03-16T19:30:27.562537+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:30:27.562537+0000","last_undegraded":"2024-03-16T19:30:27.562537+0000","last_fullsized":"2024-03-16T19:30:27.562537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938707+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:30:27.938707+0000","last_peered":"2024-03-16T19:30:27.938707+0000","last_clean":"2024-03-16T19:30:27.938707+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:30:27.938707+0000","last_undegraded":"2024-03-16T19:30:27.938707+0000","last_fullsized":"2024-03-16T19:30:27.938707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937811+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:30:27.937811+0000","last_peered":"2024-03-16T19:30:27.937811+0000","last_clean":"2024-03-16T19:30:27.937811+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:30:27.937811+0000","last_undegraded":"2024-03-16T19:30:27.937811+0000","last_fullsized":"2024-03-16T19:30:27.937811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":51,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562250+0000","last_change":"2024-03-16T19:29:44.412505+0000","last_active":"2024-03-16T19:30:27.562250+0000","last_peered":"2024-03-16T19:30:27.562250+0000","last_clean":"2024-03-16T19:30:27.562250+0000","last_became_active":"2024-03-16T19:29:44.412221+0000","last_became_peered":"2024-03-16T19:29:44.412221+0000","last_unstale":"2024-03-16T19:30:27.562250+0000","last_undegraded":"2024-03-16T19:30:27.562250+0000","last_fullsized":"2024-03-16T19:30:27.562250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:14.739638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564454+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:30:27.564454+0000","last_peered":"2024-03-16T19:30:27.564454+0000","last_clean":"2024-03-16T19:30:27.564454+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:30:27.564454+0000","last_undegraded":"2024-03-16T19:30:27.564454+0000","last_fullsized":"2024-03-16T19:30:27.564454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664372+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:30:27.664372+0000","last_peered":"2024-03-16T19:30:27.664372+0000","last_clean":"2024-03-16T19:30:27.664372+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:30:27.664372+0000","last_undegraded":"2024-03-16T19:30:27.664372+0000","last_fullsized":"2024-03-16T19:30:27.664372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563096+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:30:27.563096+0000","last_peered":"2024-03-16T19:30:27.563096+0000","last_clean":"2024-03-16T19:30:27.563096+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:30:27.563096+0000","last_undegraded":"2024-03-16T19:30:27.563096+0000","last_fullsized":"2024-03-16T19:30:27.563096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562837+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:30:27.562837+0000","last_peered":"2024-03-16T19:30:27.562837+0000","last_clean":"2024-03-16T19:30:27.562837+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:30:27.562837+0000","last_undegraded":"2024-03-16T19:30:27.562837+0000","last_fullsized":"2024-03-16T19:30:27.562837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664837+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:30:27.664837+0000","last_peered":"2024-03-16T19:30:27.664837+0000","last_clean":"2024-03-16T19:30:27.664837+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:30:27.664837+0000","last_undegraded":"2024-03-16T19:30:27.664837+0000","last_fullsized":"2024-03-16T19:30:27.664837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562956+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:30:27.562956+0000","last_peered":"2024-03-16T19:30:27.562956+0000","last_clean":"2024-03-16T19:30:27.562956+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:30:27.562956+0000","last_undegraded":"2024-03-16T19:30:27.562956+0000","last_fullsized":"2024-03-16T19:30:27.562956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938244+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:30:27.938244+0000","last_peered":"2024-03-16T19:30:27.938244+0000","last_clean":"2024-03-16T19:30:27.938244+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:30:27.938244+0000","last_undegraded":"2024-03-16T19:30:27.938244+0000","last_fullsized":"2024-03-16T19:30:27.938244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664748+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:30:27.664748+0000","last_peered":"2024-03-16T19:30:27.664748+0000","last_clean":"2024-03-16T19:30:27.664748+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:30:27.664748+0000","last_undegraded":"2024-03-16T19:30:27.664748+0000","last_fullsized":"2024-03-16T19:30:27.664748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"40'4","reported_seq":61,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664623+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:30:27.664623+0000","last_peered":"2024-03-16T19:30:27.664623+0000","last_clean":"2024-03-16T19:30:27.664623+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:30:27.664623+0000","last_undegraded":"2024-03-16T19:30:27.664623+0000","last_fullsized":"2024-03-16T19:30:27.664623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938880+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:30:27.938880+0000","last_peered":"2024-03-16T19:30:27.938880+0000","last_clean":"2024-03-16T19:30:27.938880+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:30:27.938880+0000","last_undegraded":"2024-03-16T19:30:27.938880+0000","last_fullsized":"2024-03-16T19:30:27.938880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":59,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938666+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:30:27.938666+0000","last_peered":"2024-03-16T19:30:27.938666+0000","last_clean":"2024-03-16T19:30:27.938666+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:30:27.938666+0000","last_undegraded":"2024-03-16T19:30:27.938666+0000","last_fullsized":"2024-03-16T19:30:27.938666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":50,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664420+0000","last_change":"2024-03-16T19:29:44.413626+0000","last_active":"2024-03-16T19:30:27.664420+0000","last_peered":"2024-03-16T19:30:27.664420+0000","last_clean":"2024-03-16T19:30:27.664420+0000","last_became_active":"2024-03-16T19:29:44.413174+0000","last_became_peered":"2024-03-16T19:29:44.413174+0000","last_unstale":"2024-03-16T19:30:27.664420+0000","last_undegraded":"2024-03-16T19:30:27.664420+0000","last_fullsized":"2024-03-16T19:30:27.664420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:10:39.043849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.937804+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:30:27.937804+0000","last_peered":"2024-03-16T19:30:27.937804+0000","last_clean":"2024-03-16T19:30:27.937804+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:30:27.937804+0000","last_undegraded":"2024-03-16T19:30:27.937804+0000","last_fullsized":"2024-03-16T19:30:27.937804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562259+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:30:27.562259+0000","last_peered":"2024-03-16T19:30:27.562259+0000","last_clean":"2024-03-16T19:30:27.562259+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:30:27.562259+0000","last_undegraded":"2024-03-16T19:30:27.562259+0000","last_fullsized":"2024-03-16T19:30:27.562259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664460+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:30:27.664460+0000","last_peered":"2024-03-16T19:30:27.664460+0000","last_clean":"2024-03-16T19:30:27.664460+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:30:27.664460+0000","last_undegraded":"2024-03-16T19:30:27.664460+0000","last_fullsized":"2024-03-16T19:30:27.664460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.564267+0000","last_change":"2024-03-16T19:30:07.477317+0000","last_active":"2024-03-16T19:30:27.564267+0000","last_peered":"2024-03-16T19:30:27.564267+0000","last_clean":"2024-03-16T19:30:27.564267+0000","last_became_active":"2024-03-16T19:30:07.477037+0000","last_became_peered":"2024-03-16T19:30:07.477037+0000","last_unstale":"2024-03-16T19:30:27.564267+0000","last_undegraded":"2024-03-16T19:30:27.564267+0000","last_fullsized":"2024-03-16T19:30:27.564267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:59:02.992861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":57,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.665436+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:30:27.665436+0000","last_peered":"2024-03-16T19:30:27.665436+0000","last_clean":"2024-03-16T19:30:27.665436+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:30:27.665436+0000","last_undegraded":"2024-03-16T19:30:27.665436+0000","last_fullsized":"2024-03-16T19:30:27.665436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938697+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:30:27.938697+0000","last_peered":"2024-03-16T19:30:27.938697+0000","last_clean":"2024-03-16T19:30:27.938697+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:30:27.938697+0000","last_undegraded":"2024-03-16T19:30:27.938697+0000","last_fullsized":"2024-03-16T19:30:27.938697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":60,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938423+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:30:27.938423+0000","last_peered":"2024-03-16T19:30:27.938423+0000","last_clean":"2024-03-16T19:30:27.938423+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:30:27.938423+0000","last_undegraded":"2024-03-16T19:30:27.938423+0000","last_fullsized":"2024-03-16T19:30:27.938423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938115+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:30:27.938115+0000","last_peered":"2024-03-16T19:30:27.938115+0000","last_clean":"2024-03-16T19:30:27.938115+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:30:27.938115+0000","last_undegraded":"2024-03-16T19:30:27.938115+0000","last_fullsized":"2024-03-16T19:30:27.938115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.562627+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:30:27.562627+0000","last_peered":"2024-03-16T19:30:27.562627+0000","last_clean":"2024-03-16T19:30:27.562627+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:30:27.562627+0000","last_undegraded":"2024-03-16T19:30:27.562627+0000","last_fullsized":"2024-03-16T19:30:27.562627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":60,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938027+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:30:27.938027+0000","last_peered":"2024-03-16T19:30:27.938027+0000","last_clean":"2024-03-16T19:30:27.938027+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:30:27.938027+0000","last_undegraded":"2024-03-16T19:30:27.938027+0000","last_fullsized":"2024-03-16T19:30:27.938027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.563032+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:30:27.563032+0000","last_peered":"2024-03-16T19:30:27.563032+0000","last_clean":"2024-03-16T19:30:27.563032+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:30:27.563032+0000","last_undegraded":"2024-03-16T19:30:27.563032+0000","last_fullsized":"2024-03-16T19:30:27.563032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":60,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938317+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:30:27.938317+0000","last_peered":"2024-03-16T19:30:27.938317+0000","last_clean":"2024-03-16T19:30:27.938317+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:30:27.938317+0000","last_undegraded":"2024-03-16T19:30:27.938317+0000","last_fullsized":"2024-03-16T19:30:27.938317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":31,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.664407+0000","last_change":"2024-03-16T19:30:07.684215+0000","last_active":"2024-03-16T19:30:27.664407+0000","last_peered":"2024-03-16T19:30:27.664407+0000","last_clean":"2024-03-16T19:30:27.664407+0000","last_became_active":"2024-03-16T19:30:07.683959+0000","last_became_peered":"2024-03-16T19:30:07.683959+0000","last_unstale":"2024-03-16T19:30:27.664407+0000","last_undegraded":"2024-03-16T19:30:27.664407+0000","last_fullsized":"2024-03-16T19:30:27.664407+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:23:17.567774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938953+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:30:27.938953+0000","last_peered":"2024-03-16T19:30:27.938953+0000","last_clean":"2024-03-16T19:30:27.938953+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:30:27.938953+0000","last_undegraded":"2024-03-16T19:30:27.938953+0000","last_fullsized":"2024-03-16T19:30:27.938953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938884+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:30:27.938884+0000","last_peered":"2024-03-16T19:30:27.938884+0000","last_clean":"2024-03-16T19:30:27.938884+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:30:27.938884+0000","last_undegraded":"2024-03-16T19:30:27.938884+0000","last_fullsized":"2024-03-16T19:30:27.938884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":64,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-16T19:30:27.938536+0000","last_change":"2024-03-16T19:29:44.414207+0000","last_active":"2024-03-16T19:30:27.938536+0000","last_peered":"2024-03-16T19:30:27.938536+0000","last_clean":"2024-03-16T19:30:27.938536+0000","last_became_active":"2024-03-16T19:29:44.413852+0000","last_became_peered":"2024-03-16T19:29:44.413852+0000","last_unstale":"2024-03-16T19:30:27.938536+0000","last_undegraded":"2024-03-16T19:30:27.938536+0000","last_fullsized":"2024-03-16T19:30:27.938536+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:25.285467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7460,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":14920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1073152,"data_stored":792741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":491520,"data_stored":208453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":491520,"data_stored":207917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":487424,"data_stored":201917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":503808,"data_stored":214489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":487424,"data_stored":207827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":495616,"data_stored":207987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28084,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343756,"statfs":{"total":96636764160,"available":96608006144,"internally_reserved":0,"allocated":1081344,"data_stored":792797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":6662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":6662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:30:31.264 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-16T19:30:31.264 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:30:31.264 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-16T19:30:31.264 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-16T19:30:31.264 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-16T19:30:31.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-03-16T19:30:31.265 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:31.265 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:31.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:31.286 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:31.305+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:31.373 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:31.374 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-16T19:30:31.374 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:31.387 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:31.405+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:31.423 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:31.423 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:31.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:31.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:31.507+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:31.539 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:31.539 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:31.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:31.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.583+0000 7fa1a3fff640 1 -- 172.21.15.172:0/2327067056 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v47) v1 ==== 87+0+0 (secure 0 0 0) 0x7fa1b007a500 con 0x7fa1bc0c93a0 2024-03-16T19:30:31.565 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:30:31.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1a806b3b0 msgr2=0x7fa1a806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:31.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1a806b3b0 0x7fa1a806d870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac007fd0 tx=0x7fa1ac009040 comp rx=0 tx=0).stop 2024-03-16T19:30:31.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1bc0c93a0 msgr2=0x7fa1bc1e51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1bc0c93a0 0x7fa1bc1e51a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa1b002bfe0 tx=0x7fa1b0004730 comp rx=0 tx=0).stop 2024-03-16T19:30:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 shutdown_connections 2024-03-16T19:30:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1bc152a20 0x7fa1bc1ee120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1bc0c93a0 0x7fa1bc1e51a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa1a806b3b0 0x7fa1a806d870 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 --2- 172.21.15.172:0/2327067056 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1bc0c79d0 0x7fa1bc1482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 >> 172.21.15.172:0/2327067056 conn(0x7fa1bc06db70 msgr2=0x7fa1bc06b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 shutdown_connections 2024-03-16T19:30:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:31.586+0000 7fa1c1f4b640 1 -- 172.21.15.172:0/2327067056 wait complete. 2024-03-16T19:30:31.588 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:31.607+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:31.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:31.689 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:31.708+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:31.773 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:31.773 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:31.782 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:31.826 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:31.845+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:31.889 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:31.889 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:31.890 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:31.909+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:31.898 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:31.991 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:32.009+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:32.046 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:32.046 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:32.055 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:32.092 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:32.111+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:32.161 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:32.162 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:32.170 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:32.192 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:32.211+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:32.276 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:32.276 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:32.285 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:32.293 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:32.312+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:32.391 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:32.391 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:32.393 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:32.412+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:32.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:32.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:32.494 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:32.513+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:32.507 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:32.507 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:32.516 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:32.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.543+0000 7f197829d640 1 Processor -- start 2024-03-16T19:30:32.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.543+0000 7f197829d640 1 -- start start 2024-03-16T19:30:32.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.544+0000 7f197829d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970070d20 0x7f19700c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.544+0000 7f197829d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f19700b1fe0 0x7f19700c1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.544+0000 7f197829d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19700c82a0 con 0x7f1970071100 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.544+0000 7f197829d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19700c3260 con 0x7f19700b1fe0 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.544+0000 7f197829d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19700c2a40 con 0x7f1970070d20 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f197683a640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1970071100 0x7f1970151580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39320/0 (socket says 172.21.15.172:39320) 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f197683a640 1 -- 172.21.15.172:0/3630464750 learned_addr learned my addr 172.21.15.172:0/3630464750 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f1976039640 1 --2- 172.21.15.172:0/3630464750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970070d20 0x7f19700c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f1975838640 1 --2- 172.21.15.172:0/3630464750 >> v2:172.21.15.174:3300/0 conn(0x7f19700b1fe0 0x7f19700c1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f1975037640 1 -- 172.21.15.172:0/3630464750 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837106232 0 0) 0x7f19700c82a0 con 0x7f1970071100 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f1975037640 1 -- 172.21.15.172:0/3630464750 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f195c003520 con 0x7f1970071100 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f1976039640 1 -- 172.21.15.172:0/3630464750 >> v2:172.21.15.174:3300/0 conn(0x7f19700b1fe0 msgr2=0x7f19700c1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f1976039640 1 --2- 172.21.15.172:0/3630464750 >> v2:172.21.15.174:3300/0 conn(0x7f19700b1fe0 0x7f19700c1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f1976039640 1 -- 172.21.15.172:0/3630464750 >> v1:172.21.15.172:6789/0 conn(0x7f1970071100 legacy=0x7f1970151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f1976039640 1 -- 172.21.15.172:0/3630464750 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19700c2ce0 con 0x7f1970070d20 2024-03-16T19:30:32.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.545+0000 7f1975838640 1 --2- 172.21.15.172:0/3630464750 >> v2:172.21.15.174:3300/0 conn(0x7f19700b1fe0 0x7f19700c1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:32.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.546+0000 7f1976039640 1 --2- 172.21.15.172:0/3630464750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970070d20 0x7f19700c9650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f196c0097a0 tx=0x7f196c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57ce2c66e252eb70 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.546+0000 7f1975037640 1 -- 172.21.15.172:0/3630464750 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c031830 con 0x7f1970070d20 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.546+0000 7f1975037640 1 -- 172.21.15.172:0/3630464750 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196c031990 con 0x7f1970070d20 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.546+0000 7f1975037640 1 -- 172.21.15.172:0/3630464750 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c031d00 con 0x7f1970070d20 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.546+0000 7f197829d640 1 -- 172.21.15.172:0/3630464750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970070d20 msgr2=0x7f19700c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.546+0000 7f197829d640 1 --2- 172.21.15.172:0/3630464750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970070d20 0x7f19700c9650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f196c0097a0 tx=0x7f196c02b8d0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.547+0000 7f197829d640 1 -- 172.21.15.172:0/3630464750 shutdown_connections 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.547+0000 7f197829d640 1 --2- 172.21.15.172:0/3630464750 >> v2:172.21.15.174:3300/0 conn(0x7f19700b1fe0 0x7f19700c1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.547+0000 7f197829d640 1 --2- 172.21.15.172:0/3630464750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970070d20 0x7f19700c9650 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.547+0000 7f197829d640 1 -- 172.21.15.172:0/3630464750 >> 172.21.15.172:0/3630464750 conn(0x7f19700c8b30 msgr2=0x7f19700c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.547+0000 7f197829d640 1 -- 172.21.15.172:0/3630464750 shutdown_connections 2024-03-16T19:30:32.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.547+0000 7f197829d640 1 -- 172.21.15.172:0/3630464750 wait complete. 2024-03-16T19:30:32.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f197829d640 1 Processor -- start 2024-03-16T19:30:32.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f197829d640 1 -- start start 2024-03-16T19:30:32.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f197829d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1970070d20 0x7f19701eeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f197829d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1970071100 0x7f1970141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f197829d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970149320 0x7f1970149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f197829d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1970142fd0 con 0x7f1970070d20 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f197829d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701432b0 con 0x7f1970071100 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f197829d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1970143590 con 0x7f1970149320 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f1976039640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1970070d20 0x7f19701eeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f197683a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970149320 0x7f1970149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f1976039640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1970070d20 0x7f19701eeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47588/0 (socket says 172.21.15.172:47588) 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f1976039640 1 -- 172.21.15.172:0/2981250664 learned_addr learned my addr 172.21.15.172:0/2981250664 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.548+0000 7f1975838640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1970071100 0x7f1970141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f197683a640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1970071100 msgr2=0x7f1970141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f197683a640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1970071100 0x7f1970141c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f197683a640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1970070d20 msgr2=0x7f19701eeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f197683a640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1970070d20 0x7f19701eeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f197683a640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960009400 con 0x7f1970149320 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f197683a640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970149320 0x7f1970149be0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f196800ca80 tx=0x7f196800cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196800ac30 con 0x7f1970149320 2024-03-16T19:30:32.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196800ad90 con 0x7f1970149320 2024-03-16T19:30:32.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196c009400 con 0x7f1970149320 2024-03-16T19:30:32.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1968012600 con 0x7f1970149320 2024-03-16T19:30:32.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.549+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1970143f60 con 0x7f1970149320 2024-03-16T19:30:32.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.550+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f197014f690 con 0x7f1970149320 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.551+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1968012760 con 0x7f1970149320 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.551+0000 7f1966ffd640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f195406b3b0 0x7f195406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.551+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f1968089cc0 con 0x7f1970149320 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.551+0000 7f1976039640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f195406b3b0 0x7f195406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.551+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v47) v1 ==== 54+0+16 (secure 0 0 0) 0x7f196808a110 con 0x7f1970149320 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.551+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f19700b1fe0 0x7f197014e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.551+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19700c7c50 con 0x7f19700b1fe0 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.551+0000 7f1975838640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f19700b1fe0 0x7f197014e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.551+0000 7f1976039640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f195406b3b0 0x7f195406d870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f196c02b910 tx=0x7f196c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.552+0000 7f1975838640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f19700b1fe0 0x7f197014e2f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.552+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1960009d60 con 0x7f19700b1fe0 2024-03-16T19:30:32.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.567+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19700c1590 con 0x7f19700b1fe0 2024-03-16T19:30:32.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.568+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1960009280 con 0x7f19700b1fe0 2024-03-16T19:30:32.550 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:32.551 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:32.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.569+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f1970144990 0x7f19701f1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.569+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1970149100 con 0x7f1970144990 2024-03-16T19:30:32.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.569+0000 7f1976039640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f1970144990 0x7f19701f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.569+0000 7f1976039640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f1970144990 0x7f19701f1800 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.551 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.570+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1970149100 con 0x7f1970144990 2024-03-16T19:30:32.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.585+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f197014ec10 con 0x7f1970144990 2024-03-16T19:30:32.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.587+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f197014ec10 con 0x7f1970144990 2024-03-16T19:30:32.568 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:32.569 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:32.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.587+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f19701f2250 0x7f1970061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.587+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19701f2690 con 0x7f19701f2250 2024-03-16T19:30:32.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.587+0000 7f1975838640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f19701f2250 0x7f1970061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.587+0000 7f1975838640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f19701f2250 0x7f1970061930 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.588+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19600177a0 con 0x7f19701f2250 2024-03-16T19:30:32.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.602+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19700c2610 con 0x7f19701f2250 2024-03-16T19:30:32.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.604+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19700c2610 con 0x7f19701f2250 2024-03-16T19:30:32.585 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:32.586 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:32.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.604+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1970062380 0x7f1970150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.604+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19700b1b30 con 0x7f1970062380 2024-03-16T19:30:32.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.604+0000 7f197683a640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1970062380 0x7f1970150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.604+0000 7f197683a640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1970062380 0x7f1970150070 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.605+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f19700b1b30 con 0x7f1970062380 2024-03-16T19:30:32.600 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:32.619+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:32.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.619+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f19701505b0 con 0x7f1970062380 2024-03-16T19:30:32.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.621+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f19701505b0 con 0x7f1970062380 2024-03-16T19:30:32.602 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.621+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f1970150e00 0x7f197002bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.621+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1970204290 con 0x7f1970150e00 2024-03-16T19:30:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.621+0000 7f1976039640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f1970150e00 0x7f197002bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.621+0000 7f1976039640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f1970150e00 0x7f197002bde0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.622+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1970204290 con 0x7f1970150e00 2024-03-16T19:30:32.617 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.636+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f197002c320 con 0x7f1970150e00 2024-03-16T19:30:32.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.638+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f197002c320 con 0x7f1970150e00 2024-03-16T19:30:32.619 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:32.620 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:32.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.638+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f197002ca00 0x7f197002ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.638+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1970207590 con 0x7f197002ca00 2024-03-16T19:30:32.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.638+0000 7f1975838640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f197002ca00 0x7f197002ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.639+0000 7f1975838640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f197002ca00 0x7f197002ce60 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.621 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.640+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1970207590 con 0x7f197002ca00 2024-03-16T19:30:32.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.654+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1970202b80 con 0x7f197002ca00 2024-03-16T19:30:32.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.656+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1970202b80 con 0x7f197002ca00 2024-03-16T19:30:32.637 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:32.637 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:32.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.656+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1970203260 0x7f19702036c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.656+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1970203c00 con 0x7f1970203260 2024-03-16T19:30:32.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.656+0000 7f197683a640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1970203260 0x7f19702036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.656+0000 7f197683a640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1970203260 0x7f19702036c0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.657+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1970203c00 con 0x7f1970203260 2024-03-16T19:30:32.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.671+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1970203c00 con 0x7f1970203260 2024-03-16T19:30:32.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.674+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1970203c00 con 0x7f1970203260 2024-03-16T19:30:32.655 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:32.655 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:32.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.674+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1970208420 0x7f197020a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:32.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.674+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f197020d5e0 con 0x7f1970208420 2024-03-16T19:30:32.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.674+0000 7f1976039640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1970208420 0x7f197020a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:32.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.674+0000 7f1976039640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1970208420 0x7f197020a810 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:32.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.675+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f197020d5e0 con 0x7f1970208420 2024-03-16T19:30:32.665 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:32.665 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:32.665 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:32.665 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:32.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.689+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f197020ad50 con 0x7f1970208420 2024-03-16T19:30:32.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.690+0000 7f1966ffd640 1 -- 172.21.15.172:0/2981250664 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f197020ad50 con 0x7f1970208420 2024-03-16T19:30:32.672 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:32.673 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:32.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f19700b1fe0 msgr2=0x7f197014e2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f19700b1fe0 0x7f197014e2f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f1970144990 msgr2=0x7f19701f1800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f1970144990 0x7f19701f1800 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f19701f2250 msgr2=0x7f1970061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f19701f2250 0x7f1970061930 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1970062380 msgr2=0x7f1970150070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f1970062380 0x7f1970150070 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f1970150e00 msgr2=0x7f197002bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197683a640 1 -- 172.21.15.172:0/2981250664 reap_dead start 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f1970150e00 0x7f197002bde0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197683a640 1 -- 172.21.15.172:0/2981250664 reap_dead start 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f197002ca00 msgr2=0x7f197002ce60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f197002ca00 0x7f197002ce60 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1970203260 msgr2=0x7f19702036c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f1970203260 0x7f19702036c0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1970208420 msgr2=0x7f197020a810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f1970208420 0x7f197020a810 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f195406b3b0 msgr2=0x7f195406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f195406b3b0 0x7f195406d870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f196c02b910 tx=0x7f196c03c040 comp rx=0 tx=0).stop 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197683a640 1 -- 172.21.15.172:0/2981250664 reap_dead start 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970149320 msgr2=0x7f1970149be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970149320 0x7f1970149be0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f196800ca80 tx=0x7f196800cf40 comp rx=0 tx=0).stop 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 shutdown_connections 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 --2- 172.21.15.172:0/2981250664 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1970149320 0x7f1970149be0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 >> 172.21.15.172:0/2981250664 conn(0x7f19700c8b30 msgr2=0x7f197014ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 shutdown_connections 2024-03-16T19:30:32.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:32.691+0000 7f197829d640 1 -- 172.21.15.172:0/2981250664 wait complete. 2024-03-16T19:30:32.676 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:32.696 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:32.715+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:32.797 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:32.816+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:32.827 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:32.827 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:32.827 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:32.827 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:32.827 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:32.837 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:32.897 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:32.916+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:32.979 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:32.979 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:32.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.640856177000003, 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:32.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:30:32.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:30:32.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:30:32.984 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.640832004000004, 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:32.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:30:32.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:30:32.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:32.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:32.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.640826177000001, 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:32.990 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:30:32.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:30:32.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.640822264999997, 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:30:32.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.640816282999999, 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:30:33.000 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:33.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:33.005 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.640812617000002, 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.640801957000001, 2024-03-16T19:30:33.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:30:33.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.640798068000002, 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:30:33.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:30:33.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.017 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.637281377000001, 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:33.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.637210942000003, 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:33.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.636828594999997, 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.635062312000002, 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.633249390000003, 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:30:33.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.035 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.632393641, 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:30:33.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:30:33.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:33.040 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 56.626532875000002, 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:30:33.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.193981667000003, 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.044 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:30:33.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:30:33.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:30:33.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:30:33.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 51.626487961000002, 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.049 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:30:33.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 50.118875709999998, 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:30:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:30:33.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.622012484999999, 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:30:33.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:30:33.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 7.6214765690000004, 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:30:33.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.066 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.067 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.067 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.067 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.067 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:33.068 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:33.068 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:33.087+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:33.168 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:33.187+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:33.215 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:33.216 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:33.216 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:33.216 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:33.225 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:33.270 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:33.289+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:33.371 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:33.390+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:33.372 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:33.372 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:33.372 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:33.372 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:33.372 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:33.380 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:33.471 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:33.490+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:33.489 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:33.489 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:33.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:33.489 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:33.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.489 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:33.489 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:33.489 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 77.277962263999996, 2024-03-16T19:30:33.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:30:33.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:30:33.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:30:33.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.901266716999999, 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.495 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.125290273000005, 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.499 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:30:33.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:30:33.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:30:33.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.125205859999994, 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.504 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.125198503999997, 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:30:33.509 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:30:33.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:33.514 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.932659311999998, 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:30:33.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.932653505000005, 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.932535372000004, 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:30:33.524 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.932440174999996, 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.932436163999995, 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:30:33.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:30:33.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.539 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.51350429, 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:30:33.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.544 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.513454921000005, 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:30:33.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.549 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.513376089000005, 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:30:33.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:30:33.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.554 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.513372877999998, 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.555 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:30:33.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.513369925000006, 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.560 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:30:33.564 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.513254950999993, 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.565 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:30:33.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.513251599, 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:30:33.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:33.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.513248482999998, 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.575 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.513188013999994, 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.580 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:30:33.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:30:33.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 74.904507964000004, 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:33.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:33.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:30:33.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:33.590 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:33.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:30:33.592 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:33.611+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:33.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:33.692 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:33.711+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:33.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.725+0000 7ff7dfe76640 1 Processor -- start 2024-03-16T19:30:33.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.726+0000 7ff7dfe76640 1 -- start start 2024-03-16T19:30:33.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.726+0000 7ff7dfe76640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff7d80b5f10 0x7ff7d806b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:33.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.726+0000 7ff7dfe76640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d806b110 con 0x7ff7d8154e50 2024-03-16T19:30:33.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.726+0000 7ff7dfe76640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d815b8d0 con 0x7ff7d80caf40 2024-03-16T19:30:33.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.726+0000 7ff7dfe76640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d815ad50 con 0x7ff7d80b5f10 2024-03-16T19:30:33.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.727+0000 7ff7ddc12640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff7d8154e50 0x7ff7d80cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39324/0 (socket says 172.21.15.172:39324) 2024-03-16T19:30:33.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.727+0000 7ff7dd411640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff7d80b5f10 0x7ff7d806b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:33.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.727+0000 7ff7ddc12640 1 -- 172.21.15.172:0/2376508665 learned_addr learned my addr 172.21.15.172:0/2376508665 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:33.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.727+0000 7ff7dd411640 1 -- 172.21.15.172:0/2376508665 >> v1:172.21.15.174:6789/0 conn(0x7ff7d80caf40 legacy=0x7ff7d8165260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:33.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.727+0000 7ff7dd411640 1 -- 172.21.15.172:0/2376508665 >> v1:172.21.15.172:6789/0 conn(0x7ff7d8154e50 legacy=0x7ff7d80cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:33.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.727+0000 7ff7dcc10640 1 -- 172.21.15.172:0/2376508665 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007623428 0 0) 0x7ff7d806b110 con 0x7ff7d8154e50 2024-03-16T19:30:33.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.727+0000 7ff7dd411640 1 -- 172.21.15.172:0/2376508665 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d815b030 con 0x7ff7d80b5f10 2024-03-16T19:30:33.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.728+0000 7ff7dd411640 1 --2- 172.21.15.172:0/2376508665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80b5f10 0x7ff7d806b5a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40097a0 tx=0x7ff7d402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5cabb7fcad477 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:33.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.728+0000 7ff7dcc10640 1 -- 172.21.15.172:0/2376508665 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d402f810 con 0x7ff7d80b5f10 2024-03-16T19:30:33.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.728+0000 7ff7dcc10640 1 -- 172.21.15.172:0/2376508665 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d402f970 con 0x7ff7d80b5f10 2024-03-16T19:30:33.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.728+0000 7ff7dcc10640 1 -- 172.21.15.172:0/2376508665 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d402fd00 con 0x7ff7d80b5f10 2024-03-16T19:30:33.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.728+0000 7ff7dfe76640 1 -- 172.21.15.172:0/2376508665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80b5f10 msgr2=0x7ff7d806b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:33.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.728+0000 7ff7dfe76640 1 --2- 172.21.15.172:0/2376508665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80b5f10 0x7ff7d806b5a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40097a0 tx=0x7ff7d402b9f0 comp rx=0 tx=0).stop 2024-03-16T19:30:33.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.728+0000 7ff7dfe76640 1 -- 172.21.15.172:0/2376508665 shutdown_connections 2024-03-16T19:30:33.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.728+0000 7ff7dfe76640 1 --2- 172.21.15.172:0/2376508665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80b5f10 0x7ff7d806b5a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:33.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.728+0000 7ff7dfe76640 1 -- 172.21.15.172:0/2376508665 >> 172.21.15.172:0/2376508665 conn(0x7ff7d8066c20 msgr2=0x7ff7d8067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:33.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.729+0000 7ff7dfe76640 1 -- 172.21.15.172:0/2376508665 shutdown_connections 2024-03-16T19:30:33.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.729+0000 7ff7dfe76640 1 -- 172.21.15.172:0/2376508665 wait complete. 2024-03-16T19:30:33.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dfe76640 1 Processor -- start 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dfe76640 1 -- start start 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dfe76640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7d80b5f10 0x7ff7d8157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dfe76640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80caf40 0x7ff7d814a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dfe76640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7d8154e50 0x7ff7d81f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dfe76640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d814cc40 con 0x7ff7d80caf40 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dfe76640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d814adc0 con 0x7ff7d8154e50 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dfe76640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d814d6e0 con 0x7ff7d80b5f10 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dd411640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80caf40 0x7ff7d814a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dd411640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80caf40 0x7ff7d814a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47598/0 (socket says 172.21.15.172:47598) 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.730+0000 7ff7dd411640 1 -- 172.21.15.172:0/3606464732 learned_addr learned my addr 172.21.15.172:0/3606464732 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7ddc12640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7d80b5f10 0x7ff7d8157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7de413640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7d8154e50 0x7ff7d81f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7dd411640 1 -- 172.21.15.172:0/3606464732 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7d80b5f10 msgr2=0x7ff7d8157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:33.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7dd411640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7d80b5f10 0x7ff7d8157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7dd411640 1 -- 172.21.15.172:0/3606464732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7d8154e50 msgr2=0x7ff7d81f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7dd411640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7d8154e50 0x7ff7d81f45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7dd411640 1 -- 172.21.15.172:0/3606464732 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d4009400 con 0x7ff7d80caf40 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7dd411640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80caf40 0x7ff7d814a880 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40098c0 tx=0x7ff7d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7ce7fc640 1 -- 172.21.15.172:0/3606464732 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d403c040 con 0x7ff7d80caf40 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7ce7fc640 1 -- 172.21.15.172:0/3606464732 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d40379f0 con 0x7ff7d80caf40 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d815ab20 con 0x7ff7d80caf40 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7ce7fc640 1 -- 172.21.15.172:0/3606464732 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4037da0 con 0x7ff7d80caf40 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.731+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d81f3e20 con 0x7ff7d80caf40 2024-03-16T19:30:33.714 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:33.715 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:33.715 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:33.715 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:33.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.732+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d80c9ab0 con 0x7ff7d80caf40 2024-03-16T19:30:33.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.733+0000 7ff7ce7fc640 1 -- 172.21.15.172:0/3606464732 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff7d404b3b0 con 0x7ff7d80caf40 2024-03-16T19:30:33.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.733+0000 7ff7ce7fc640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff7bc06b3b0 0x7ff7bc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:33.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.733+0000 7ff7ce7fc640 1 -- 172.21.15.172:0/3606464732 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5828+0+0 (secure 0 0 0) 0x7ff7d40abc90 con 0x7ff7d80caf40 2024-03-16T19:30:33.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.733+0000 7ff7ddc12640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff7bc06b3b0 0x7ff7bc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:33.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.734+0000 7ff7ddc12640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff7bc06b3b0 0x7ff7bc06d870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8000f60 tx=0x7ff7c8007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:33.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.738+0000 7ff7ce7fc640 1 -- 172.21.15.172:0/3606464732 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7d40769b0 con 0x7ff7d80caf40 2024-03-16T19:30:33.723 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:33.808 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:33.827+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:33.873 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:33.874 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:33.874 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:33.874 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:33.874 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:33.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:33.894 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:33.913+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:33.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:33.928+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff7d8075a70 con 0x7ff7d80caf40 2024-03-16T19:30:33.994 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:34.013+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:34.017 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:34.017 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:34.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:34.017 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:34.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.017 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:30:34.017 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.442724264999995, 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:34.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:30:34.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:30:34.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.021 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.652498093999995, 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:30:34.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:30:34.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.025 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.651737858999994, 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:34.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:30:34.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:30:34.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.650923903999995, 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:30:34.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:30:34.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.033 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.650283978999994, 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:30:34.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:30:34.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:30:34.037 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.649625881999995, 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:34.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.649093307000001, 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:30:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:30:34.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:30:34.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.590921403999999, 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.045 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:34.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:30:34.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.590319210000004, 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.049 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:30:34.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:30:34.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.589676264999994, 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.053 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:30:34.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:30:34.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.589172304000002, 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.057 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:30:34.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:30:34.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:30:34.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 75.431026037999999, 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.061 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:30:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 75.430495902000004, 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.678526642999998, 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:30:34.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.678502107, 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:30:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:30:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:30:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:30:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.678497577000002, 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:30:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.678487765, 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:30:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:30:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:30:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.678483911999997, 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.678480596, 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:30:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:30:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.677285013000002, 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:30:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:30:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.102 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:34.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:34.104 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:34.122+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:34.204 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:34.223+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:34.211 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:34.211 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:34.211 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:34.211 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:34.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:34.305 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:34.324+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:34.327 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:34.327 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:34.327 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:34.327 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:34.327 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:34.336 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:34.406 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:34.425+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:34.443 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.105332148000002, 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:30:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.105315034, 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:30:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.105307754000002, 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:30:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:30:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:30:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.105293887000002, 2024-03-16T19:30:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:30:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:30:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.104621051999999, 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:30:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.102322802000003, 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:30:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:30:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.102277700999998, 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.102249645999997, 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.102228232000002, 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:30:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:30:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.102205881000003, 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:30:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:30:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.102185482000003, 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:30:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.102079490000001, 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.100287778999999, 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.100240278000001, 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:30:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.100001026999998, 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:30:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:30:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.091502882, 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:30:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:30:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.090163005999997, 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.087172023999999, 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.0865412219999993, 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:30:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:30:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:47 2.6 e40/18 ondisk, result = 0)", 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.0850588129999998, 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017677, 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377611+0000", 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9599999999999998e-07 2024-03-16T19:30:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377618+0000", 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9990000000000002e-06 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377626+0000", 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4249999999999998e-06 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377632+0000", 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377707+0000", 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4506999999999997e-05 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377714+0000", 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5819999999999998e-06 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377787+0000", 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3232000000000006e-05 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:34.529 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:34.530 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:34.531 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:34.548+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:34.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.590+0000 7ff7ce7fc640 1 -- 172.21.15.172:0/3606464732 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v48) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff7d4047090 con 0x7ff7d80caf40 2024-03-16T19:30:34.571 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:30:34.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.592+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff7bc06b3b0 msgr2=0x7ff7bc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.592+0000 7ff7dfe76640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff7bc06b3b0 0x7ff7bc06d870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8000f60 tx=0x7ff7c8007e40 comp rx=0 tx=0).stop 2024-03-16T19:30:34.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.592+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80caf40 msgr2=0x7ff7d814a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.592+0000 7ff7dfe76640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80caf40 0x7ff7d814a880 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40098c0 tx=0x7ff7d4004730 comp rx=0 tx=0).stop 2024-03-16T19:30:34.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.592+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 shutdown_connections 2024-03-16T19:30:34.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.592+0000 7ff7dfe76640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7d8154e50 0x7ff7d81f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.592+0000 7ff7dfe76640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80caf40 0x7ff7d814a880 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.592+0000 7ff7dfe76640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff7bc06b3b0 0x7ff7bc06d870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.592+0000 7ff7dfe76640 1 --2- 172.21.15.172:0/3606464732 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7d80b5f10 0x7ff7d8157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.593+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 >> 172.21.15.172:0/3606464732 conn(0x7ff7d8066c20 msgr2=0x7ff7d8067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:34.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.593+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 shutdown_connections 2024-03-16T19:30:34.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.593+0000 7ff7dfe76640 1 -- 172.21.15.172:0/3606464732 wait complete. 2024-03-16T19:30:34.630 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:34.649+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:34.679 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:34.680 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:34.680 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:34.680 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:34.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:34.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:34.732 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:34.750+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.803+0000 7f9b58e4e640 1 Processor -- start 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.803+0000 7f9b58e4e640 1 -- start start 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b58e4e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9b54154ed0 0x7f9b5415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b58e4e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b540b4270 con 0x7f9b540b3630 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b58e4e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b541552b0 con 0x7f9b540b4700 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b58e4e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b5415b4b0 con 0x7f9b54154ed0 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b51d9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9b540b3630 0x7f9b540cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39340/0 (socket says 172.21.15.172:39340) 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b51d9b640 1 -- 172.21.15.172:0/3245680919 learned_addr learned my addr 172.21.15.172:0/3245680919 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b5259c640 1 --2- 172.21.15.172:0/3245680919 >> v2:172.21.15.174:3300/0 conn(0x7f9b54154ed0 0x7f9b5415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b5159a640 1 -- 172.21.15.172:0/3245680919 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177439208 0 0) 0x7f9b540b4270 con 0x7f9b540b3630 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b5159a640 1 -- 172.21.15.172:0/3245680919 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3c003520 con 0x7f9b540b3630 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b5259c640 1 -- 172.21.15.172:0/3245680919 >> v1:172.21.15.174:6789/0 conn(0x7f9b540b4700 legacy=0x7f9b541654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b5159a640 1 -- 172.21.15.172:0/3245680919 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095361287 0 0) 0x7f9b541552b0 con 0x7f9b540b4700 2024-03-16T19:30:34.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b5259c640 1 -- 172.21.15.172:0/3245680919 >> v1:172.21.15.172:6789/0 conn(0x7f9b540b3630 legacy=0x7f9b540cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.804+0000 7f9b5259c640 1 -- 172.21.15.172:0/3245680919 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5415b790 con 0x7f9b54154ed0 2024-03-16T19:30:34.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.805+0000 7f9b5259c640 1 --2- 172.21.15.172:0/3245680919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b54154ed0 0x7f9b5415ab40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9b440097a0 tx=0x7f9b4402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab097eb74c9b931 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.805+0000 7f9b5159a640 1 -- 172.21.15.172:0/3245680919 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b44030610 con 0x7f9b54154ed0 2024-03-16T19:30:34.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.805+0000 7f9b5159a640 1 -- 172.21.15.172:0/3245680919 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b44030770 con 0x7f9b54154ed0 2024-03-16T19:30:34.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.805+0000 7f9b5159a640 1 -- 172.21.15.172:0/3245680919 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b44030b00 con 0x7f9b54154ed0 2024-03-16T19:30:34.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.806+0000 7f9b58e4e640 1 -- 172.21.15.172:0/3245680919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b54154ed0 msgr2=0x7f9b5415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.806+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/3245680919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b54154ed0 0x7f9b5415ab40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9b440097a0 tx=0x7f9b4402b960 comp rx=0 tx=0).stop 2024-03-16T19:30:34.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.806+0000 7f9b58e4e640 1 -- 172.21.15.172:0/3245680919 shutdown_connections 2024-03-16T19:30:34.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.806+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/3245680919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b54154ed0 0x7f9b5415ab40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.806+0000 7f9b58e4e640 1 -- 172.21.15.172:0/3245680919 >> 172.21.15.172:0/3245680919 conn(0x7f9b5406bf60 msgr2=0x7f9b5406d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:34.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.806+0000 7f9b58e4e640 1 -- 172.21.15.172:0/3245680919 shutdown_connections 2024-03-16T19:30:34.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.806+0000 7f9b58e4e640 1 -- 172.21.15.172:0/3245680919 wait complete. 2024-03-16T19:30:34.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.807+0000 7f9b58e4e640 1 Processor -- start 2024-03-16T19:30:34.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.807+0000 7f9b58e4e640 1 -- start start 2024-03-16T19:30:34.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b58e4e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b540b3630 0x7f9b54156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b58e4e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b540b4700 0x7f9b54155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b58e4e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b54154ed0 0x7f9b540c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b58e4e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b54156e40 con 0x7f9b540b4700 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b58e4e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b541f01a0 con 0x7f9b54154ed0 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b58e4e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b541f0fe0 con 0x7f9b540b3630 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b51d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b540b4700 0x7f9b54155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b51d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b540b4700 0x7f9b54155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47604/0 (socket says 172.21.15.172:47604) 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b51d9b640 1 -- 172.21.15.172:0/16428078 learned_addr learned my addr 172.21.15.172:0/16428078 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b52d9d640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b54154ed0 0x7f9b540c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.808+0000 7f9b5259c640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b540b3630 0x7f9b54156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b51d9b640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b540b3630 msgr2=0x7f9b54156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b51d9b640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b540b3630 0x7f9b54156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b51d9b640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b54154ed0 msgr2=0x7f9b540c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b51d9b640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b54154ed0 0x7f9b540c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b51d9b640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b44009400 con 0x7f9b540b4700 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b51d9b640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b540b4700 0x7f9b54155700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c460 tx=0x7f9b4800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b48005ce0 con 0x7f9b540b4700 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b48005e40 con 0x7f9b540b4700 2024-03-16T19:30:34.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b541f03d0 con 0x7f9b540b4700 2024-03-16T19:30:34.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b48007290 con 0x7f9b540b4700 2024-03-16T19:30:34.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.809+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b541f0990 con 0x7f9b540b4700 2024-03-16T19:30:34.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.810+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9b541f1120 con 0x7f9b540b4700 2024-03-16T19:30:34.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.810+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9b480073f0 con 0x7f9b540b4700 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.811+0000 7f9b42ffd640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9b3406b3e0 0x7f9b3406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.811+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f9b480890c0 con 0x7f9b540b4700 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.811+0000 7f9b5259c640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9b3406b3e0 0x7f9b3406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.811+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v48) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9b541f1120 con 0x7f9b540b4700 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.811+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9b540c4c00 0x7f9b540d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.811+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b541f1120 con 0x7f9b540c4c00 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.811+0000 7f9b52d9d640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9b540c4c00 0x7f9b540d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.811+0000 7f9b5259c640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9b3406b3e0 0x7f9b3406d8a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9b44036000 tx=0x7f9b44008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.812+0000 7f9b52d9d640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9b540c4c00 0x7f9b540d0240 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.812+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b541f1120 con 0x7f9b540c4c00 2024-03-16T19:30:34.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.842+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b54068d20 con 0x7f9b540c4c00 2024-03-16T19:30:34.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.843+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b54068d20 con 0x7f9b540c4c00 2024-03-16T19:30:34.825 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:34.825 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:34.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.844+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9b5415edc0 0x7f9b54159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.844+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b540cbbe0 con 0x7f9b5415edc0 2024-03-16T19:30:34.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.844+0000 7f9b5259c640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9b5415edc0 0x7f9b54159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.844+0000 7f9b5259c640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9b5415edc0 0x7f9b54159c00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.845+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b540cbbe0 con 0x7f9b5415edc0 2024-03-16T19:30:34.831 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:34.850+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:34.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.869+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b540b3f00 con 0x7f9b5415edc0 2024-03-16T19:30:34.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.871+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b4403f040 con 0x7f9b5415edc0 2024-03-16T19:30:34.853 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:34.854 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:34.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.872+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9b541f3c20 0x7f9b541f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.872+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b541f4540 con 0x7f9b541f3c20 2024-03-16T19:30:34.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.872+0000 7f9b51d9b640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9b541f3c20 0x7f9b541f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.872+0000 7f9b51d9b640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9b541f3c20 0x7f9b541f4000 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.873+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b541f4540 con 0x7f9b541f3c20 2024-03-16T19:30:34.858 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:34.858 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:34.858 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:34.858 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:34.859 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:34.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:34.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.887+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b541533e0 con 0x7f9b541f3c20 2024-03-16T19:30:34.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.889+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b541533e0 con 0x7f9b541f3c20 2024-03-16T19:30:34.870 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:34.871 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:34.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.889+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9b54062200 0x7f9b540cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.889+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b541f0e00 con 0x7f9b54062200 2024-03-16T19:30:34.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.889+0000 7f9b52d9d640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9b54062200 0x7f9b540cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.889+0000 7f9b52d9d640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9b54062200 0x7f9b540cc760 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.890+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b4c0194f0 con 0x7f9b54062200 2024-03-16T19:30:34.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.904+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b54062870 con 0x7f9b54062200 2024-03-16T19:30:34.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.906+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b54062870 con 0x7f9b54062200 2024-03-16T19:30:34.887 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:34.888 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:34.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.906+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9b540cd1b0 0x7f9b5402bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.906+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b54153100 con 0x7f9b540cd1b0 2024-03-16T19:30:34.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.906+0000 7f9b5259c640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9b540cd1b0 0x7f9b5402bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.907+0000 7f9b5259c640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9b540cd1b0 0x7f9b5402bde0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.907+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b54153100 con 0x7f9b540cd1b0 2024-03-16T19:30:34.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.922+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b54153100 con 0x7f9b540cd1b0 2024-03-16T19:30:34.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.923+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b54153100 con 0x7f9b540cd1b0 2024-03-16T19:30:34.905 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:34.905 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:34.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.923+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9b5402c830 0x7f9b5402ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.923+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b540cd630 con 0x7f9b5402c830 2024-03-16T19:30:34.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.924+0000 7f9b51d9b640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9b5402c830 0x7f9b5402ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.924+0000 7f9b51d9b640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9b5402c830 0x7f9b5402ccb0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.925+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b540cd630 con 0x7f9b5402c830 2024-03-16T19:30:34.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.939+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b540cd630 con 0x7f9b5402c830 2024-03-16T19:30:34.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.940+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b540cd630 con 0x7f9b5402c830 2024-03-16T19:30:34.922 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:34.922 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:34.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.941+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9b54206fb0 0x7f9b542093a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.941+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b540cd630 con 0x7f9b54206fb0 2024-03-16T19:30:34.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.941+0000 7f9b52d9d640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9b54206fb0 0x7f9b542093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.941+0000 7f9b52d9d640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9b54206fb0 0x7f9b542093a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.942+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b540cd630 con 0x7f9b54206fb0 2024-03-16T19:30:34.932 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:34.951+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:34.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.956+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b540cd630 con 0x7f9b54206fb0 2024-03-16T19:30:34.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.957+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b540cd630 con 0x7f9b54206fb0 2024-03-16T19:30:34.939 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:34.940 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:34.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.958+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9b54209df0 0x7f9b5420c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:34.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.958+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b5420f110 con 0x7f9b54209df0 2024-03-16T19:30:34.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.958+0000 7f9b5259c640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9b54209df0 0x7f9b5420c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:34.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.958+0000 7f9b5259c640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9b54209df0 0x7f9b5420c1e0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:34.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.959+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b5420f110 con 0x7f9b54209df0 2024-03-16T19:30:34.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.973+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9b5420c7a0 con 0x7f9b54209df0 2024-03-16T19:30:34.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b42ffd640 1 -- 172.21.15.172:0/16428078 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9b5420c7a0 con 0x7f9b54209df0 2024-03-16T19:30:34.957 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:34.959 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:34.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9b540c4c00 msgr2=0x7f9b540d0240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9b540c4c00 0x7f9b540d0240 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9b5415edc0 msgr2=0x7f9b54159c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9b5415edc0 0x7f9b54159c00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9b541f3c20 msgr2=0x7f9b541f4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f9b541f3c20 0x7f9b541f4000 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9b54062200 msgr2=0x7f9b540cc760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9b54062200 0x7f9b540cc760 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9b540cd1b0 msgr2=0x7f9b5402bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9b540cd1b0 0x7f9b5402bde0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b52d9d640 1 -- 172.21.15.172:0/16428078 reap_dead start 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9b5402c830 msgr2=0x7f9b5402ccb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9b5402c830 0x7f9b5402ccb0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b52d9d640 1 -- 172.21.15.172:0/16428078 reap_dead start 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9b54206fb0 msgr2=0x7f9b542093a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9b54206fb0 0x7f9b542093a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9b54209df0 msgr2=0x7f9b5420c1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9b54209df0 0x7f9b5420c1e0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b52d9d640 1 -- 172.21.15.172:0/16428078 reap_dead start 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.976+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9b3406b3e0 msgr2=0x7f9b3406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9b3406b3e0 0x7f9b3406d8a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9b44036000 tx=0x7f9b44008030 comp rx=0 tx=0).stop 2024-03-16T19:30:34.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b540b4700 msgr2=0x7f9b54155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:34.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b540b4700 0x7f9b54155700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c460 tx=0x7f9b4800c920 comp rx=0 tx=0).stop 2024-03-16T19:30:34.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 shutdown_connections 2024-03-16T19:30:34.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f9b54209df0 0x7f9b5420c1e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b540b4700 0x7f9b54155700 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 --2- 172.21.15.172:0/16428078 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9b3406b3e0 0x7f9b3406d8a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:34.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 >> 172.21.15.172:0/16428078 conn(0x7f9b5406bf60 msgr2=0x7f9b54153ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:34.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 shutdown_connections 2024-03-16T19:30:34.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:34.977+0000 7f9b58e4e640 1 -- 172.21.15.172:0/16428078 wait complete. 2024-03-16T19:30:34.977 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:34.977 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:34.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:35.032 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:35.051+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:35.133 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:35.152+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:35.134 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:35.134 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:35.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:35.235 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:35.254+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:35.290 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:35.291 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:35.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:35.340 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:35.359+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:35.437 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:35.456+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:35.445 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:35.446 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:35.455 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:35.538 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:35.557+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:35.562 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:35.562 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:35.571 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:35.638 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:35.657+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:35.677 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:35.677 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:35.686 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:35.770 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:35.789+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:35.840 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:35.840 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:35.840 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:35.859+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:35.848 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:35.940 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:35.959+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:35.995 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:35.995 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:36.004 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:36.041 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:36.060+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:36.141 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:36.160+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:36.150 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:36.150 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:36.158 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:36.242 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:36.261+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:36.320 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:36.320 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:36.329 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:36.344 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:36.363+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:36.383 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:30:36.384 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-16T19:30:36.445 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:36.464+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:36.475 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:36.475 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:36.484 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:36.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.516+0000 7f199b4d4640 1 Processor -- start 2024-03-16T19:30:36.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.516+0000 7f199b4d4640 1 -- start start 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f199b4d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f199406f4f0 0x7f1994166280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f199b4d4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199406e010 con 0x7f1994158150 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f199b4d4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199406f0f0 con 0x7f199406c500 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f199b4d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19940c8f20 con 0x7f199406f4f0 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f1999270640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1994158150 0x7f199406e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39350/0 (socket says 172.21.15.172:39350) 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f1999270640 1 -- 172.21.15.172:0/3906850619 learned_addr learned my addr 172.21.15.172:0/3906850619 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f1999a71640 1 --2- 172.21.15.172:0/3906850619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f199406f4f0 0x7f1994166280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f198bfff640 1 -- 172.21.15.172:0/3906850619 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272759715 0 0) 0x7f199406e010 con 0x7f1994158150 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f198bfff640 1 -- 172.21.15.172:0/3906850619 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1980003520 con 0x7f1994158150 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f198bfff640 1 -- 172.21.15.172:0/3906850619 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785937444 0 0) 0x7f199406f0f0 con 0x7f199406c500 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f198bfff640 1 -- 172.21.15.172:0/3906850619 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199406e010 con 0x7f199406c500 2024-03-16T19:30:36.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f1999a71640 1 -- 172.21.15.172:0/3906850619 >> v1:172.21.15.174:6789/0 conn(0x7f199406c500 legacy=0x7f19940ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:36.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f1999a71640 1 -- 172.21.15.172:0/3906850619 >> v1:172.21.15.172:6789/0 conn(0x7f1994158150 legacy=0x7f199406e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:36.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.517+0000 7f1999a71640 1 -- 172.21.15.172:0/3906850619 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199415b580 con 0x7f199406f4f0 2024-03-16T19:30:36.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.518+0000 7f1999a71640 1 --2- 172.21.15.172:0/3906850619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f199406f4f0 0x7f1994166280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f198c009dc0 tx=0x7f198c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bf437eccd3838e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:36.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.518+0000 7f198bfff640 1 -- 172.21.15.172:0/3906850619 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c02cc40 con 0x7f199406f4f0 2024-03-16T19:30:36.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.518+0000 7f198bfff640 1 -- 172.21.15.172:0/3906850619 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c02cda0 con 0x7f199406f4f0 2024-03-16T19:30:36.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.518+0000 7f198bfff640 1 -- 172.21.15.172:0/3906850619 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c034660 con 0x7f199406f4f0 2024-03-16T19:30:36.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.519+0000 7f199b4d4640 1 -- 172.21.15.172:0/3906850619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f199406f4f0 msgr2=0x7f1994166280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.519+0000 7f199b4d4640 1 --2- 172.21.15.172:0/3906850619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f199406f4f0 0x7f1994166280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f198c009dc0 tx=0x7f198c02be90 comp rx=0 tx=0).stop 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.519+0000 7f199b4d4640 1 -- 172.21.15.172:0/3906850619 shutdown_connections 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.519+0000 7f199b4d4640 1 --2- 172.21.15.172:0/3906850619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f199406f4f0 0x7f1994166280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.519+0000 7f199b4d4640 1 -- 172.21.15.172:0/3906850619 >> 172.21.15.172:0/3906850619 conn(0x7f1994066c20 msgr2=0x7f1994067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.519+0000 7f199b4d4640 1 -- 172.21.15.172:0/3906850619 shutdown_connections 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.519+0000 7f199b4d4640 1 -- 172.21.15.172:0/3906850619 wait complete. 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.520+0000 7f199b4d4640 1 Processor -- start 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.520+0000 7f199b4d4640 1 -- start start 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f199b4d4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f199406c500 0x7f19941f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f199b4d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f199406f4f0 0x7f1994159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f199b4d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1994158150 0x7f19941f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f199b4d4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19941f0a70 con 0x7f199406c500 2024-03-16T19:30:36.504 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f199b4d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994151f80 con 0x7f199406f4f0 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f199b4d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19941f4c90 con 0x7f1994158150 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1999270640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f199406c500 0x7f19941f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1999270640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f199406c500 0x7f19941f4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47616/0 (socket says 172.21.15.172:47616) 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1999270640 1 -- 172.21.15.172:0/866037576 learned_addr learned my addr 172.21.15.172:0/866037576 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1999a71640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1994158150 0x7f19941f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1998a6f640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f199406f4f0 0x7f1994159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1999270640 1 -- 172.21.15.172:0/866037576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1994158150 msgr2=0x7f19941f0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1999270640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1994158150 0x7f19941f0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1999270640 1 -- 172.21.15.172:0/866037576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f199406f4f0 msgr2=0x7f1994159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1999270640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f199406f4f0 0x7f1994159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.521+0000 7f1999270640 1 -- 172.21.15.172:0/866037576 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c009a70 con 0x7f199406c500 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.522+0000 7f1999270640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f199406c500 0x7f19941f4750 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f199000c440 tx=0x7f199000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.522+0000 7f1989ffb640 1 -- 172.21.15.172:0/866037576 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990005ce0 con 0x7f199406c500 2024-03-16T19:30:36.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.522+0000 7f1989ffb640 1 -- 172.21.15.172:0/866037576 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1990005e40 con 0x7f199406c500 2024-03-16T19:30:36.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.522+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1994151a50 con 0x7f199406c500 2024-03-16T19:30:36.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.522+0000 7f1989ffb640 1 -- 172.21.15.172:0/866037576 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19900072d0 con 0x7f199406c500 2024-03-16T19:30:36.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.522+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19941521b0 con 0x7f199406c500 2024-03-16T19:30:36.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.523+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19941f0ce0 con 0x7f199406c500 2024-03-16T19:30:36.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.523+0000 7f1989ffb640 1 -- 172.21.15.172:0/866037576 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1990007430 con 0x7f199406c500 2024-03-16T19:30:36.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.524+0000 7f1989ffb640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f197806b3b0 0x7f197806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.524+0000 7f1989ffb640 1 -- 172.21.15.172:0/866037576 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f1990089120 con 0x7f199406c500 2024-03-16T19:30:36.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.524+0000 7f1998a6f640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f197806b3b0 0x7f197806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.524+0000 7f1998a6f640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f197806b3b0 0x7f197806d870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1994152920 tx=0x7f1984009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:36.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.528+0000 7f1989ffb640 1 -- 172.21.15.172:0/866037576 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1990053db0 con 0x7f199406c500 2024-03-16T19:30:36.547 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:36.566+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:36.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:30:36.592 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:36.592 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:36.601 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:36.647 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:36.666+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:36.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.708+0000 7fd344064640 1 Processor -- start 2024-03-16T19:30:36.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.709+0000 7fd344064640 1 -- start start 2024-03-16T19:30:36.691 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.709+0000 7fd344064640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c154e50 0x7fd33c0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.709+0000 7fd344064640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd33c0b5f10 0x7fd33c0cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.709+0000 7fd344064640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c06b110 con 0x7fd33c0caf40 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.709+0000 7fd344064640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c15b8d0 con 0x7fd33c0b5f10 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.709+0000 7fd344064640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c15ad50 con 0x7fd33c154e50 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.710+0000 7fd3415ff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd33c0b5f10 0x7fd33c0cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.710+0000 7fd3415ff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd33c0b5f10 0x7fd33c0cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47632/0 (socket says 172.21.15.172:47632) 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.710+0000 7fd3415ff640 1 -- 172.21.15.172:0/2045044513 learned_addr learned my addr 172.21.15.172:0/2045044513 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.710+0000 7fd341e00640 1 --2- 172.21.15.172:0/2045044513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c154e50 0x7fd33c0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.710+0000 7fd3415ff640 1 -- 172.21.15.172:0/2045044513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c154e50 msgr2=0x7fd33c0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.710+0000 7fd3415ff640 1 --2- 172.21.15.172:0/2045044513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c154e50 0x7fd33c0cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.710+0000 7fd3415ff640 1 -- 172.21.15.172:0/2045044513 >> v1:172.21.15.172:6789/0 conn(0x7fd33c0caf40 legacy=0x7fd33c06b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.710+0000 7fd340dfe640 1 -- 172.21.15.172:0/2045044513 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224715701 0 0) 0x7fd33c06b110 con 0x7fd33c0caf40 2024-03-16T19:30:36.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.710+0000 7fd3415ff640 1 -- 172.21.15.172:0/2045044513 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c15b030 con 0x7fd33c0b5f10 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.711+0000 7fd3415ff640 1 --2- 172.21.15.172:0/2045044513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0b5f10 0x7fd33c0cc170 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd338009870 tx=0x7fd33802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756e4fdc64c2abf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.711+0000 7fd340dfe640 1 -- 172.21.15.172:0/2045044513 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33802f840 con 0x7fd33c0b5f10 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.711+0000 7fd340dfe640 1 -- 172.21.15.172:0/2045044513 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33802f9a0 con 0x7fd33c0b5f10 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.711+0000 7fd340dfe640 1 -- 172.21.15.172:0/2045044513 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33802fd30 con 0x7fd33c0b5f10 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.712+0000 7fd344064640 1 -- 172.21.15.172:0/2045044513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0b5f10 msgr2=0x7fd33c0cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.712+0000 7fd344064640 1 --2- 172.21.15.172:0/2045044513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0b5f10 0x7fd33c0cc170 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd338009870 tx=0x7fd33802ba20 comp rx=0 tx=0).stop 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.712+0000 7fd344064640 1 -- 172.21.15.172:0/2045044513 shutdown_connections 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.712+0000 7fd344064640 1 --2- 172.21.15.172:0/2045044513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0b5f10 0x7fd33c0cc170 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.712+0000 7fd344064640 1 --2- 172.21.15.172:0/2045044513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c154e50 0x7fd33c0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.712+0000 7fd344064640 1 -- 172.21.15.172:0/2045044513 >> 172.21.15.172:0/2045044513 conn(0x7fd33c066c20 msgr2=0x7fd33c067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.712+0000 7fd344064640 1 -- 172.21.15.172:0/2045044513 shutdown_connections 2024-03-16T19:30:36.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.712+0000 7fd344064640 1 -- 172.21.15.172:0/2045044513 wait complete. 2024-03-16T19:30:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.714+0000 7fd344064640 1 Processor -- start 2024-03-16T19:30:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.714+0000 7fd344064640 1 -- start start 2024-03-16T19:30:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd344064640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c0b5f10 0x7fd33c1f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd344064640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0caf40 0x7fd33c14fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd344064640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd33c154e50 0x7fd33c14c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd344064640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c14c130 con 0x7fd33c0caf40 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd344064640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c155530 con 0x7fd33c154e50 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd344064640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c1f1350 con 0x7fd33c0b5f10 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd3415ff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0caf40 0x7fd33c14fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd3415ff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0caf40 0x7fd33c14fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47640/0 (socket says 172.21.15.172:47640) 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd3415ff640 1 -- 172.21.15.172:0/1272376762 learned_addr learned my addr 172.21.15.172:0/1272376762 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd341e00640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c0b5f10 0x7fd33c1f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd342601640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd33c154e50 0x7fd33c14c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd3415ff640 1 -- 172.21.15.172:0/1272376762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c0b5f10 msgr2=0x7fd33c1f0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd3415ff640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c0b5f10 0x7fd33c1f0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd3415ff640 1 -- 172.21.15.172:0/1272376762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd33c154e50 msgr2=0x7fd33c14c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd3415ff640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd33c154e50 0x7fd33c14c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd3415ff640 1 -- 172.21.15.172:0/1272376762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3380094d0 con 0x7fd33c0caf40 2024-03-16T19:30:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.715+0000 7fd3415ff640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0caf40 0x7fd33c14fd70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd338000c00 tx=0x7fd338004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.716+0000 7fd3327fc640 1 -- 172.21.15.172:0/1272376762 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd338037c40 con 0x7fd33c0caf40 2024-03-16T19:30:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.716+0000 7fd3327fc640 1 -- 172.21.15.172:0/1272376762 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd338037dc0 con 0x7fd33c0caf40 2024-03-16T19:30:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.716+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c15ab20 con 0x7fd33c0caf40 2024-03-16T19:30:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.716+0000 7fd3327fc640 1 -- 172.21.15.172:0/1272376762 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd338041640 con 0x7fd33c0caf40 2024-03-16T19:30:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.716+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c153940 con 0x7fd33c0caf40 2024-03-16T19:30:36.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.717+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c0c9ab0 con 0x7fd33c0caf40 2024-03-16T19:30:36.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.717+0000 7fd3327fc640 1 -- 172.21.15.172:0/1272376762 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd3380417a0 con 0x7fd33c0caf40 2024-03-16T19:30:36.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.717+0000 7fd3327fc640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd32006b3b0 0x7fd32006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:36.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.717+0000 7fd3327fc640 1 -- 172.21.15.172:0/1272376762 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fd33803c070 con 0x7fd33c0caf40 2024-03-16T19:30:36.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.717+0000 7fd341e00640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd32006b3b0 0x7fd32006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:36.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.718+0000 7fd341e00640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd32006b3b0 0x7fd32006d870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009aa0 tx=0x7fd32c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:36.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.719+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f1994070550 con 0x7f199406c500 2024-03-16T19:30:36.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.722+0000 7fd3327fc640 1 -- 172.21.15.172:0/1272376762 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd338077480 con 0x7fd33c0caf40 2024-03-16T19:30:36.707 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:36.708 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:36.716 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:36.749 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:36.767+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:36.823 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:36.824 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:36.833 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:36.878 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:36.898+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:36.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:36.903+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fd33c14c270 con 0x7fd33c0caf40 2024-03-16T19:30:36.939 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:36.940 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:36.948 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:36.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:36.989 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:37.008+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:37.052 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:37.070+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:37.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.088+0000 7f836611e640 1 Processor -- start 2024-03-16T19:30:37.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.088+0000 7f836611e640 1 -- start start 2024-03-16T19:30:37.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.088+0000 7f836611e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83600c79a0 0x7f836015aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.088+0000 7f836611e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f836006aed0 0x7f836015b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.088+0000 7f836611e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836006b480 con 0x7f83601549a0 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.088+0000 7f836611e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f836006b800 con 0x7f83600c79a0 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.088+0000 7f836611e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f836006bac0 con 0x7f836006aed0 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f8357fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83600c79a0 0x7f836015aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f835f7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f83601549a0 0x7f8360071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39370/0 (socket says 172.21.15.172:39370) 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f8357fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83600c79a0 0x7f836015aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47642/0 (socket says 172.21.15.172:47642) 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f835f7fe640 1 -- 172.21.15.172:0/707974004 learned_addr learned my addr 172.21.15.172:0/707974004 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f835ffff640 1 --2- 172.21.15.172:0/707974004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f836006aed0 0x7f836015b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f8357fff640 1 -- 172.21.15.172:0/707974004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f836006aed0 msgr2=0x7f836015b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f8357fff640 1 --2- 172.21.15.172:0/707974004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f836006aed0 0x7f836015b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f8357fff640 1 -- 172.21.15.172:0/707974004 >> v1:172.21.15.172:6789/0 conn(0x7f83601549a0 legacy=0x7f8360071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f8357fff640 1 -- 172.21.15.172:0/707974004 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836006bda0 con 0x7f83600c79a0 2024-03-16T19:30:37.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.089+0000 7f835effd640 1 -- 172.21.15.172:0/707974004 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679333052 0 0) 0x7f836006b480 con 0x7f83601549a0 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f8357fff640 1 --2- 172.21.15.172:0/707974004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83600c79a0 0x7f836015aac0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f83480097a0 tx=0x7f834802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a007bb2c371bda9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f835effd640 1 -- 172.21.15.172:0/707974004 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348031830 con 0x7f83600c79a0 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f835effd640 1 -- 172.21.15.172:0/707974004 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8348031990 con 0x7f83600c79a0 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f835effd640 1 -- 172.21.15.172:0/707974004 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348031d20 con 0x7f83600c79a0 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f836611e640 1 -- 172.21.15.172:0/707974004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83600c79a0 msgr2=0x7f836015aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f836611e640 1 --2- 172.21.15.172:0/707974004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83600c79a0 0x7f836015aac0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f83480097a0 tx=0x7f834802b8d0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f836611e640 1 -- 172.21.15.172:0/707974004 shutdown_connections 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f836611e640 1 --2- 172.21.15.172:0/707974004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f836006aed0 0x7f836015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f836611e640 1 --2- 172.21.15.172:0/707974004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83600c79a0 0x7f836015aac0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f836611e640 1 -- 172.21.15.172:0/707974004 >> 172.21.15.172:0/707974004 conn(0x7f8360066c20 msgr2=0x7f8360067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:37.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f836611e640 1 -- 172.21.15.172:0/707974004 shutdown_connections 2024-03-16T19:30:37.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.090+0000 7f836611e640 1 -- 172.21.15.172:0/707974004 wait complete. 2024-03-16T19:30:37.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f836611e640 1 Processor -- start 2024-03-16T19:30:37.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f836611e640 1 -- start start 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f836611e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f836006aed0 0x7f836014bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f836611e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83600c79a0 0x7f836006f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f836611e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83601549a0 0x7f836015d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f836611e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83601effa0 con 0x7f83601549a0 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f836611e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83601f0190 con 0x7f83600c79a0 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f836611e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83601f0430 con 0x7f836006aed0 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f835ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83601549a0 0x7f836015d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f835ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83601549a0 0x7f836015d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47656/0 (socket says 172.21.15.172:47656) 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f835ffff640 1 -- 172.21.15.172:0/1283507704 learned_addr learned my addr 172.21.15.172:0/1283507704 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f835f7fe640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f836006aed0 0x7f836014bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.092+0000 7f8357fff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83600c79a0 0x7f836006f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f835ffff640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f836006aed0 msgr2=0x7f836014bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f835ffff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f836006aed0 0x7f836014bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f835ffff640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83600c79a0 msgr2=0x7f836006f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f835ffff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83600c79a0 0x7f836006f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f835ffff640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8348009400 con 0x7f83601549a0 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f835ffff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83601549a0 0x7f836015d010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f835800efc0 tx=0x7f835800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8358009ba0 con 0x7f83601549a0 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835800f040 con 0x7f83601549a0 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83601f0710 con 0x7f83601549a0 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83580146c0 con 0x7f83601549a0 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.093+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83601f0cd0 con 0x7f83601549a0 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.094+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f83600681c0 con 0x7f83601549a0 2024-03-16T19:30:37.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.094+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8358014820 con 0x7f83601549a0 2024-03-16T19:30:37.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.095+0000 7f835cff9640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f834006b3b0 0x7f834006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.095+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f835808a9c0 con 0x7f83601549a0 2024-03-16T19:30:37.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.095+0000 7f835f7fe640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f834006b3b0 0x7f834006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.095+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v48) v1 ==== 54+0+16 (secure 0 0 0) 0x7f835808f220 con 0x7f83601549a0 2024-03-16T19:30:37.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.095+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f836015d550 0x7f83600c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.095+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f836006c360 con 0x7f836015d550 2024-03-16T19:30:37.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.095+0000 7f835f7fe640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f834006b3b0 0x7f834006d870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8350000f60 tx=0x7f8350009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.095+0000 7f8357fff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f836015d550 0x7f83600c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.096+0000 7f8357fff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f836015d550 0x7f83600c6ca0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.096+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8348031b00 con 0x7f836015d550 2024-03-16T19:30:37.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.111+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8360154d80 con 0x7f836015d550 2024-03-16T19:30:37.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.112+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f834803a420 con 0x7f836015d550 2024-03-16T19:30:37.094 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:37.095 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:37.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.112+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f8360149db0 0x7f83601f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.112+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8360074580 con 0x7f8360149db0 2024-03-16T19:30:37.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.112+0000 7f835f7fe640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f8360149db0 0x7f83601f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.113+0000 7f835f7fe640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f8360149db0 0x7f83601f3510 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.113+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8360074580 con 0x7f8360149db0 2024-03-16T19:30:37.096 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:37.096 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:37.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:37.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.128+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f83600700d0 con 0x7f8360149db0 2024-03-16T19:30:37.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.130+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f83600700d0 con 0x7f8360149db0 2024-03-16T19:30:37.112 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:37.112 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:37.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.130+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f836014a190 0x7f83601f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.130+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f836006fd00 con 0x7f836014a190 2024-03-16T19:30:37.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.130+0000 7f8357fff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f836014a190 0x7f83601f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.131+0000 7f8357fff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f836014a190 0x7f83601f3f60 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.131+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f836006fd00 con 0x7f836014a190 2024-03-16T19:30:37.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.146+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f836015eca0 con 0x7f836014a190 2024-03-16T19:30:37.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.147+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f834803a960 con 0x7f836014a190 2024-03-16T19:30:37.129 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:37.130 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:37.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.148+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8360062200 0x7f8360158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.148+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8360062640 con 0x7f8360062200 2024-03-16T19:30:37.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.148+0000 7f835ffff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8360062200 0x7f8360158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.148+0000 7f835ffff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8360062200 0x7f8360158660 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.149+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8360062640 con 0x7f8360062200 2024-03-16T19:30:37.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.163+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8360158ba0 con 0x7f8360062200 2024-03-16T19:30:37.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.165+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8360158ba0 con 0x7f8360062200 2024-03-16T19:30:37.146 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:37.147 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:37.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.165+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f8360159280 0x7f836002bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.165+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8360205fa0 con 0x7f8360159280 2024-03-16T19:30:37.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.165+0000 7f835f7fe640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f8360159280 0x7f836002bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.166+0000 7f835f7fe640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f8360159280 0x7f836002bde0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.167+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8360205fa0 con 0x7f8360159280 2024-03-16T19:30:37.152 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:37.171+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:37.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.181+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f836002c320 con 0x7f8360159280 2024-03-16T19:30:37.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.182+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f836002c320 con 0x7f8360159280 2024-03-16T19:30:37.164 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:37.165 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:37.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.183+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f836002ca00 0x7f836002ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.183+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8360209130 con 0x7f836002ca00 2024-03-16T19:30:37.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.183+0000 7f8357fff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f836002ca00 0x7f836002ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.183+0000 7f8357fff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f836002ca00 0x7f836002ce60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.184+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8360209130 con 0x7f836002ca00 2024-03-16T19:30:37.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.199+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f83602068a0 con 0x7f836002ca00 2024-03-16T19:30:37.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.201+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f83602068a0 con 0x7f836002ca00 2024-03-16T19:30:37.183 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:37.183 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:37.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.201+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8360206f20 0x7f8360209310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.201+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f836020c2b0 con 0x7f8360206f20 2024-03-16T19:30:37.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.202+0000 7f835ffff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8360206f20 0x7f8360209310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.202+0000 7f835ffff640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8360206f20 0x7f8360209310 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.203+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f836020c2b0 con 0x7f8360206f20 2024-03-16T19:30:37.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.217+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8360062890 con 0x7f8360206f20 2024-03-16T19:30:37.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.218+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f8360062890 con 0x7f8360206f20 2024-03-16T19:30:37.200 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:37.201 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:37.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.218+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f836020b730 0x7f836020bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.219+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f836020f640 con 0x7f836020b730 2024-03-16T19:30:37.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.219+0000 7f835f7fe640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f836020b730 0x7f836020bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.219+0000 7f835f7fe640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f836020b730 0x7f836020bb90 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.220+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f836020f640 con 0x7f836020b730 2024-03-16T19:30:37.214 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:37.214 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:37.215 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:37.215 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:37.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.234+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f836020e4c0 con 0x7f836020b730 2024-03-16T19:30:37.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f835cff9640 1 -- 172.21.15.172:0/1283507704 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f836020e4c0 con 0x7f836020b730 2024-03-16T19:30:37.217 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:37.218 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:37.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f836015d550 msgr2=0x7f83600c6ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f836015d550 0x7f83600c6ca0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f8360149db0 msgr2=0x7f83601f3510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f8360149db0 0x7f83601f3510 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f836014a190 msgr2=0x7f83601f3f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f836014a190 0x7f83601f3f60 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8360062200 msgr2=0x7f8360158660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f8360062200 0x7f8360158660 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f835ffff640 1 -- 172.21.15.172:0/1283507704 reap_dead start 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f8360159280 msgr2=0x7f836002bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f8360159280 0x7f836002bde0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f836002ca00 msgr2=0x7f836002ce60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f835ffff640 1 -- 172.21.15.172:0/1283507704 reap_dead start 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f836002ca00 0x7f836002ce60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8360206f20 msgr2=0x7f8360209310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f8360206f20 0x7f8360209310 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f836020b730 msgr2=0x7f836020bb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f836020b730 0x7f836020bb90 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f834006b3b0 msgr2=0x7f834006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f834006b3b0 0x7f834006d870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8350000f60 tx=0x7f8350009040 comp rx=0 tx=0).stop 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83601549a0 msgr2=0x7f836015d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 --2- 172.21.15.172:0/1283507704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83601549a0 0x7f836015d010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f835800efc0 tx=0x7f835800c490 comp rx=0 tx=0).stop 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f835ffff640 1 -- 172.21.15.172:0/1283507704 reap_dead start 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 shutdown_connections 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 >> 172.21.15.172:0/1283507704 conn(0x7f8360066c20 msgr2=0x7f836015aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 shutdown_connections 2024-03-16T19:30:37.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.236+0000 7f836611e640 1 -- 172.21.15.172:0/1283507704 wait complete. 2024-03-16T19:30:37.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:37.272 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:37.290+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:37.354 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:37.373+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:37.375 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:37.376 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:37.376 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:37.376 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:37.376 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:37.384 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:37.455 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:37.474+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:37.493 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.919130839000005, 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:30:37.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:30:37.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:30:37.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.128904668000004, 2024-03-16T19:30:37.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.128144433000003, 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.127330478000005, 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:30:37.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.126690553000003, 2024-03-16T19:30:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:30:37.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.126032456000004, 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:30:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.125499880999996, 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.520 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:30:37.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:30:37.524 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.067327977999994, 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.066725783999999, 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.529 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:30:37.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.066082839000003, 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:30:37.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:30:37.538 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.065578877999997, 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:30:37.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:30:37.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.907432611999994, 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:37.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:30:37.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.906902475999999, 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:37.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:30:37.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:30:37.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:37.552 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.154933217000007, 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:30:37.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.154908680999995, 2024-03-16T19:30:37.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:30:37.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:30:37.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.154904150999997, 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:37.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:30:37.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:37.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.154894338999995, 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:30:37.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:30:37.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:30:37.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.154890485999999, 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.154887169999995, 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:37.581 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:30:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:30:37.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.153691586999997, 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:37.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:30:37.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:30:37.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.589 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.589 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.589 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:37.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:37.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.598+0000 7f1989ffb640 1 -- 172.21.15.172:0/866037576 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v49) v1 ==== 113+0+0 (secure 0 0 0) 0x7f1990057470 con 0x7f199406c500 2024-03-16T19:30:37.591 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.598+0000 7fd3327fc640 1 -- 172.21.15.172:0/1272376762 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v49) v1 ==== 91+0+0 (secure 0 0 0) 0x7fd33c14c270 con 0x7fd33c0caf40 2024-03-16T19:30:37.592 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.0 to 1 (10000) 2024-03-16T19:30:37.592 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f197806b3b0 msgr2=0x7f197806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f197806b3b0 0x7f197806d870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1994152920 tx=0x7f1984009040 comp rx=0 tx=0).stop 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f199406c500 msgr2=0x7f19941f4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f199406c500 0x7f19941f4750 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f199000c440 tx=0x7f199000c900 comp rx=0 tx=0).stop 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 shutdown_connections 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f197806b3b0 0x7f197806d870 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1994158150 0x7f19941f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f199406f4f0 0x7f1994159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 --2- 172.21.15.172:0/866037576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f199406c500 0x7f19941f4750 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 >> 172.21.15.172:0/866037576 conn(0x7f1994066c20 msgr2=0x7f1994156ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 shutdown_connections 2024-03-16T19:30:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.601+0000 7f199b4d4640 1 -- 172.21.15.172:0/866037576 wait complete. 2024-03-16T19:30:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd32006b3b0 msgr2=0x7fd32006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd32006b3b0 0x7fd32006d870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009aa0 tx=0x7fd32c009190 comp rx=0 tx=0).stop 2024-03-16T19:30:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0caf40 msgr2=0x7fd33c14fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0caf40 0x7fd33c14fd70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd338000c00 tx=0x7fd338004730 comp rx=0 tx=0).stop 2024-03-16T19:30:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 shutdown_connections 2024-03-16T19:30:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd33c154e50 0x7fd33c14c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd33c0caf40 0x7fd33c14fd70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd32006b3b0 0x7fd32006d870 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 --2- 172.21.15.172:0/1272376762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd33c0b5f10 0x7fd33c1f0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 >> 172.21.15.172:0/1272376762 conn(0x7fd33c066c20 msgr2=0x7fd33c0c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:37.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 shutdown_connections 2024-03-16T19:30:37.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.602+0000 7fd344064640 1 -- 172.21.15.172:0/1272376762 wait complete. 2024-03-16T19:30:37.595 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:37.610+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:37.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-16T19:30:37.690 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:37.710+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:37.703 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:37.703 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:37.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:37.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.762+0000 7f23ac433640 1 Processor -- start 2024-03-16T19:30:37.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.762+0000 7f23ac433640 1 -- start start 2024-03-16T19:30:37.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23ac433640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f23a4154c90 0x7f23a40c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23ac433640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23ac433640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a406b2b0 con 0x7f23a40cd290 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23ac433640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a415b470 con 0x7f23a4154c90 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23ac433640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a406b590 con 0x7f23a406cc50 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23aa1cf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f23a4154c90 0x7f23a40c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23a99ce640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23aa9d0640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f23a40cd290 0x7f23a415dfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:41066/0 (socket says 172.21.15.172:41066) 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23a99ce640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:57414/0 (socket says 172.21.15.172:57414) 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23aa9d0640 1 -- 172.21.15.172:0/1041640538 learned_addr learned my addr 172.21.15.172:0/1041640538 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23aa1cf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f23a4154c90 0x7f23a40c79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:41184/0 (socket says 172.21.15.172:41184) 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23aa1cf640 1 -- 172.21.15.172:0/1041640538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 msgr2=0x7f23a40c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23aa1cf640 1 --2- 172.21.15.172:0/1041640538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23aa1cf640 1 -- 172.21.15.172:0/1041640538 >> v1:172.21.15.174:6789/0 conn(0x7f23a40cd290 legacy=0x7f23a415dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23aa1cf640 1 -- 172.21.15.172:0/1041640538 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a415adb0 con 0x7f23a4154c90 2024-03-16T19:30:37.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23a91cd640 1 -- 172.21.15.172:0/1041640538 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844251429 0 0) 0x7f23a406b2b0 con 0x7f23a40cd290 2024-03-16T19:30:37.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.763+0000 7f23a99ce640 1 --2- 172.21.15.172:0/1041640538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.764+0000 7f23aa1cf640 1 --2- 172.21.15.172:0/1041640538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a4154c90 0x7f23a40c79d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f23a00097a0 tx=0x7f23a002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22adb83eeb7fe46f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.764+0000 7f23a91cd640 1 -- 172.21.15.172:0/1041640538 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0031830 con 0x7f23a4154c90 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.764+0000 7f23a91cd640 1 -- 172.21.15.172:0/1041640538 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a0031990 con 0x7f23a4154c90 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.764+0000 7f23a91cd640 1 -- 172.21.15.172:0/1041640538 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0031d20 con 0x7f23a4154c90 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.765+0000 7f23ac433640 1 -- 172.21.15.172:0/1041640538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a4154c90 msgr2=0x7f23a40c79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.765+0000 7f23ac433640 1 --2- 172.21.15.172:0/1041640538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a4154c90 0x7f23a40c79d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f23a00097a0 tx=0x7f23a002b8d0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.765+0000 7f23ac433640 1 -- 172.21.15.172:0/1041640538 shutdown_connections 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.765+0000 7f23ac433640 1 --2- 172.21.15.172:0/1041640538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.765+0000 7f23ac433640 1 --2- 172.21.15.172:0/1041640538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a4154c90 0x7f23a40c79d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.765+0000 7f23ac433640 1 -- 172.21.15.172:0/1041640538 >> 172.21.15.172:0/1041640538 conn(0x7f23a4066c20 msgr2=0x7f23a4067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.765+0000 7f23ac433640 1 -- 172.21.15.172:0/1041640538 shutdown_connections 2024-03-16T19:30:37.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.765+0000 7f23ac433640 1 -- 172.21.15.172:0/1041640538 wait complete. 2024-03-16T19:30:37.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.766+0000 7f23ac433640 1 Processor -- start 2024-03-16T19:30:37.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.766+0000 7f23ac433640 1 -- start start 2024-03-16T19:30:37.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23ac433640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23ac433640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a40cd290 0x7f23a40b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23ac433640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23a4154c90 0x7f23a40c4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23ac433640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40c5270 con 0x7f23a4154c90 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23ac433640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40c7190 con 0x7f23a40cd290 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23ac433640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40c6580 con 0x7f23a406cc50 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa9d0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23a4154c90 0x7f23a40c4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa1cf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23a99ce640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a40cd290 0x7f23a40b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa9d0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23a4154c90 0x7f23a40c4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:47668/0 (socket says 172.21.15.172:47668) 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa9d0640 1 -- 172.21.15.172:0/3739308483 learned_addr learned my addr 172.21.15.172:0/3739308483 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa1cf640 1 -- 172.21.15.172:0/3739308483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a40cd290 msgr2=0x7f23a40b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa1cf640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a40cd290 0x7f23a40b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa1cf640 1 -- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23a4154c90 msgr2=0x7f23a40c4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa1cf640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23a4154c90 0x7f23a40c4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa1cf640 1 -- 172.21.15.172:0/3739308483 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2394009400 con 0x7f23a406cc50 2024-03-16T19:30:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa9d0640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23a4154c90 0x7f23a40c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:37.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23a99ce640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a40cd290 0x7f23a40b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:37.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f23aa1cf640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40b04d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f23a0033000 tx=0x7f23a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.767+0000 7f239affd640 1 -- 172.21.15.172:0/3739308483 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0004230 con 0x7f23a406cc50 2024-03-16T19:30:37.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.768+0000 7f239affd640 1 -- 172.21.15.172:0/3739308483 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a0004390 con 0x7f23a406cc50 2024-03-16T19:30:37.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.768+0000 7f239affd640 1 -- 172.21.15.172:0/3739308483 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a003a8a0 con 0x7f23a406cc50 2024-03-16T19:30:37.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.768+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a0009400 con 0x7f23a406cc50 2024-03-16T19:30:37.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.768+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a40c6b40 con 0x7f23a406cc50 2024-03-16T19:30:37.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.769+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a40688d0 con 0x7f23a406cc50 2024-03-16T19:30:37.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.769+0000 7f239affd640 1 -- 172.21.15.172:0/3739308483 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f23a003aa00 con 0x7f23a406cc50 2024-03-16T19:30:37.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.769+0000 7f239affd640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f238806b3b0 0x7f238806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:37.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.769+0000 7f239affd640 1 -- 172.21.15.172:0/3739308483 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f23a00ac1d0 con 0x7f23a406cc50 2024-03-16T19:30:37.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.769+0000 7f23a99ce640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f238806b3b0 0x7f238806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:37.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.770+0000 7f23a99ce640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f238806b3b0 0x7f238806d870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f23a40c7540 tx=0x7f2394009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:37.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.774+0000 7f239affd640 1 -- 172.21.15.172:0/3739308483 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23a0076e50 con 0x7f23a406cc50 2024-03-16T19:30:37.791 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:37.811+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:37.823 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:37.823 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:37.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:37.891 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:37.911+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:37.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:37.964+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f23a406b910 con 0x7f23a406cc50 2024-03-16T19:30:37.982 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:37.982 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:37.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:37.982 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:37.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.982 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.982 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:37.982 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.643594974999999, 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.643577860999997, 2024-03-16T19:30:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.643570581000006, 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:30:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.643556713999999, 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.642883878999996, 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:30:38.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:30:38.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.640585629, 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:38.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:38.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:30:38.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:30:38.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.640540528000002, 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:30:38.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:30:38.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:30:38.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.640512473000001, 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:30:38.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.640491058999999, 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.640468708, 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.025 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.640448309000007, 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:38.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:30:38.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:38.029 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.640342317000005, 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.638550605999995, 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:30:38.033 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:30:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.638503104999998, 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:30:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:30:38.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.040 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.638263854000002, 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.629765708999997, 2024-03-16T19:30:38.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:30:38.045 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.045 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.045 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:38.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:38.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.628425833000001, 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:30:38.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:30:38.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.625434851000001, 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:30:38.054 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:30:38.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:30:38.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:30:38.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.624804049, 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.059 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:30:38.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:30:38.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:47 2.6 e40/18 ondisk, result = 0)", 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.62332164, 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017677, 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:38.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377611+0000", 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9599999999999998e-07 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377618+0000", 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9990000000000002e-06 2024-03-16T19:30:38.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377626+0000", 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4249999999999998e-06 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377632+0000", 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377707+0000", 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4506999999999997e-05 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377714+0000", 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5819999999999998e-06 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377787+0000", 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3232000000000006e-05 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.067 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:38.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:38.069 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:38.088+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:38.169 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:38.189+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:38.177 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:38.177 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:38.177 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:38.177 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:38.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:38.269 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:38.289+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:38.293 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:38.294 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:38.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:38.371 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:38.390+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072643171999999, 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053568110000000004, 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.411 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357146+0000", 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5860000000000001e-06 2024-03-16T19:30:38.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357154+0000", 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5439999999999999e-06 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357217+0000", 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2986999999999999e-05 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357229+0000", 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1853e-05 2024-03-16T19:30:38.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357343+0000", 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113663 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357504+0000", 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016176 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013908599999999999 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360705+0000", 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030619419999999998 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362409+0000", 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001704021 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362456+0000", 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6140000000000002e-05 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362496+0000", 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0228999999999997e-05 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072623144999994, 2024-03-16T19:30:38.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078702890000000008, 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4299999999999998e-07 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357163+0000", 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2559999999999998e-06 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357254+0000", 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1701999999999997e-05 2024-03-16T19:30:38.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357258+0000", 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5719999999999999e-06 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357931+0000", 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067308000000000005 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357999+0000", 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8323000000000003e-05 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358113+0000", 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011366699999999999 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362505+0000", 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004391509 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024372970000000002 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364992+0000", 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9741999999999999e-05 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365029+0000", 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7697999999999997e-05 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072616343000007, 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079944349999999994, 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6e-07 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357172+0000", 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7960000000000001e-06 2024-03-16T19:30:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357285+0000", 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113183 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357290+0000", 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6190000000000002e-06 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358306+0000", 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010160670000000001 2024-03-16T19:30:38.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358400+0000", 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3854e-05 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358492+0000", 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2954000000000005e-05 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362510+0000", 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040175469999999998 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025680310000000001 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365123+0000", 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4515000000000003e-05 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365160+0000", 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7608999999999999e-05 2024-03-16T19:30:38.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072609447000005, 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080787759999999993, 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.36e-07 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1270000000000001e-06 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357297+0000", 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121885 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357300+0000", 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049999999999999e-06 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358709+0000", 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001409083 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358776+0000", 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6489999999999995e-05 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358885+0000", 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109271 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362514+0000", 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036292199999999998 2024-03-16T19:30:38.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365197+0000", 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026832100000000001 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365226+0000", 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9136999999999999e-05 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365251+0000", 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5012e-05 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.431 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072606463, 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081787609999999997, 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357177+0000", 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5090000000000001e-06 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357306+0000", 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012856900000000001 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357308+0000", 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3700000000000002e-06 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359071+0000", 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017631210000000001 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359182+0000", 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110265 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359280+0000", 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8595e-05 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362520+0000", 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032394120000000001 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365297+0000", 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002777542 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365328+0000", 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0899999999999999e-05 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365354+0000", 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6276000000000001e-05 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072603775000005, 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083197830000000007, 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:38.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357179+0000", 2024-03-16T19:30:38.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:38.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357187+0000", 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357314+0000", 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126978 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357316+0000", 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3800000000000001e-06 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359512+0000", 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021950799999999999 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359585+0000", 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3168000000000001e-05 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359702+0000", 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011741 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362525+0000", 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028225419999999999 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365408+0000", 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028829010000000002 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365460+0000", 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839e-05 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365498+0000", 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7673999999999999e-05 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072592650000004, 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084493169999999996, 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357190+0000", 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-07 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357191+0000", 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7069999999999999e-06 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357325+0000", 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133585 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357330+0000", 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0799999999999996e-06 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359915+0000", 2024-03-16T19:30:38.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025849290000000001 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9741999999999995e-05 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360121+0000", 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362826+0000", 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027046819999999999 2024-03-16T19:30:38.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365564+0000", 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027387179999999998 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365606+0000", 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1715999999999998e-05 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365639+0000", 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660999999999998e-05 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.446 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072589980999993, 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085533069999999996, 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0699999999999999e-07 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3030000000000001e-06 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357339+0000", 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014513199999999999 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000003e-06 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030087619999999999 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360414+0000", 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1297999999999995e-05 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360498+0000", 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4538999999999998e-05 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362832+0000", 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023338410000000001 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365670+0000", 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028379550000000001 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365724+0000", 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3533999999999999e-05 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365745+0000", 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1725999999999998e-05 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072586756999996, 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052850789999999998, 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357196+0000", 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.173e-06 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357359+0000", 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016222400000000001 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357365+0000", 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7320000000000003e-06 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357451+0000", 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6494000000000002e-05 2024-03-16T19:30:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357555+0000", 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010401200000000001 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8224000000000001e-05 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362294+0000", 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046502399999999999 2024-03-16T19:30:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362388+0000", 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4448000000000001e-05 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362433+0000", 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4888000000000001e-05 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362480+0000", 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7441000000000002e-05 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.072584719999995, 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053762239999999998, 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357198+0000", 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357199+0000", 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:30:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357415+0000", 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021638699999999999 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0069e-05 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358132+0000", 2024-03-16T19:30:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070650599999999997 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358207+0000", 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5197999999999994e-05 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358342+0000", 2024-03-16T19:30:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013503199999999999 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362377+0000", 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040354600000000003 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362493+0000", 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115951 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362536+0000", 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3189999999999998e-05 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362574+0000", 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7113999999999997e-05 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.462 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:6 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.069525819000006, 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045449999999999999, 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360257+0000", 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7400000000000002e-07 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360260+0000", 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3550000000000001e-06 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360270+0000", 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-03-16T19:30:38.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360277+0000", 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5760000000000002e-06 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360697+0000", 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042039200000000001 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360702+0000", 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7140000000000001e-06 2024-03-16T19:30:38.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360711+0000", 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1209999999999996e-06 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:22 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.067581644000001, 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100673, 2024-03-16T19:30:38.466 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362201+0000", 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5799999999999998e-07 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362206+0000", 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8579999999999999e-06 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362219+0000", 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.239e-05 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362226+0000", 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2710000000000003e-06 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362281+0000", 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4756000000000002e-05 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362288+0000", 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.075e-06 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362301+0000", 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3365e-05 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:23 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.067470374999999, 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017205200000000001, 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4499999999999997e-07 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362315+0000", 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1350000000000001e-06 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362322+0000", 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9469999999999997e-06 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362328+0000", 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362366+0000", 2024-03-16T19:30:38.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8160000000000001e-05 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362373+0000", 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5799999999999997e-06 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362484+0000", 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111173 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:7 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.064926502999995, 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018177499999999999, 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364861+0000", 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3699999999999997e-06 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364875+0000", 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4059e-05 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364881+0000", 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5270000000000004e-06 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364929+0000", 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000002e-05 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:38.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3699999999999997e-06 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365037+0000", 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100819 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:8 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.477 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.064869446000003, 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025451199999999998, 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364916+0000", 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8109999999999999e-06 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364924+0000", 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1419999999999993e-06 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364930+0000", 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2260000000000002e-06 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365071+0000", 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014030100000000001 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365075+0000", 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365167+0000", 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2196999999999998e-05 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:9 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:38.481 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.064843206000006, 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032082900000000001, 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.22e-07 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3980000000000002e-06 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364947+0000", 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3090000000000003e-06 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364951+0000", 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1309999999999999e-06 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365191+0000", 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023992600000000001 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365195+0000", 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.704e-06 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365260+0000", 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5139000000000001e-05 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:10 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.064814819999995, 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039415499999999998, 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364968+0000", 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364970+0000", 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4650000000000001e-06 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364977+0000", 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.443e-06 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364981+0000", 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2320000000000003e-06 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365289+0000", 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030856900000000002 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365294+0000", 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6920000000000001e-06 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365361+0000", 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7358000000000001e-05 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:11 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.064794320000004, 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051803199999999998, 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9299999999999999e-07 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364993+0000", 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9080000000000003e-06 2024-03-16T19:30:38.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4240000000000001e-06 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365002+0000", 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7570000000000002e-06 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365395+0000", 2024-03-16T19:30:38.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039237199999999999 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365401+0000", 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9380000000000003e-06 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365506+0000", 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010433999999999999 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:12 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.064772316000003, 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063430500000000005, 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9500000000000001e-07 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365012+0000", 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7129999999999999e-06 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365015+0000", 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2629999999999999e-06 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365017+0000", 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1749999999999999e-06 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365544+0000", 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052690099999999998 2024-03-16T19:30:38.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365559+0000", 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4610000000000001e-05 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365644+0000", 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5447999999999999e-05 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:13 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:38.496 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.064761817999994, 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00073113500000000005, 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1000000000000001e-07 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365021+0000", 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.23e-07 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365024+0000", 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7369999999999998e-06 2024-03-16T19:30:38.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365026+0000", 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1289999999999999e-06 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365663+0000", 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063679600000000002 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365667+0000", 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.233e-06 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365751+0000", 2024-03-16T19:30:38.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4307000000000004e-05 2024-03-16T19:30:38.500 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.500 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.500 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.500 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.500 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.500 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:38.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:38.502 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:38.521+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:38.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.602+0000 7f239affd640 1 -- 172.21.15.172:0/3739308483 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v50) v1 ==== 113+0+0 (secure 0 0 0) 0x7f23a003acb0 con 0x7f23a406cc50 2024-03-16T19:30:38.584 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.1 to 1 (10000) 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f238806b3b0 msgr2=0x7f238806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f238806b3b0 0x7f238806d870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f23a40c7540 tx=0x7f2394009190 comp rx=0 tx=0).stop 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 msgr2=0x7f23a40b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40b04d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f23a0033000 tx=0x7f23a00046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 shutdown_connections 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23a4154c90 0x7f23a40c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23a40cd290 0x7f23a40b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f238806b3b0 0x7f238806d870 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 --2- 172.21.15.172:0/3739308483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23a406cc50 0x7f23a40b04d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 >> 172.21.15.172:0/3739308483 conn(0x7f23a4066c20 msgr2=0x7f23a40c76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 shutdown_connections 2024-03-16T19:30:38.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.605+0000 7f23ac433640 1 -- 172.21.15.172:0/3739308483 wait complete. 2024-03-16T19:30:38.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-16T19:30:38.602 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:38.622+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:38.652 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:38.652 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:38.652 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:38.652 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:38.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:38.703 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:38.722+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:38.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.762+0000 7fa3ba0a7640 1 Processor -- start 2024-03-16T19:30:38.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.762+0000 7fa3ba0a7640 1 -- start start 2024-03-16T19:30:38.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.762+0000 7fa3ba0a7640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa3b4152580 0x7fa3b4159090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:38.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.762+0000 7fa3ba0a7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa3b40c9ec0 0x7fa3b4162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:38.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.762+0000 7fa3ba0a7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b40cd4c0 con 0x7fa3b40c9aa0 2024-03-16T19:30:38.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.762+0000 7fa3ba0a7640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b41595d0 con 0x7fa3b4152580 2024-03-16T19:30:38.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.762+0000 7fa3ba0a7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b40cd760 con 0x7fa3b40c9ec0 2024-03-16T19:30:38.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.763+0000 7fa3b37fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa3b4152580 0x7fa3b4159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:38.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.763+0000 7fa3b2ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa3b40c9aa0 0x7fa3b406b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45040/0 (socket says 172.21.15.172:45040) 2024-03-16T19:30:38.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.763+0000 7fa3b2ffd640 1 -- 172.21.15.172:0/1253842541 learned_addr learned my addr 172.21.15.172:0/1253842541 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:38.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.763+0000 7fa3b3fff640 1 --2- 172.21.15.172:0/1253842541 >> v2:172.21.15.174:3300/0 conn(0x7fa3b40c9ec0 0x7fa3b4162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:38.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.763+0000 7fa3b37fe640 1 -- 172.21.15.172:0/1253842541 >> v2:172.21.15.174:3300/0 conn(0x7fa3b40c9ec0 msgr2=0x7fa3b4162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:38.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.763+0000 7fa3b37fe640 1 --2- 172.21.15.172:0/1253842541 >> v2:172.21.15.174:3300/0 conn(0x7fa3b40c9ec0 0x7fa3b4162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.763+0000 7fa3b37fe640 1 -- 172.21.15.172:0/1253842541 >> v1:172.21.15.172:6789/0 conn(0x7fa3b40c9aa0 legacy=0x7fa3b406b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:38.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.763+0000 7fa3b37fe640 1 -- 172.21.15.172:0/1253842541 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b40cda20 con 0x7fa3b4152580 2024-03-16T19:30:38.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.763+0000 7fa3b27fc640 1 -- 172.21.15.172:0/1253842541 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759749565 0 0) 0x7fa3b40cd4c0 con 0x7fa3b40c9aa0 2024-03-16T19:30:38.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.764+0000 7fa3b37fe640 1 --2- 172.21.15.172:0/1253842541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 0x7fa3b4159090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4073670 tx=0x7fa3a402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5789694d9e9c794a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:38.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.764+0000 7fa3b27fc640 1 -- 172.21.15.172:0/1253842541 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4030610 con 0x7fa3b4152580 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.764+0000 7fa3b27fc640 1 -- 172.21.15.172:0/1253842541 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a4030770 con 0x7fa3b4152580 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.764+0000 7fa3b27fc640 1 -- 172.21.15.172:0/1253842541 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4030b00 con 0x7fa3b4152580 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.765+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/1253842541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 msgr2=0x7fa3b4159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.765+0000 7fa3ba0a7640 1 --2- 172.21.15.172:0/1253842541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 0x7fa3b4159090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4073670 tx=0x7fa3a402b960 comp rx=0 tx=0).stop 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.765+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/1253842541 shutdown_connections 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.765+0000 7fa3ba0a7640 1 --2- 172.21.15.172:0/1253842541 >> v2:172.21.15.174:3300/0 conn(0x7fa3b40c9ec0 0x7fa3b4162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.765+0000 7fa3ba0a7640 1 --2- 172.21.15.172:0/1253842541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 0x7fa3b4159090 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.765+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/1253842541 >> 172.21.15.172:0/1253842541 conn(0x7fa3b406bfe0 msgr2=0x7fa3b406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.765+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/1253842541 shutdown_connections 2024-03-16T19:30:38.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.765+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/1253842541 wait complete. 2024-03-16T19:30:38.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.766+0000 7fa3ba0a7640 1 Processor -- start 2024-03-16T19:30:38.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.766+0000 7fa3ba0a7640 1 -- start start 2024-03-16T19:30:38.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3ba0a7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b40c9aa0 0x7fa3b40aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:38.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3ba0a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b40c9ec0 0x7fa3b414d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:38.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3ba0a7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 0x7fa3b4147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:38.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3ba0a7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b41f0140 con 0x7fa3b4152580 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3ba0a7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b40c26c0 con 0x7fa3b40c9ec0 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3ba0a7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b40b0a10 con 0x7fa3b40c9aa0 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b3fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 0x7fa3b4147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b37fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b40c9aa0 0x7fa3b40aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b3fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 0x7fa3b4147ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57362/0 (socket says 172.21.15.172:57362) 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b37fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b40c9aa0 0x7fa3b40aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48004/0 (socket says 172.21.15.172:48004) 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b3fff640 1 -- 172.21.15.172:0/3522973189 learned_addr learned my addr 172.21.15.172:0/3522973189 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b2ffd640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b40c9ec0 0x7fa3b414d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b3fff640 1 -- 172.21.15.172:0/3522973189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b40c9aa0 msgr2=0x7fa3b40aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b3fff640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b40c9aa0 0x7fa3b40aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b3fff640 1 -- 172.21.15.172:0/3522973189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b40c9ec0 msgr2=0x7fa3b414d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b3fff640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b40c9ec0 0x7fa3b414d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:38.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b3fff640 1 -- 172.21.15.172:0/3522973189 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a4009400 con 0x7fa3b4152580 2024-03-16T19:30:38.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3b3fff640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 0x7fa3b4147ac0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00efc0 tx=0x7fa3ac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:38.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa39bfff640 1 -- 172.21.15.172:0/3522973189 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac009ba0 con 0x7fa3b4152580 2024-03-16T19:30:38.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa39bfff640 1 -- 172.21.15.172:0/3522973189 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ac00f040 con 0x7fa3b4152580 2024-03-16T19:30:38.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa39bfff640 1 -- 172.21.15.172:0/3522973189 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac0146c0 con 0x7fa3b4152580 2024-03-16T19:30:38.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.767+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b40b0cf0 con 0x7fa3b4152580 2024-03-16T19:30:38.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.768+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b40b29b0 con 0x7fa3b4152580 2024-03-16T19:30:38.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.768+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b40c2830 con 0x7fa3b4152580 2024-03-16T19:30:38.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.769+0000 7fa39bfff640 1 -- 172.21.15.172:0/3522973189 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa3ac014820 con 0x7fa3b4152580 2024-03-16T19:30:38.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.769+0000 7fa39bfff640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa3a006b3b0 0x7fa3a006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:38.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.769+0000 7fa39bfff640 1 -- 172.21.15.172:0/3522973189 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fa3ac08a630 con 0x7fa3b4152580 2024-03-16T19:30:38.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.769+0000 7fa3b37fe640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa3a006b3b0 0x7fa3a006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:38.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.770+0000 7fa3b37fe640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa3a006b3b0 0x7fa3a006d870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4036000 tx=0x7fa3a4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:38.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.778+0000 7fa39bfff640 1 -- 172.21.15.172:0/3522973189 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3ac0552c0 con 0x7fa3b4152580 2024-03-16T19:30:38.803 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:38.822+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:38.815 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:38.815 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:38.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:38.909 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:38.928+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:38.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:38.969+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fa3b40688d0 con 0x7fa3b4152580 2024-03-16T19:30:38.975 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.637191393999998, 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.637176921000005, 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:30:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:30:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.636672431999997, 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.636657036000003, 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.636198352999997, 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:30:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.635652183000005, 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.635095092, 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:30:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:30:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:39.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.633893344000001, 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.002 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.633314635000005, 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:39.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.633265108000003, 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:30:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.632304465000004, 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:30:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:30:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.628590267000007, 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 59.188189354999999, 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:30:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.622636352000001, 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.115037627, 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:39.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:39.032 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:39.033 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:39.033 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:39.052+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:39.134 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:39.153+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:39.140 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:39.140 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:39.140 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:39.140 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:39.148 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:39.230 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:39.258 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:39.258 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:39.258 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:39.258 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:39.258 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:39.267 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:39.274 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:39.293+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:39.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.347+0000 7f519080e640 1 Processor -- start 2024-03-16T19:30:39.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.347+0000 7f519080e640 1 -- start start 2024-03-16T19:30:39.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.348+0000 7f519080e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f518c0b3630 0x7f518c0cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.348+0000 7f519080e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c0b4270 con 0x7f518c0cb5e0 2024-03-16T19:30:39.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.348+0000 7f519080e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c15b250 con 0x7f518c154ed0 2024-03-16T19:30:39.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.348+0000 7f519080e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f518c0cb9f0 con 0x7f518c0b3630 2024-03-16T19:30:39.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.348+0000 7f518ad9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f518c0cb5e0 0x7f518c0b4700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45050/0 (socket says 172.21.15.172:45050) 2024-03-16T19:30:39.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.348+0000 7f518ad9d640 1 -- 172.21.15.172:0/2730177443 learned_addr learned my addr 172.21.15.172:0/2730177443 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:39.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.349+0000 7f5189d9b640 1 --2- 172.21.15.172:0/2730177443 >> v2:172.21.15.174:3300/0 conn(0x7f518c0b3630 0x7f518c0cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.349+0000 7f518959a640 1 -- 172.21.15.172:0/2730177443 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346624985 0 0) 0x7f518c0b4270 con 0x7f518c0cb5e0 2024-03-16T19:30:39.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.349+0000 7f518959a640 1 -- 172.21.15.172:0/2730177443 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5174003520 con 0x7f518c0cb5e0 2024-03-16T19:30:39.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.349+0000 7f5189d9b640 1 -- 172.21.15.172:0/2730177443 >> v1:172.21.15.174:6789/0 conn(0x7f518c154ed0 legacy=0x7f518c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.349+0000 7f5189d9b640 1 -- 172.21.15.172:0/2730177443 >> v1:172.21.15.172:6789/0 conn(0x7f518c0cb5e0 legacy=0x7f518c0b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.349+0000 7f5189d9b640 1 -- 172.21.15.172:0/2730177443 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c1552b0 con 0x7f518c0b3630 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f5189d9b640 1 --2- 172.21.15.172:0/2730177443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f518c0b3630 0x7f518c0cb0a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5180009d90 tx=0x7f518002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4f82e27ef5f2a18 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f518959a640 1 -- 172.21.15.172:0/2730177443 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518002cd00 con 0x7f518c0b3630 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f518959a640 1 -- 172.21.15.172:0/2730177443 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518002ce60 con 0x7f518c0b3630 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f518959a640 1 -- 172.21.15.172:0/2730177443 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5180034640 con 0x7f518c0b3630 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f519080e640 1 -- 172.21.15.172:0/2730177443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f518c0b3630 msgr2=0x7f518c0cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f519080e640 1 --2- 172.21.15.172:0/2730177443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f518c0b3630 0x7f518c0cb0a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5180009d90 tx=0x7f518002bf50 comp rx=0 tx=0).stop 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f519080e640 1 -- 172.21.15.172:0/2730177443 shutdown_connections 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f519080e640 1 --2- 172.21.15.172:0/2730177443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f518c0b3630 0x7f518c0cb0a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f519080e640 1 -- 172.21.15.172:0/2730177443 >> 172.21.15.172:0/2730177443 conn(0x7f518c06bf60 msgr2=0x7f518c06d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:39.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f519080e640 1 -- 172.21.15.172:0/2730177443 shutdown_connections 2024-03-16T19:30:39.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.350+0000 7f519080e640 1 -- 172.21.15.172:0/2730177443 wait complete. 2024-03-16T19:30:39.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.352+0000 7f519080e640 1 Processor -- start 2024-03-16T19:30:39.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.352+0000 7f519080e640 1 -- start start 2024-03-16T19:30:39.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.352+0000 7f519080e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f518c0b3630 0x7f518c147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.352+0000 7f519080e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f518c0cb5e0 0x7f518c0c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.352+0000 7f519080e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f518c154ed0 0x7f518c1f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.352+0000 7f519080e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c14f000 con 0x7f518c154ed0 2024-03-16T19:30:39.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.352+0000 7f519080e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c0c4bb0 con 0x7f518c0cb5e0 2024-03-16T19:30:39.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.352+0000 7f519080e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518c1f04c0 con 0x7f518c0b3630 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518ad9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f518c154ed0 0x7f518c1f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518ad9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f518c154ed0 0x7f518c1f0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57374/0 (socket says 172.21.15.172:57374) 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518ad9d640 1 -- 172.21.15.172:0/1957589706 learned_addr learned my addr 172.21.15.172:0/1957589706 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518a59c640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f518c0b3630 0x7f518c147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f5189d9b640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f518c0cb5e0 0x7f518c0c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518ad9d640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f518c0b3630 msgr2=0x7f518c147840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518ad9d640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f518c0b3630 0x7f518c147840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518ad9d640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f518c0cb5e0 msgr2=0x7f518c0c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518ad9d640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f518c0cb5e0 0x7f518c0c64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518ad9d640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5180009a70 con 0x7f518c154ed0 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f518ad9d640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f518c154ed0 0x7f518c1f0bf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f517c00c430 tx=0x7f517c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c005ce0 con 0x7f518c154ed0 2024-03-16T19:30:39.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c005e40 con 0x7f518c154ed0 2024-03-16T19:30:39.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c1f4730 con 0x7f518c154ed0 2024-03-16T19:30:39.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c007310 con 0x7f518c154ed0 2024-03-16T19:30:39.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.353+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c14ed20 con 0x7f518c154ed0 2024-03-16T19:30:39.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.354+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f518c068d20 con 0x7f518c154ed0 2024-03-16T19:30:39.338 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.355+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f517c007470 con 0x7f518c154ed0 2024-03-16T19:30:39.338 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.355+0000 7f517affd640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516c06b3b0 0x7f516c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.338 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.355+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f517c089230 con 0x7f518c154ed0 2024-03-16T19:30:39.338 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.355+0000 7f518a59c640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516c06b3b0 0x7f516c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.355+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v50) v1 ==== 54+0+16 (secure 0 0 0) 0x7f517c090070 con 0x7f518c154ed0 2024-03-16T19:30:39.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.355+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f518c1f0630 0x7f518c0d0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.355+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f518c0b3f00 con 0x7f518c1f0630 2024-03-16T19:30:39.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.356+0000 7f518a59c640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516c06b3b0 0x7f516c06d870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f51840016e0 tx=0x7f5184009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.356+0000 7f5189d9b640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f518c1f0630 0x7f518c0d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.356+0000 7f5189d9b640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f518c1f0630 0x7f518c0d0260 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.356+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5180003a10 con 0x7f518c1f0630 2024-03-16T19:30:39.339 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:39.356+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:39.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.371+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f518c0681c0 con 0x7f518c1f0630 2024-03-16T19:30:39.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.373+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f518c0681c0 con 0x7f518c1f0630 2024-03-16T19:30:39.354 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:39.355 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:39.355 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.373+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f518c0c6a00 0x7f518c0c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.355 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.373+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f518c0d10c0 con 0x7f518c0c6a00 2024-03-16T19:30:39.355 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.373+0000 7f518a59c640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f518c0c6a00 0x7f518c0c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.355 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.373+0000 7f518a59c640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f518c0c6a00 0x7f518c0c7860 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.355 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.374+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f518c0d10c0 con 0x7f518c0c6a00 2024-03-16T19:30:39.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.389+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f518c0cbf90 con 0x7f518c0c6a00 2024-03-16T19:30:39.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.391+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f518c0cbf90 con 0x7f518c0c6a00 2024-03-16T19:30:39.372 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:39.373 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:39.373 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.391+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f518c1f1130 0x7f518c0c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.373 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.391+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f518c158a00 con 0x7f518c1f1130 2024-03-16T19:30:39.373 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.391+0000 7f5189d9b640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f518c1f1130 0x7f518c0c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.373 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.391+0000 7f5189d9b640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f518c1f1130 0x7f518c0c82b0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.373 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.392+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f518c158a00 con 0x7f518c1f1130 2024-03-16T19:30:39.374 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:39.374 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:39.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:39.374 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:39.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.374 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 68.036715435999994, 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:39.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:39.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:39.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:39.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 68.035488013999995, 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:39.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:39.383 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 68.035085308000006, 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.384 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:39.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:39.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 68.034476557000005, 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:39.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:39.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:39.391 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 68.034032687999996, 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:39.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:39.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:39.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 68.033889912000006, 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:39.396 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:39.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.399 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 68.033875370999993, 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:39.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:39.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:39.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 68.030997675999998, 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:39.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:39.407 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.408 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:39.408 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.408 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:39.408 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:39.408 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:39.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.406+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f518c061c00 con 0x7f518c1f1130 2024-03-16T19:30:39.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.407+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f518c061c00 con 0x7f518c1f1130 2024-03-16T19:30:39.409 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:39.410 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:39.411 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:39.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.407+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f518c0626a0 0x7f518c0ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.407+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f518c0ce5d0 con 0x7f518c0626a0 2024-03-16T19:30:39.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.408+0000 7f518ad9d640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f518c0626a0 0x7f518c0ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.408+0000 7f518ad9d640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f518c0626a0 0x7f518c0ce090 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.408+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f518c0ce5d0 con 0x7f518c0626a0 2024-03-16T19:30:39.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.422+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f518c0ceaa0 con 0x7f518c0626a0 2024-03-16T19:30:39.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.424+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f518c0ceaa0 con 0x7f518c0626a0 2024-03-16T19:30:39.411 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:39.412 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:39.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.424+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f518c02bde0 0x7f518c02c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.424+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f518c02c740 con 0x7f518c02bde0 2024-03-16T19:30:39.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.424+0000 7f518a59c640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f518c02bde0 0x7f518c02c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.425+0000 7f518a59c640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f518c02bde0 0x7f518c02c200 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.425+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f518c02c740 con 0x7f518c02bde0 2024-03-16T19:30:39.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.439+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f518c02c740 con 0x7f518c02bde0 2024-03-16T19:30:39.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.441+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f518c02c740 con 0x7f518c02bde0 2024-03-16T19:30:39.422 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:39.423 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:39.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.441+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f518c02ce50 0x7f518c206dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.441+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f518c02d2b0 con 0x7f518c02ce50 2024-03-16T19:30:39.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.441+0000 7f5189d9b640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f518c02ce50 0x7f518c206dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.442+0000 7f5189d9b640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f518c02ce50 0x7f518c206dc0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.442+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f518c02d2b0 con 0x7f518c02ce50 2024-03-16T19:30:39.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.456+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f518c02d2b0 con 0x7f518c02ce50 2024-03-16T19:30:39.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.458+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f518c02d2b0 con 0x7f518c02ce50 2024-03-16T19:30:39.439 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:39.440 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:39.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.458+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f518c207810 0x7f518c209c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.458+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f518c20c9d0 con 0x7f518c207810 2024-03-16T19:30:39.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.458+0000 7f518ad9d640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f518c207810 0x7f518c209c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.459+0000 7f518ad9d640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f518c207810 0x7f518c209c00 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.440 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.459+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f518c20c9d0 con 0x7f518c207810 2024-03-16T19:30:39.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.473+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f518c20a140 con 0x7f518c207810 2024-03-16T19:30:39.456 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:39.476+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:39.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.475+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f518c20a140 con 0x7f518c207810 2024-03-16T19:30:39.457 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:39.457 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:39.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.475+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f518c20a820 0x7f518c20cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.475+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f518c20f9e0 con 0x7f518c20a820 2024-03-16T19:30:39.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.476+0000 7f518a59c640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f518c20a820 0x7f518c20cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.476+0000 7f518a59c640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f518c20a820 0x7f518c20cc10 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.477+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f518c20f9e0 con 0x7f518c20a820 2024-03-16T19:30:39.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.491+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f518c20d150 con 0x7f518c20a820 2024-03-16T19:30:39.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.492+0000 7f517affd640 1 -- 172.21.15.172:0/1957589706 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f518c20d150 con 0x7f518c20a820 2024-03-16T19:30:39.474 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f518c1f0630 msgr2=0x7f518c0d0260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f518c1f0630 0x7f518c0d0260 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f518c0c6a00 msgr2=0x7f518c0c7860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f518c0c6a00 0x7f518c0c7860 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f518c1f1130 msgr2=0x7f518c0c82b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f518c1f1130 0x7f518c0c82b0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f518c0626a0 msgr2=0x7f518c0ce090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f518c0626a0 0x7f518c0ce090 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f518ad9d640 1 -- 172.21.15.172:0/1957589706 reap_dead start 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f518c02bde0 msgr2=0x7f518c02c200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f518c02bde0 0x7f518c02c200 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f518c02ce50 msgr2=0x7f518c206dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f518ad9d640 1 -- 172.21.15.172:0/1957589706 reap_dead start 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f518c02ce50 0x7f518c206dc0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f518c207810 msgr2=0x7f518c209c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f518c207810 0x7f518c209c00 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f518c20a820 msgr2=0x7f518c20cc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f518c20a820 0x7f518c20cc10 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516c06b3b0 msgr2=0x7f516c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f516c06b3b0 0x7f516c06d870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f51840016e0 tx=0x7f5184009040 comp rx=0 tx=0).stop 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f518c154ed0 msgr2=0x7f518c1f0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 --2- 172.21.15.172:0/1957589706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f518c154ed0 0x7f518c1f0bf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f517c00c430 tx=0x7f517c00c8f0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f518ad9d640 1 -- 172.21.15.172:0/1957589706 reap_dead start 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 shutdown_connections 2024-03-16T19:30:39.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 >> 172.21.15.172:0/1957589706 conn(0x7f518c06bf60 msgr2=0x7f518c0b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:39.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 shutdown_connections 2024-03-16T19:30:39.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.493+0000 7f519080e640 1 -- 172.21.15.172:0/1957589706 wait complete. 2024-03-16T19:30:39.520 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:39.520 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:39.529 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:39.538 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:39.557+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:39.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.604+0000 7fa39bfff640 1 -- 172.21.15.172:0/3522973189 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v51) v1 ==== 113+0+0 (secure 0 0 0) 0x7fa3ac058980 con 0x7fa3b4152580 2024-03-16T19:30:39.586 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.2 to 1 (10000) 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa3a006b3b0 msgr2=0x7fa3a006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa3a006b3b0 0x7fa3a006d870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4036000 tx=0x7fa3a4008030 comp rx=0 tx=0).stop 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 msgr2=0x7fa3b4147ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 0x7fa3b4147ac0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00efc0 tx=0x7fa3ac00c490 comp rx=0 tx=0).stop 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 shutdown_connections 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3b4152580 0x7fa3b4147ac0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b40c9ec0 0x7fa3b414d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa3a006b3b0 0x7fa3a006d870 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 --2- 172.21.15.172:0/3522973189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b40c9aa0 0x7fa3b40aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 >> 172.21.15.172:0/3522973189 conn(0x7fa3b406bfe0 msgr2=0x7fa3b406af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:39.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 shutdown_connections 2024-03-16T19:30:39.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.607+0000 7fa3ba0a7640 1 -- 172.21.15.172:0/3522973189 wait complete. 2024-03-16T19:30:39.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:30:39.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-16T19:30:39.642 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:39.661+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:39.680 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:39.680 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:39.690 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:39.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.721+0000 7f09c7d1e640 1 Processor -- start 2024-03-16T19:30:39.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.721+0000 7f09c7d1e640 1 -- start start 2024-03-16T19:30:39.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.722+0000 7f09c7d1e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f09c01575b0 0x7f09c0157a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.722+0000 7f09c7d1e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c0151d60 con 0x7f09c00c7a40 2024-03-16T19:30:39.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.722+0000 7f09c7d1e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c00727b0 con 0x7f09c0151980 2024-03-16T19:30:39.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.722+0000 7f09c7d1e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c0073130 con 0x7f09c01575b0 2024-03-16T19:30:39.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.722+0000 7f09c52b9640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f09c00c7a40 0x7f09c00c2bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45066/0 (socket says 172.21.15.172:45066) 2024-03-16T19:30:39.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.722+0000 7f09c52b9640 1 -- 172.21.15.172:0/1894435700 learned_addr learned my addr 172.21.15.172:0/1894435700 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:39.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.722+0000 7f09c62bb640 1 --2- 172.21.15.172:0/1894435700 >> v2:172.21.15.174:3300/0 conn(0x7f09c01575b0 0x7f09c0157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.722+0000 7f09c4ab8640 1 -- 172.21.15.172:0/1894435700 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229221813 0 0) 0x7f09c0151d60 con 0x7f09c00c7a40 2024-03-16T19:30:39.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.723+0000 7f09c4ab8640 1 -- 172.21.15.172:0/1894435700 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ac003520 con 0x7f09c00c7a40 2024-03-16T19:30:39.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.723+0000 7f09c62bb640 1 -- 172.21.15.172:0/1894435700 >> v1:172.21.15.174:6789/0 conn(0x7f09c0151980 legacy=0x7f09c00c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.723+0000 7f09c4ab8640 1 -- 172.21.15.172:0/1894435700 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138407214 0 0) 0x7f09c00727b0 con 0x7f09c0151980 2024-03-16T19:30:39.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.723+0000 7f09c62bb640 1 -- 172.21.15.172:0/1894435700 >> v1:172.21.15.172:6789/0 conn(0x7f09c00c7a40 legacy=0x7f09c00c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.723+0000 7f09c62bb640 1 -- 172.21.15.172:0/1894435700 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c00c8ce0 con 0x7f09c01575b0 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.723+0000 7f09c62bb640 1 --2- 172.21.15.172:0/1894435700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09c01575b0 0x7f09c0157a00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009d90 tx=0x7f09bc02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61c48f3bd81f8ce server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c4ab8640 1 -- 172.21.15.172:0/1894435700 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc02cd00 con 0x7f09c01575b0 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c4ab8640 1 -- 172.21.15.172:0/1894435700 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc02ce60 con 0x7f09c01575b0 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c4ab8640 1 -- 172.21.15.172:0/1894435700 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc034660 con 0x7f09c01575b0 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c7d1e640 1 -- 172.21.15.172:0/1894435700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09c01575b0 msgr2=0x7f09c0157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c7d1e640 1 --2- 172.21.15.172:0/1894435700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09c01575b0 0x7f09c0157a00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009d90 tx=0x7f09bc02bf50 comp rx=0 tx=0).stop 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c7d1e640 1 -- 172.21.15.172:0/1894435700 shutdown_connections 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c7d1e640 1 --2- 172.21.15.172:0/1894435700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09c01575b0 0x7f09c0157a00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c7d1e640 1 -- 172.21.15.172:0/1894435700 >> 172.21.15.172:0/1894435700 conn(0x7f09c00cd040 msgr2=0x7f09c00cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c7d1e640 1 -- 172.21.15.172:0/1894435700 shutdown_connections 2024-03-16T19:30:39.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.724+0000 7f09c7d1e640 1 -- 172.21.15.172:0/1894435700 wait complete. 2024-03-16T19:30:39.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.725+0000 7f09c7d1e640 1 Processor -- start 2024-03-16T19:30:39.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.725+0000 7f09c7d1e640 1 -- start start 2024-03-16T19:30:39.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c7d1e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09c00c7a40 0x7f09c01ef760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c7d1e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09c0151980 0x7f09c0148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c7d1e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c01575b0 0x7f09c0148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c7d1e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c01477e0 con 0x7f09c01575b0 2024-03-16T19:30:39.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c7d1e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0146c70 con 0x7f09c00c7a40 2024-03-16T19:30:39.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c7d1e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c014aec0 con 0x7f09c0151980 2024-03-16T19:30:39.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c62bb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c01575b0 0x7f09c0148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c52b9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09c0151980 0x7f09c0148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c62bb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c01575b0 0x7f09c0148c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57376/0 (socket says 172.21.15.172:57376) 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c62bb640 1 -- 172.21.15.172:0/2463991741 learned_addr learned my addr 172.21.15.172:0/2463991741 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.726+0000 7f09c5aba640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09c00c7a40 0x7f09c01ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09c62bb640 1 -- 172.21.15.172:0/2463991741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09c0151980 msgr2=0x7f09c0148700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09c62bb640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09c0151980 0x7f09c0148700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09c62bb640 1 -- 172.21.15.172:0/2463991741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09c00c7a40 msgr2=0x7f09c01ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09c62bb640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09c00c7a40 0x7f09c01ef760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09c62bb640 1 -- 172.21.15.172:0/2463991741 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc009a70 con 0x7f09c01575b0 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09c62bb640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c01575b0 0x7f09c0148c40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f09bc037040 tx=0x7f09bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09b67fc640 1 -- 172.21.15.172:0/2463991741 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc03d600 con 0x7f09c01575b0 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09b67fc640 1 -- 172.21.15.172:0/2463991741 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc03d780 con 0x7f09c01575b0 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09b67fc640 1 -- 172.21.15.172:0/2463991741 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc03db30 con 0x7f09c01575b0 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c01f0400 con 0x7f09c01575b0 2024-03-16T19:30:39.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.727+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c014a870 con 0x7f09c01575b0 2024-03-16T19:30:39.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.728+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c0075a70 con 0x7f09c01575b0 2024-03-16T19:30:39.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.729+0000 7f09b67fc640 1 -- 172.21.15.172:0/2463991741 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f09bc03b040 con 0x7f09c01575b0 2024-03-16T19:30:39.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.729+0000 7f09b67fc640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f09a406b3b0 0x7f09a406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.729+0000 7f09c5aba640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f09a406b3b0 0x7f09a406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.729+0000 7f09b67fc640 1 -- 172.21.15.172:0/2463991741 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f09bc0aba30 con 0x7f09c01575b0 2024-03-16T19:30:39.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.730+0000 7f09c5aba640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f09a406b3b0 0x7f09a406d870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f09b8000f60 tx=0x7f09b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.734+0000 7f09b67fc640 1 -- 172.21.15.172:0/2463991741 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09bc0766c0 con 0x7f09c01575b0 2024-03-16T19:30:39.740 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:39.760+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:39.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.768+0000 7f858bd5c640 1 Processor -- start 2024-03-16T19:30:39.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.768+0000 7f858bd5c640 1 -- start start 2024-03-16T19:30:39.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f858bd5c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f85840ce010 0x7f85840c3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f858bd5c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858415ca60 con 0x7f85840c8320 2024-03-16T19:30:39.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f858bd5c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85840c8e00 con 0x7f858406f3d0 2024-03-16T19:30:39.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f858bd5c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85840cba80 con 0x7f85840ce010 2024-03-16T19:30:39.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f8589af8640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f85840ce010 0x7f85840c3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f85892f7640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f85840c8320 0x7f858415bc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45076/0 (socket says 172.21.15.172:45076) 2024-03-16T19:30:39.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f85892f7640 1 -- 172.21.15.172:0/3777321665 learned_addr learned my addr 172.21.15.172:0/3777321665 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:39.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f8589af8640 1 -- 172.21.15.172:0/3777321665 >> v1:172.21.15.174:6789/0 conn(0x7f858406f3d0 legacy=0x7f858415c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f8589af8640 1 -- 172.21.15.172:0/3777321665 >> v1:172.21.15.172:6789/0 conn(0x7f85840c8320 legacy=0x7f858415bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.769+0000 7f8589af8640 1 -- 172.21.15.172:0/3777321665 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858406e730 con 0x7f85840ce010 2024-03-16T19:30:39.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.770+0000 7f8589af8640 1 --2- 172.21.15.172:0/3777321665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 0x7f85840c3320 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f85800097a0 tx=0x7f858002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63aad466ba13879b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.770+0000 7f8588af6640 1 -- 172.21.15.172:0/3777321665 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580030610 con 0x7f85840ce010 2024-03-16T19:30:39.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.770+0000 7f8588af6640 1 -- 172.21.15.172:0/3777321665 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8580030770 con 0x7f85840ce010 2024-03-16T19:30:39.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.770+0000 7f8588af6640 1 -- 172.21.15.172:0/3777321665 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580030b00 con 0x7f85840ce010 2024-03-16T19:30:39.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.771+0000 7f858bd5c640 1 -- 172.21.15.172:0/3777321665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 msgr2=0x7f85840c3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.771+0000 7f858bd5c640 1 --2- 172.21.15.172:0/3777321665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 0x7f85840c3320 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f85800097a0 tx=0x7f858002b960 comp rx=0 tx=0).stop 2024-03-16T19:30:39.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.771+0000 7f858bd5c640 1 -- 172.21.15.172:0/3777321665 shutdown_connections 2024-03-16T19:30:39.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.771+0000 7f858bd5c640 1 --2- 172.21.15.172:0/3777321665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 0x7f85840c3320 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.771+0000 7f858bd5c640 1 -- 172.21.15.172:0/3777321665 >> 172.21.15.172:0/3777321665 conn(0x7f8584066c20 msgr2=0x7f8584067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:39.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.771+0000 7f858bd5c640 1 -- 172.21.15.172:0/3777321665 shutdown_connections 2024-03-16T19:30:39.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.771+0000 7f858bd5c640 1 -- 172.21.15.172:0/3777321665 wait complete. 2024-03-16T19:30:39.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.772+0000 7f858bd5c640 1 Processor -- start 2024-03-16T19:30:39.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.773+0000 7f858bd5c640 1 -- start start 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858bd5c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858406f3d0 0x7f8584158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858bd5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85840c8320 0x7f858414ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858bd5c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 0x7f85841572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858bd5c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85841e4c10 con 0x7f85840ce010 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858bd5c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85840cce20 con 0x7f858406f3d0 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858bd5c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85841e5570 con 0x7f85840c8320 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858a2f9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 0x7f85841572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f8589af8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858406f3d0 0x7f8584158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f8589af8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858406f3d0 0x7f8584158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40470/0 (socket says 172.21.15.172:40470) 2024-03-16T19:30:39.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f8589af8640 1 -- 172.21.15.172:0/2968948833 learned_addr learned my addr 172.21.15.172:0/2968948833 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858a2f9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 0x7f85841572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57402/0 (socket says 172.21.15.172:57402) 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f85892f7640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85840c8320 0x7f858414ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858a2f9640 1 -- 172.21.15.172:0/2968948833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85840c8320 msgr2=0x7f858414ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858a2f9640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85840c8320 0x7f858414ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858a2f9640 1 -- 172.21.15.172:0/2968948833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858406f3d0 msgr2=0x7f8584158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858a2f9640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858406f3d0 0x7f8584158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858a2f9640 1 -- 172.21.15.172:0/2968948833 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8580009400 con 0x7f85840ce010 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f8589af8640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858406f3d0 0x7f8584158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f858a2f9640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 0x7f85841572e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f857c00c700 tx=0x7f857c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.774+0000 7f857a7fc640 1 -- 172.21.15.172:0/2968948833 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c003b70 con 0x7f85840ce010 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.775+0000 7f857a7fc640 1 -- 172.21.15.172:0/2968948833 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857c003cd0 con 0x7f85840ce010 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.775+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85841e5850 con 0x7f85840ce010 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.775+0000 7f857a7fc640 1 -- 172.21.15.172:0/2968948833 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c005310 con 0x7f85840ce010 2024-03-16T19:30:39.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.775+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85841e45c0 con 0x7f85840ce010 2024-03-16T19:30:39.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.776+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85840cd9c0 con 0x7f85840ce010 2024-03-16T19:30:39.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.776+0000 7f857a7fc640 1 -- 172.21.15.172:0/2968948833 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f857c005470 con 0x7f85840ce010 2024-03-16T19:30:39.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.776+0000 7f857a7fc640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f856806b3b0 0x7f856806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:39.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.777+0000 7f857a7fc640 1 -- 172.21.15.172:0/2968948833 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f857c089810 con 0x7f85840ce010 2024-03-16T19:30:39.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.777+0000 7f8589af8640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f856806b3b0 0x7f856806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:39.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.777+0000 7f8589af8640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f856806b3b0 0x7f856806d870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f858002b9a0 tx=0x7f8580008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:39.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.781+0000 7f857a7fc640 1 -- 172.21.15.172:0/2968948833 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f857c054410 con 0x7f85840ce010 2024-03-16T19:30:39.797 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:39.797 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:39.806 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:39.847 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:39.867+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:39.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.925+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f09c0156430 con 0x7f09c01575b0 2024-03-16T19:30:39.941 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:39.960+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:39.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:39.970+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f85840681c0 con 0x7f85840ce010 2024-03-16T19:30:39.953 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:39.953 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:39.962 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:40.042 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:40.062+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:40.069 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:40.070 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:40.078 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:40.162 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:40.182+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:40.245 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:40.265+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:40.266 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:40.266 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:40.277 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:40.362 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:40.382+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:40.442 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:40.442 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:40.447 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:40.467+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:40.451 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:40.547 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:40.567+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:40.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.613+0000 7f09b67fc640 1 -- 172.21.15.172:0/2463991741 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v52) v1 ==== 101+0+0 (secure 0 0 0) 0x7f09bc04d090 con 0x7f09c01575b0 2024-03-16T19:30:40.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.613+0000 7f857a7fc640 1 -- 172.21.15.172:0/2968948833 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v52) v1 ==== 113+0+0 (secure 0 0 0) 0x7f857c057ad0 con 0x7f85840ce010 2024-03-16T19:30:40.595 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:30:40.595 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.3 to 1 (10000) 2024-03-16T19:30:40.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f856806b3b0 msgr2=0x7f856806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:40.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f856806b3b0 0x7f856806d870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f858002b9a0 tx=0x7f8580008030 comp rx=0 tx=0).stop 2024-03-16T19:30:40.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 msgr2=0x7f85841572e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:40.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 0x7f85841572e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f857c00c700 tx=0x7f857c00cbc0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 shutdown_connections 2024-03-16T19:30:40.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f09a406b3b0 msgr2=0x7f09a406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:40.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f09a406b3b0 0x7f09a406d870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f09b8000f60 tx=0x7f09b8009040 comp rx=0 tx=0).stop 2024-03-16T19:30:40.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c01575b0 msgr2=0x7f09c0148c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:40.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c01575b0 0x7f09c0148c40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f09bc037040 tx=0x7f09bc004a40 comp rx=0 tx=0).stop 2024-03-16T19:30:40.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f85840ce010 0x7f85841572e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f856806b3b0 0x7f856806d870 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85840c8320 0x7f858414ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 --2- 172.21.15.172:0/2968948833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858406f3d0 0x7f8584158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 >> 172.21.15.172:0/2968948833 conn(0x7f8584066c20 msgr2=0x7f85840cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:40.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 shutdown_connections 2024-03-16T19:30:40.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 shutdown_connections 2024-03-16T19:30:40.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f09c01575b0 0x7f09c0148c40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f09a406b3b0 0x7f09a406d870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09c0151980 0x7f09c0148700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 --2- 172.21.15.172:0/2463991741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09c00c7a40 0x7f09c01ef760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 >> 172.21.15.172:0/2463991741 conn(0x7f09c00cd040 msgr2=0x7f09c00c2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:40.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f858bd5c640 1 -- 172.21.15.172:0/2968948833 wait complete. 2024-03-16T19:30:40.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 shutdown_connections 2024-03-16T19:30:40.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.616+0000 7f09c7d1e640 1 -- 172.21.15.172:0/2463991741 wait complete. 2024-03-16T19:30:40.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-16T19:30:40.616 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:40.616 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:40.628 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:40.670 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:40.690+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:40.748 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:40.768+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:40.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.774+0000 7f134150d640 1 Processor -- start 2024-03-16T19:30:40.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.774+0000 7f134150d640 1 -- start start 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f134150d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f133c154e00 0x7f133c06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f134150d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f133c06b500 0x7f133c165260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f134150d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c15b990 con 0x7f133c0b5f10 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f134150d640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133c15b570 con 0x7f133c06b500 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f134150d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133c1551e0 con 0x7f133c154e00 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f133b7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f133c06b500 0x7f133c165260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f133a7fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f133c0b5f10 0x7f133c0caed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45080/0 (socket says 172.21.15.172:45080) 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f133a7fc640 1 -- 172.21.15.172:0/1085944569 learned_addr learned my addr 172.21.15.172:0/1085944569 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f133b7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f133c06b500 0x7f133c165260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57416/0 (socket says 172.21.15.172:57416) 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f133affd640 1 --2- 172.21.15.172:0/1085944569 >> v2:172.21.15.174:3300/0 conn(0x7f133c154e00 0x7f133c06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f133b7fe640 1 -- 172.21.15.172:0/1085944569 >> v2:172.21.15.174:3300/0 conn(0x7f133c154e00 msgr2=0x7f133c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f133b7fe640 1 --2- 172.21.15.172:0/1085944569 >> v2:172.21.15.174:3300/0 conn(0x7f133c154e00 0x7f133c06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f133b7fe640 1 -- 172.21.15.172:0/1085944569 >> v1:172.21.15.172:6789/0 conn(0x7f133c0b5f10 legacy=0x7f133c0caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.775+0000 7f133b7fe640 1 -- 172.21.15.172:0/1085944569 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0cc090 con 0x7f133c06b500 2024-03-16T19:30:40.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.776+0000 7f133b7fe640 1 --2- 172.21.15.172:0/1085944569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c06b500 0x7f133c165260 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f133c0cd910 tx=0x7f133402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e291cc7819ca3d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.776+0000 7f1339ffb640 1 -- 172.21.15.172:0/1085944569 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133402cd00 con 0x7f133c06b500 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.776+0000 7f1339ffb640 1 -- 172.21.15.172:0/1085944569 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133402ce60 con 0x7f133c06b500 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.776+0000 7f1339ffb640 1 -- 172.21.15.172:0/1085944569 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334034660 con 0x7f133c06b500 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.776+0000 7f134150d640 1 -- 172.21.15.172:0/1085944569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c06b500 msgr2=0x7f133c165260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.776+0000 7f134150d640 1 --2- 172.21.15.172:0/1085944569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c06b500 0x7f133c165260 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f133c0cd910 tx=0x7f133402bf50 comp rx=0 tx=0).stop 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.777+0000 7f134150d640 1 -- 172.21.15.172:0/1085944569 shutdown_connections 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.777+0000 7f134150d640 1 --2- 172.21.15.172:0/1085944569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c06b500 0x7f133c165260 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.777+0000 7f134150d640 1 --2- 172.21.15.172:0/1085944569 >> v2:172.21.15.174:3300/0 conn(0x7f133c154e00 0x7f133c06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.777+0000 7f134150d640 1 -- 172.21.15.172:0/1085944569 >> 172.21.15.172:0/1085944569 conn(0x7f133c066c20 msgr2=0x7f133c067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.777+0000 7f134150d640 1 -- 172.21.15.172:0/1085944569 shutdown_connections 2024-03-16T19:30:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.777+0000 7f134150d640 1 -- 172.21.15.172:0/1085944569 wait complete. 2024-03-16T19:30:40.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.778+0000 7f134150d640 1 Processor -- start 2024-03-16T19:30:40.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.778+0000 7f134150d640 1 -- start start 2024-03-16T19:30:40.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.778+0000 7f134150d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f133c06b500 0x7f133c14bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:40.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f134150d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f133c0b5f10 0x7f133c14fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:40.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f134150d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c154e00 0x7f133c1500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f134150d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c1f41a0 con 0x7f133c154e00 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f134150d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c151290 con 0x7f133c06b500 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f134150d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133c146cb0 con 0x7f133c0b5f10 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c154e00 0x7f133c1500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c154e00 0x7f133c1500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57420/0 (socket says 172.21.15.172:57420) 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f133c06b500 0x7f133c14bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133b7fe640 1 -- 172.21.15.172:0/2468528666 learned_addr learned my addr 172.21.15.172:0/2468528666 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133a7fc640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f133c0b5f10 0x7f133c14fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133b7fe640 1 -- 172.21.15.172:0/2468528666 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f133c0b5f10 msgr2=0x7f133c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133b7fe640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f133c0b5f10 0x7f133c14fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133b7fe640 1 -- 172.21.15.172:0/2468528666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f133c06b500 msgr2=0x7f133c14bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133b7fe640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f133c06b500 0x7f133c14bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133b7fe640 1 -- 172.21.15.172:0/2468528666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1334009a70 con 0x7f133c154e00 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f133b7fe640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c154e00 0x7f133c1500e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1334009e80 tx=0x7f1334004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:40.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f13237fe640 1 -- 172.21.15.172:0/2468528666 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133403d600 con 0x7f133c154e00 2024-03-16T19:30:40.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f13237fe640 1 -- 172.21.15.172:0/2468528666 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133403d780 con 0x7f133c154e00 2024-03-16T19:30:40.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c14c820 con 0x7f133c154e00 2024-03-16T19:30:40.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f13237fe640 1 -- 172.21.15.172:0/2468528666 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133403db30 con 0x7f133c154e00 2024-03-16T19:30:40.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.779+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c151550 con 0x7f133c154e00 2024-03-16T19:30:40.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.780+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c1f4310 con 0x7f133c154e00 2024-03-16T19:30:40.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.781+0000 7f13237fe640 1 -- 172.21.15.172:0/2468528666 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f133403b040 con 0x7f133c154e00 2024-03-16T19:30:40.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.781+0000 7f13237fe640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f132806b3b0 0x7f132806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:40.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.781+0000 7f133affd640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f132806b3b0 0x7f132806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:40.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.781+0000 7f13237fe640 1 -- 172.21.15.172:0/2468528666 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f13340ac100 con 0x7f133c154e00 2024-03-16T19:30:40.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.782+0000 7f133affd640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f132806b3b0 0x7f132806d870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f132c009aa0 tx=0x7f132c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:40.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.786+0000 7f13237fe640 1 -- 172.21.15.172:0/2468528666 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1334076e20 con 0x7f133c154e00 2024-03-16T19:30:40.797 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:40.797 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:40.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:40.849 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:40.869+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:40.913 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:40.913 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:40.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:40.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:40.977+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f133c0cc710 con 0x7f133c154e00 2024-03-16T19:30:40.959 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:40.978+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:41.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:41.051 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:41.071+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:41.152 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:41.172+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:41.184 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:41.185 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:41.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:41.277 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:41.296+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:41.340 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:41.340 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:41.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:41.354 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:41.374+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:41.455 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:41.474+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:41.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:41.500 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:41.500 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:41.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:41.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.613+0000 7f2543f81640 1 Processor -- start 2024-03-16T19:30:41.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.613+0000 7f2543f81640 1 -- start start 2024-03-16T19:30:41.594 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:41.613+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:41.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.614+0000 7f2543f81640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f253c070af0 0x7f253c159430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.614+0000 7f2543f81640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f253c070ed0 0x7f253c0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.614+0000 7f2543f81640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253c159970 con 0x7f253c0b0e60 2024-03-16T19:30:41.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.614+0000 7f2543f81640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253c15ce70 con 0x7f253c070ed0 2024-03-16T19:30:41.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.614+0000 7f2543f81640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253c15bfc0 con 0x7f253c070af0 2024-03-16T19:30:41.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.614+0000 7f2541d1d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f253c070af0 0x7f253c159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.614+0000 7f2541d1d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f253c070af0 0x7f253c159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40504/0 (socket says 172.21.15.172:40504) 2024-03-16T19:30:41.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.614+0000 7f2541d1d640 1 -- 172.21.15.172:0/673243975 learned_addr learned my addr 172.21.15.172:0/673243975 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:41.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.614+0000 7f254151c640 1 --2- 172.21.15.172:0/673243975 >> v2:172.21.15.172:3300/0 conn(0x7f253c070ed0 0x7f253c0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.615+0000 7f2541d1d640 1 -- 172.21.15.172:0/673243975 >> v2:172.21.15.172:3300/0 conn(0x7f253c070ed0 msgr2=0x7f253c0c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:41.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.615+0000 7f2541d1d640 1 --2- 172.21.15.172:0/673243975 >> v2:172.21.15.172:3300/0 conn(0x7f253c070ed0 0x7f253c0c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.615+0000 7f2541d1d640 1 -- 172.21.15.172:0/673243975 >> v1:172.21.15.174:6789/0 conn(0x7f253c0b0e60 legacy=0x7f253c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.615+0000 7f2541d1d640 1 -- 172.21.15.172:0/673243975 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c15c260 con 0x7f253c070af0 2024-03-16T19:30:41.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.615+0000 7f2541d1d640 1 --2- 172.21.15.172:0/673243975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f253c070af0 0x7f253c159430 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f253c0cf330 tx=0x7f253802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7b61a770a31f927 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.616+0000 7f2540d1b640 1 -- 172.21.15.172:0/673243975 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538030860 con 0x7f253c070af0 2024-03-16T19:30:41.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.616+0000 7f2540d1b640 1 -- 172.21.15.172:0/673243975 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25380309c0 con 0x7f253c070af0 2024-03-16T19:30:41.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.616+0000 7f2540d1b640 1 -- 172.21.15.172:0/673243975 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538030d50 con 0x7f253c070af0 2024-03-16T19:30:41.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.617+0000 7f2543f81640 1 -- 172.21.15.172:0/673243975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f253c070af0 msgr2=0x7f253c159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.617+0000 7f2543f81640 1 --2- 172.21.15.172:0/673243975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f253c070af0 0x7f253c159430 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f253c0cf330 tx=0x7f253802b9a0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.617+0000 7f2543f81640 1 -- 172.21.15.172:0/673243975 shutdown_connections 2024-03-16T19:30:41.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.617+0000 7f2543f81640 1 --2- 172.21.15.172:0/673243975 >> v2:172.21.15.172:3300/0 conn(0x7f253c070ed0 0x7f253c0c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.617+0000 7f2543f81640 1 --2- 172.21.15.172:0/673243975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f253c070af0 0x7f253c159430 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.617+0000 7f2543f81640 1 -- 172.21.15.172:0/673243975 >> 172.21.15.172:0/673243975 conn(0x7f253c06e880 msgr2=0x7f253c06ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:41.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.617+0000 7f2543f81640 1 -- 172.21.15.172:0/673243975 shutdown_connections 2024-03-16T19:30:41.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.617+0000 7f2543f81640 1 -- 172.21.15.172:0/673243975 wait complete. 2024-03-16T19:30:41.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.618+0000 7f13237fe640 1 -- 172.21.15.172:0/2468528666 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v53) v1 ==== 113+0+0 (secure 0 0 0) 0x7f1334009450 con 0x7f133c154e00 2024-03-16T19:30:41.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.618+0000 7f2543f81640 1 Processor -- start 2024-03-16T19:30:41.600 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.4 to 1 (10000) 2024-03-16T19:30:41.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.618+0000 7f2543f81640 1 -- start start 2024-03-16T19:30:41.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.619+0000 7f2543f81640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f253c070af0 0x7f253c1f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.619+0000 7f2543f81640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f253c070ed0 0x7f253c158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.619+0000 7f2543f81640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f253c0b0e60 0x7f253c14ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.620+0000 7f254151c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f253c070ed0 0x7f253c158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.620+0000 7f254151c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f253c070ed0 0x7f253c158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48054/0 (socket says 172.21.15.172:48054) 2024-03-16T19:30:41.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.620+0000 7f254151c640 1 -- 172.21.15.172:0/1815250739 learned_addr learned my addr 172.21.15.172:0/1815250739 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:41.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.620+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f253c070af0 0x7f253c1f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.620+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f253c0b0e60 0x7f253c14ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.620+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c153510 con 0x7f253c070af0 2024-03-16T19:30:41.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.620+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c1540c0 con 0x7f253c0b0e60 2024-03-16T19:30:41.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.620+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c1efd00 con 0x7f253c070ed0 2024-03-16T19:30:41.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f2541d1d640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f253c070ed0 msgr2=0x7f253c158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f253c070ed0 0x7f253c158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f2541d1d640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f253c0b0e60 msgr2=0x7f253c14ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f253c0b0e60 0x7f253c14ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f2541d1d640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25380094d0 con 0x7f253c070af0 2024-03-16T19:30:41.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f253c070af0 0x7f253c1f2810 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2538033000 tx=0x7f2538004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f132806b3b0 msgr2=0x7f132806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538039dd0 con 0x7f253c070af0 2024-03-16T19:30:41.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f134150d640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f132806b3b0 0x7f132806d870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f132c009aa0 tx=0x7f132c009190 comp rx=0 tx=0).stop 2024-03-16T19:30:41.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c154e00 msgr2=0x7f133c1500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f134150d640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c154e00 0x7f133c1500e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1334009e80 tx=0x7f1334004a40 comp rx=0 tx=0).stop 2024-03-16T19:30:41.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2538043400 con 0x7f253c070af0 2024-03-16T19:30:41.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 shutdown_connections 2024-03-16T19:30:41.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c1f2d50 con 0x7f253c070af0 2024-03-16T19:30:41.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f134150d640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f133c154e00 0x7f133c1500e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f134150d640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f132806b3b0 0x7f132806d870 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.621+0000 7f134150d640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f133c0b5f10 0x7f133c14fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.622+0000 7f134150d640 1 --2- 172.21.15.172:0/2468528666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f133c06b500 0x7f133c14bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.622+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 >> 172.21.15.172:0/2468528666 conn(0x7f133c066c20 msgr2=0x7f133c0c77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:41.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.622+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25380437b0 con 0x7f253c070af0 2024-03-16T19:30:41.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.622+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c0c5aa0 con 0x7f253c070af0 2024-03-16T19:30:41.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.622+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 shutdown_connections 2024-03-16T19:30:41.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.622+0000 7f134150d640 1 -- 172.21.15.172:0/2468528666 wait complete. 2024-03-16T19:30:41.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.623+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f253c068d20 con 0x7f253c070af0 2024-03-16T19:30:41.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.624+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f2538043910 con 0x7f253c070af0 2024-03-16T19:30:41.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.624+0000 7f25327fc640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f252006b3b0 0x7f252006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.624+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f253803e070 con 0x7f253c070af0 2024-03-16T19:30:41.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.624+0000 7f254151c640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f252006b3b0 0x7f252006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.624+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v53) v1 ==== 54+0+16 (secure 0 0 0) 0x7f25380b1240 con 0x7f253c070af0 2024-03-16T19:30:41.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.625+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f253c1efe40 0x7f253c1f0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.625+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253c152930 con 0x7f253c1efe40 2024-03-16T19:30:41.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.625+0000 7f254151c640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f252006b3b0 0x7f252006d870 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f253c1591c0 tx=0x7f252c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.625+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f253c1efe40 0x7f253c1f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.625+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f253c1efe40 0x7f253c1f0220 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.626+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f253c152930 con 0x7f253c1efe40 2024-03-16T19:30:41.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-16T19:30:41.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.641+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f253c15c820 con 0x7f253c1efe40 2024-03-16T19:30:41.624 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.643+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2534001df0 con 0x7f253c1efe40 2024-03-16T19:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:41.625 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:41.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.643+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f253c1f0f40 0x7f253c0cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.643+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253c0681c0 con 0x7f253c1f0f40 2024-03-16T19:30:41.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.644+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f253c1f0f40 0x7f253c0cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.644+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f253c1f0f40 0x7f253c0cd120 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.645+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f253c0681c0 con 0x7f253c1f0f40 2024-03-16T19:30:41.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.662+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f253c0c3a50 con 0x7f253c1f0f40 2024-03-16T19:30:41.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.664+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f253c0c3a50 con 0x7f253c1f0f40 2024-03-16T19:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:41.646 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:41.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.664+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f253c0cdb70 0x7f253c061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.664+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253c0cdf50 con 0x7f253c0cdb70 2024-03-16T19:30:41.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.664+0000 7f254151c640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f253c0cdb70 0x7f253c061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.665+0000 7f254151c640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f253c0cdb70 0x7f253c061930 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.666+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f253c0cdf50 con 0x7f253c0cdb70 2024-03-16T19:30:41.657 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:41.677+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:41.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.683+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f253c0c3490 con 0x7f253c0cdb70 2024-03-16T19:30:41.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.685+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f253c0c3490 con 0x7f253c0cdb70 2024-03-16T19:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:41.667 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:41.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.685+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 0x7f253c02bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.685+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253c157f30 con 0x7f253c062380 2024-03-16T19:30:41.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.686+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 0x7f253c02bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.686+0000 7f254251e640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 msgr2=0x7f253c02bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 24 2024-03-16T19:30:41.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.686+0000 7f254251e640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 msgr2=0x7f253c02bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:30:41.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.686+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 0x7f253c02bde0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:30:41.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.686+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 0x7f253c02bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.334787235999997, 2024-03-16T19:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.334770121999995, 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:41.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.334762842000004, 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:30:41.683 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:41.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.334748974999997, 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:30:41.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.334076139999993, 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:41.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:30:41.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:30:41.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.696 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.331777889999998, 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:30:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:30:41.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:30:41.700 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.331732789, 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:41.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:30:41.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:30:41.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.704 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.331704733999999, 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:30:41.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.708 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.331683319999996, 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:30:41.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:30:41.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:30:41.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.331660968999998, 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.331640570000005, 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:30:41.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.331534578000003, 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.329742866999993, 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.329695365999996, 2024-03-16T19:30:41.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:30:41.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.329456114999999, 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.320957969999995, 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.319618094000006, 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:41.744 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.316627111999999, 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:30:41.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:30:41.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.315996309999999, 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:30:41.749 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:47 2.6 e40/18 ondisk, result = 0)", 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.314513901000002, 2024-03-16T19:30:41.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017677, 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377611+0000", 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9599999999999998e-07 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377618+0000", 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9990000000000002e-06 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377626+0000", 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4249999999999998e-06 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377632+0000", 2024-03-16T19:30:41.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377707+0000", 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4506999999999997e-05 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377714+0000", 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5819999999999998e-06 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377787+0000", 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3232000000000006e-05 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:41.758 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:41.758 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:41.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:41.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.749+0000 7fada922e640 1 Processor -- start 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.749+0000 7fada922e640 1 -- start start 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada922e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fada4154e00 0x7fada406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada922e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada40b5f10 0x7fada40caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada922e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada415b990 con 0x7fada406b500 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada922e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada415b570 con 0x7fada4154e00 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada922e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada41551e0 con 0x7fada40b5f10 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada359e640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fada406b500 0x7fada415df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45088/0 (socket says 172.21.15.172:45088) 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada359e640 1 -- 172.21.15.172:0/189005337 learned_addr learned my addr 172.21.15.172:0/189005337 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada259c640 1 --2- 172.21.15.172:0/189005337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada40b5f10 0x7fada40caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada2d9d640 1 --2- 172.21.15.172:0/189005337 >> v2:172.21.15.174:3300/0 conn(0x7fada4154e00 0x7fada406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada1d9b640 1 -- 172.21.15.172:0/189005337 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717570981 0 0) 0x7fada415b990 con 0x7fada406b500 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada259c640 1 -- 172.21.15.172:0/189005337 >> v2:172.21.15.174:3300/0 conn(0x7fada4154e00 msgr2=0x7fada406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada259c640 1 --2- 172.21.15.172:0/189005337 >> v2:172.21.15.174:3300/0 conn(0x7fada4154e00 0x7fada406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada259c640 1 -- 172.21.15.172:0/189005337 >> v1:172.21.15.172:6789/0 conn(0x7fada406b500 legacy=0x7fada415df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.750+0000 7fada259c640 1 -- 172.21.15.172:0/189005337 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40cc090 con 0x7fada40b5f10 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.751+0000 7fada259c640 1 --2- 172.21.15.172:0/189005337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada40b5f10 0x7fada40caed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fad940097a0 tx=0x7fad9402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=115c228d2a507737 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.751+0000 7fada1d9b640 1 -- 172.21.15.172:0/189005337 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94030610 con 0x7fada40b5f10 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.751+0000 7fada1d9b640 1 -- 172.21.15.172:0/189005337 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad94030770 con 0x7fada40b5f10 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.751+0000 7fada1d9b640 1 -- 172.21.15.172:0/189005337 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94030b00 con 0x7fada40b5f10 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.752+0000 7fada922e640 1 -- 172.21.15.172:0/189005337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada40b5f10 msgr2=0x7fada40caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.752+0000 7fada922e640 1 --2- 172.21.15.172:0/189005337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada40b5f10 0x7fada40caed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fad940097a0 tx=0x7fad9402b960 comp rx=0 tx=0).stop 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.752+0000 7fada922e640 1 -- 172.21.15.172:0/189005337 shutdown_connections 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.752+0000 7fada922e640 1 --2- 172.21.15.172:0/189005337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada40b5f10 0x7fada40caed0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.752+0000 7fada922e640 1 --2- 172.21.15.172:0/189005337 >> v2:172.21.15.174:3300/0 conn(0x7fada4154e00 0x7fada406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.752+0000 7fada922e640 1 -- 172.21.15.172:0/189005337 >> 172.21.15.172:0/189005337 conn(0x7fada4066c20 msgr2=0x7fada4067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.752+0000 7fada922e640 1 -- 172.21.15.172:0/189005337 shutdown_connections 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.752+0000 7fada922e640 1 -- 172.21.15.172:0/189005337 wait complete. 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.753+0000 7fada922e640 1 Processor -- start 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.753+0000 7fada922e640 1 -- start start 2024-03-16T19:30:41.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada922e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada406b500 0x7fada415a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada922e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada40b5f10 0x7fada4143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada922e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada4154e00 0x7fada41f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada922e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada4145630 con 0x7fada4154e00 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada922e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada4144a90 con 0x7fada40b5f10 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada922e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada4144d00 con 0x7fada406b500 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada359e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada4154e00 0x7fada41f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada359e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada4154e00 0x7fada41f3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57460/0 (socket says 172.21.15.172:57460) 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada359e640 1 -- 172.21.15.172:0/3875908451 learned_addr learned my addr 172.21.15.172:0/3875908451 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada2d9d640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada406b500 0x7fada415a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada259c640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada40b5f10 0x7fada4143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada359e640 1 -- 172.21.15.172:0/3875908451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada406b500 msgr2=0x7fada415a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada359e640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada406b500 0x7fada415a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada359e640 1 -- 172.21.15.172:0/3875908451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada40b5f10 msgr2=0x7fada4143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada359e640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada40b5f10 0x7fada4143290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada359e640 1 -- 172.21.15.172:0/3875908451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94009400 con 0x7fada4154e00 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.754+0000 7fada2d9d640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada406b500 0x7fada415a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.755+0000 7fada359e640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada4154e00 0x7fada41f3d70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c780 tx=0x7fad9c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.755+0000 7fad9b7fe640 1 -- 172.21.15.172:0/3875908451 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00ce90 con 0x7fada4154e00 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.755+0000 7fad9b7fe640 1 -- 172.21.15.172:0/3875908451 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c00a640 con 0x7fada4154e00 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.755+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada4144fe0 con 0x7fada4154e00 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.755+0000 7fad9b7fe640 1 -- 172.21.15.172:0/3875908451 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00aa70 con 0x7fada4154e00 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.755+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada41f47f0 con 0x7fada4154e00 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.756+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada41457a0 con 0x7fada4154e00 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.756+0000 7fad9b7fe640 1 -- 172.21.15.172:0/3875908451 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fad9c00abd0 con 0x7fada4154e00 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.757+0000 7fad9b7fe640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8406b3b0 0x7fad8406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.757+0000 7fad9b7fe640 1 -- 172.21.15.172:0/3875908451 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fad9c089370 con 0x7fada4154e00 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.757+0000 7fada2d9d640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8406b3b0 0x7fad8406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.757+0000 7fada2d9d640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8406b3b0 0x7fad8406d870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009e90 tx=0x7fad8c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.761+0000 7fad9b7fe640 1 -- 172.21.15.172:0/3875908451 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad9c054090 con 0x7fada4154e00 2024-03-16T19:30:41.768 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:41.779+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:41.859 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:41.879+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:41.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.887+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 0x7f253c02bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.887+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 0x7f253c02bde0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.888+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f253c157f30 con 0x7f253c062380 2024-03-16T19:30:41.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.902+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f253c0c5380 con 0x7f253c062380 2024-03-16T19:30:41.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.904+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f253c0c5380 con 0x7f253c062380 2024-03-16T19:30:41.885 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:41.886 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:41.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.904+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f253c02c9a0 0x7f253c02ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.904+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253c02d340 con 0x7f253c02c9a0 2024-03-16T19:30:41.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.905+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f253c02c9a0 0x7f253c02ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.905+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f253c02c9a0 0x7f253c02ce00 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.906+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f253c02d340 con 0x7f253c02c9a0 2024-03-16T19:30:41.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.920+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f253c02d340 con 0x7f253c02c9a0 2024-03-16T19:30:41.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.922+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f253c02d340 con 0x7f253c02c9a0 2024-03-16T19:30:41.903 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:41.903 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:41.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.922+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f253c156d30 0x7f253c1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.922+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253c1576f0 con 0x7f253c156d30 2024-03-16T19:30:41.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.922+0000 7f254151c640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f253c156d30 0x7f253c1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.922+0000 7f254151c640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f253c156d30 0x7f253c1571b0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.923+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f253c1576f0 con 0x7f253c156d30 2024-03-16T19:30:41.910 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:41.910 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:41.910 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:41.910 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:41.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:41.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.941+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f253c1576f0 con 0x7f253c156d30 2024-03-16T19:30:41.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.942+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f253c1576f0 con 0x7f253c156d30 2024-03-16T19:30:41.924 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:41.924 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:41.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.943+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f253c2051f0 0x7f253c2075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.943+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253c20a3b0 con 0x7f253c2051f0 2024-03-16T19:30:41.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.943+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f253c2051f0 0x7f253c2075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.943+0000 7f254251e640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f253c2051f0 0x7f253c2075e0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.944+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f253c20a3b0 con 0x7f253c2051f0 2024-03-16T19:30:41.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.959+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f253c207b20 con 0x7f253c2051f0 2024-03-16T19:30:41.942 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.960+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f253c207b20 con 0x7f253c2051f0 2024-03-16T19:30:41.942 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:41.942 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:41.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.961+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f253c208370 0x7f253c2087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:41.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.961+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253c20d740 con 0x7f253c208370 2024-03-16T19:30:41.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.961+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f253c208370 0x7f253c2087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:41.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.961+0000 7f2541d1d640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f253c208370 0x7f253c2087d0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:41.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.962+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f253c20d740 con 0x7f253c208370 2024-03-16T19:30:41.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.964+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7fada40cc710 con 0x7fada4154e00 2024-03-16T19:30:41.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.977+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f253c208d90 con 0x7f253c208370 2024-03-16T19:30:41.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f25327fc640 1 -- 172.21.15.172:0/1815250739 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f253c208d90 con 0x7f253c208370 2024-03-16T19:30:41.960 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f253c1efe40 msgr2=0x7f253c1f0220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f253c1efe40 0x7f253c1f0220 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f253c1f0f40 msgr2=0x7f253c0cd120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f253c1f0f40 0x7f253c0cd120 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f253c0cdb70 msgr2=0x7f253c061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f253c0cdb70 0x7f253c061930 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 msgr2=0x7f253c02bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f253c062380 0x7f253c02bde0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f253c02c9a0 msgr2=0x7f253c02ce00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f253c02c9a0 0x7f253c02ce00 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f254251e640 1 -- 172.21.15.172:0/1815250739 reap_dead start 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f254251e640 1 -- 172.21.15.172:0/1815250739 reap_dead start 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f253c156d30 msgr2=0x7f253c1571b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f253c156d30 0x7f253c1571b0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f253c2051f0 msgr2=0x7f253c2075e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f253c2051f0 0x7f253c2075e0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f253c208370 msgr2=0x7f253c2087d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f253c208370 0x7f253c2087d0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f252006b3b0 msgr2=0x7f252006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f252006b3b0 0x7f252006d870 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f253c1591c0 tx=0x7f252c009190 comp rx=0 tx=0).stop 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f253c070af0 msgr2=0x7f253c1f2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 --2- 172.21.15.172:0/1815250739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f253c070af0 0x7f253c1f2810 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2538033000 tx=0x7f2538004730 comp rx=0 tx=0).stop 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f254251e640 1 -- 172.21.15.172:0/1815250739 reap_dead start 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 shutdown_connections 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 >> 172.21.15.172:0/1815250739 conn(0x7f253c06e880 msgr2=0x7f253c0b2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 shutdown_connections 2024-03-16T19:30:41.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:41.979+0000 7f2543f81640 1 -- 172.21.15.172:0/1815250739 wait complete. 2024-03-16T19:30:41.964 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:41.980+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:42.030 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:42.030 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:42.030 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:42.030 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:42.030 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:42.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:42.061 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:42.081+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:42.151 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.812806601999995, 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053568110000000004, 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357139+0000", 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357146+0000", 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5860000000000001e-06 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357154+0000", 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5439999999999999e-06 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357217+0000", 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2986999999999999e-05 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357229+0000", 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1853e-05 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357343+0000", 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113663 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357504+0000", 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016176 2024-03-16T19:30:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013908599999999999 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360705+0000", 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030619419999999998 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362409+0000", 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001704021 2024-03-16T19:30:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362456+0000", 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6140000000000002e-05 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362496+0000", 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0228999999999997e-05 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.812786575000004, 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078702890000000008, 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357159+0000", 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4299999999999998e-07 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357163+0000", 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2559999999999998e-06 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357254+0000", 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1701999999999997e-05 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357258+0000", 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5719999999999999e-06 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357931+0000", 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067308000000000005 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357999+0000", 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8323000000000003e-05 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358113+0000", 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011366699999999999 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362505+0000", 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004391509 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024372970000000002 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364992+0000", 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9741999999999999e-05 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365029+0000", 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7697999999999997e-05 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.812779773000003, 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079944349999999994, 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357166+0000", 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6e-07 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357172+0000", 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7960000000000001e-06 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357285+0000", 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113183 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357290+0000", 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6190000000000002e-06 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358306+0000", 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010160670000000001 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358400+0000", 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3854e-05 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358492+0000", 2024-03-16T19:30:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2954000000000005e-05 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362510+0000", 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040175469999999998 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365078+0000", 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025680310000000001 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365123+0000", 2024-03-16T19:30:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4515000000000003e-05 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365160+0000", 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7608999999999999e-05 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.812772877, 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080787759999999993, 2024-03-16T19:30:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357173+0000", 2024-03-16T19:30:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.36e-07 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1270000000000001e-06 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357297+0000", 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121885 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357300+0000", 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049999999999999e-06 2024-03-16T19:30:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358709+0000", 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001409083 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358776+0000", 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6489999999999995e-05 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358885+0000", 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109271 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362514+0000", 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036292199999999998 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365197+0000", 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026832100000000001 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365226+0000", 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9136999999999999e-05 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365251+0000", 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5012e-05 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.812769892999995, 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081787609999999997, 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357176+0000", 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357177+0000", 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5090000000000001e-06 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357306+0000", 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012856900000000001 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357308+0000", 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3700000000000002e-06 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359071+0000", 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017631210000000001 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359182+0000", 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110265 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359280+0000", 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8595e-05 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362520+0000", 2024-03-16T19:30:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032394120000000001 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365297+0000", 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002777542 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365328+0000", 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0899999999999999e-05 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365354+0000", 2024-03-16T19:30:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6276000000000001e-05 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.812767205, 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083197830000000007, 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:42.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357178+0000", 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357179+0000", 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357187+0000", 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-03-16T19:30:42.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357314+0000", 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126978 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357316+0000", 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3800000000000001e-06 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359512+0000", 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021950799999999999 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359585+0000", 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3168000000000001e-05 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359702+0000", 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011741 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362525+0000", 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028225419999999999 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365408+0000", 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028829010000000002 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365460+0000", 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839e-05 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365498+0000", 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7673999999999999e-05 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.180 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.81275608, 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084493169999999996, 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357189+0000", 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357190+0000", 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-07 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357191+0000", 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7069999999999999e-06 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357325+0000", 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133585 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357330+0000", 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0799999999999996e-06 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359915+0000", 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025849290000000001 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:42.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9741999999999995e-05 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360121+0000", 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362826+0000", 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027046819999999999 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365564+0000", 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027387179999999998 2024-03-16T19:30:42.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365606+0000", 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1715999999999998e-05 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365639+0000", 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660999999999998e-05 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:42.185 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.812753411000003, 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085533069999999996, 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357192+0000", 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0699999999999999e-07 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3030000000000001e-06 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357339+0000", 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014513199999999999 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000003e-06 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030087619999999999 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360414+0000", 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1297999999999995e-05 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360498+0000", 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4538999999999998e-05 2024-03-16T19:30:42.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362832+0000", 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023338410000000001 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365670+0000", 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028379550000000001 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365724+0000", 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3533999999999999e-05 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365745+0000", 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1725999999999998e-05 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.812750187000006, 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052850789999999998, 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.190 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357195+0000", 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357196+0000", 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.173e-06 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357359+0000", 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016222400000000001 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357365+0000", 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7320000000000003e-06 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357451+0000", 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6494000000000002e-05 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357555+0000", 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010401200000000001 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357643+0000", 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8224000000000001e-05 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362294+0000", 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046502399999999999 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362388+0000", 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4448000000000001e-05 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362433+0000", 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4888000000000001e-05 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362480+0000", 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7441000000000002e-05 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.812748150000004, 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053762239999999998, 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357197+0000", 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357198+0000", 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-07 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357199+0000", 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357415+0000", 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021638699999999999 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0069e-05 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358132+0000", 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070650599999999997 2024-03-16T19:30:42.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358207+0000", 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5197999999999994e-05 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358342+0000", 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013503199999999999 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362377+0000", 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040354600000000003 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362493+0000", 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115951 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362536+0000", 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3189999999999998e-05 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362574+0000", 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7113999999999997e-05 2024-03-16T19:30:42.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:6 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.809689249000002, 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045449999999999999, 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:42.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360256+0000", 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360257+0000", 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7400000000000002e-07 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360260+0000", 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3550000000000001e-06 2024-03-16T19:30:42.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360270+0000", 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360277+0000", 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5760000000000002e-06 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360697+0000", 2024-03-16T19:30:42.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042039200000000001 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360702+0000", 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7140000000000001e-06 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360711+0000", 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1209999999999996e-06 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.203 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:22 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.807745073999996, 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100673, 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362200+0000", 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362201+0000", 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5799999999999998e-07 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362206+0000", 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8579999999999999e-06 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362219+0000", 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.239e-05 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362226+0000", 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2710000000000003e-06 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362281+0000", 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4756000000000002e-05 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362288+0000", 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.075e-06 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362301+0000", 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3365e-05 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:23 2.e e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.807633804999995, 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017205200000000001, 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362312+0000", 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4499999999999997e-07 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362315+0000", 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1350000000000001e-06 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362322+0000", 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9469999999999997e-06 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362328+0000", 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362366+0000", 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8160000000000001e-05 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362373+0000", 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5799999999999997e-06 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362484+0000", 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111173 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:7 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.805089933000005, 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018177499999999999, 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364856+0000", 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364861+0000", 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3699999999999997e-06 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364875+0000", 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4059e-05 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364881+0000", 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5270000000000004e-06 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364929+0000", 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000002e-05 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3699999999999997e-06 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365037+0000", 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100819 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:8 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.805032875999999, 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025451199999999998, 2024-03-16T19:30:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364913+0000", 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364916+0000", 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8109999999999999e-06 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364924+0000", 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1419999999999993e-06 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364930+0000", 2024-03-16T19:30:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2260000000000002e-06 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365071+0000", 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014030100000000001 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365075+0000", 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365167+0000", 2024-03-16T19:30:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2196999999999998e-05 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:9 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.805006636000002, 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032082900000000001, 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.22e-07 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364942+0000", 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3980000000000002e-06 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364947+0000", 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3090000000000003e-06 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364951+0000", 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1309999999999999e-06 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365191+0000", 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023992600000000001 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365195+0000", 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.704e-06 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365260+0000", 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5139000000000001e-05 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:10 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.804978250000005, 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039415499999999998, 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364968+0000", 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364970+0000", 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4650000000000001e-06 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364977+0000", 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.443e-06 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364981+0000", 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2320000000000003e-06 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365289+0000", 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030856900000000002 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365294+0000", 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6920000000000001e-06 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365361+0000", 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7358000000000001e-05 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:11 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.80495775, 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051803199999999998, 2024-03-16T19:30:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364988+0000", 2024-03-16T19:30:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9299999999999999e-07 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364993+0000", 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9080000000000003e-06 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4240000000000001e-06 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365002+0000", 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7570000000000002e-06 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365395+0000", 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039237199999999999 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365401+0000", 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9380000000000003e-06 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365506+0000", 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010433999999999999 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:12 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.804935745999998, 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063430500000000005, 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365010+0000", 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9500000000000001e-07 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365012+0000", 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7129999999999999e-06 2024-03-16T19:30:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365015+0000", 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2629999999999999e-06 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365017+0000", 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1749999999999999e-06 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365544+0000", 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052690099999999998 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365559+0000", 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4610000000000001e-05 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365644+0000", 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5447999999999999e-05 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:13 2.10 e21/18 ondisk, result = 0)", 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.804925248000004, 2024-03-16T19:30:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00073113500000000005, 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365020+0000", 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1000000000000001e-07 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365021+0000", 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.23e-07 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365024+0000", 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7369999999999998e-06 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365026+0000", 2024-03-16T19:30:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1289999999999999e-06 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365663+0000", 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063679600000000002 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365667+0000", 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.233e-06 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365751+0000", 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4307000000000004e-05 2024-03-16T19:30:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.236 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.236 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.236 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:42.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:42.237 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:42.257+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:42.337 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:42.357+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:42.346 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:42.346 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:42.346 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:42.346 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:42.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:42.438 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:42.458+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:42.462 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:42.462 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:42.462 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:42.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:42.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:42.560+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:42.581 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.243516909999997, 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.582 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:42.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:30:42.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.585 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.243502437000004, 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:42.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:30:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:30:42.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.242997947999996, 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.242982552000001, 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.597 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.242523868999996, 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.601 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.241977699000003, 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:42.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:42.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:30:42.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:30:42.605 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.241420607999999, 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:42.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:30:42.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:30:42.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:30:42.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.240218859999999, 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.239640151000003, 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:42.617 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.239590624000002, 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:30:42.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:30:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.238629981000003, 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:30:42.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:30:42.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.624 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.234915783000005, 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:30:42.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:42.628 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.794514870999997, 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:30:42.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:30:42.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:30:42.632 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.228961867999999, 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:30:42.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:30:42.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.636 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 59.721363142999998, 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:42.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:42.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:42.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:42.641 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:42.641 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:42.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:30:42.642 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.621+0000 7fad9b7fe640 1 -- 172.21.15.172:0/3875908451 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v54) v1 ==== 113+0+0 (secure 0 0 0) 0x7fad9c057750 con 0x7fada4154e00 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.5 to 1 (10000) 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8406b3b0 msgr2=0x7fad8406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8406b3b0 0x7fad8406d870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009e90 tx=0x7fad8c009190 comp rx=0 tx=0).stop 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada4154e00 msgr2=0x7fada41f3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada4154e00 0x7fada41f3d70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c780 tx=0x7fad9c00cc40 comp rx=0 tx=0).stop 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 shutdown_connections 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada4154e00 0x7fada41f3d70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada40b5f10 0x7fada4143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8406b3b0 0x7fad8406d870 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 --2- 172.21.15.172:0/3875908451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada406b500 0x7fada415a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 >> 172.21.15.172:0/3875908451 conn(0x7fada4066c20 msgr2=0x7fada40c77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:42.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 shutdown_connections 2024-03-16T19:30:42.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.624+0000 7fada922e640 1 -- 172.21.15.172:0/3875908451 wait complete. 2024-03-16T19:30:42.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-16T19:30:42.645 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:42.662+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:42.743 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:42.763+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:42.753 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:42.754 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:42.754 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:42.754 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:42.761 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:42.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.806+0000 7f0556b3e640 1 Processor -- start 2024-03-16T19:30:42.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.806+0000 7f0556b3e640 1 -- start start 2024-03-16T19:30:42.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.806+0000 7f504bef4640 1 Processor -- start 2024-03-16T19:30:42.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f504bef4640 1 -- start start 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f0556b3e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0550154e50 0x7f05500cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f0556b3e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f05500b5f10 0x7f05500cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f0556b3e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055006b110 con 0x7f05500caf40 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f0556b3e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055015b8d0 con 0x7f05500b5f10 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f0556b3e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055015ad50 con 0x7f0550154e50 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f054ffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f05500b5f10 0x7f05500cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f05550db640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f05500caf40 0x7f055006b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45094/0 (socket says 172.21.15.172:45094) 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f05550db640 1 -- 172.21.15.172:0/3133346876 learned_addr learned my addr 172.21.15.172:0/3133346876 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f05548da640 1 --2- 172.21.15.172:0/3133346876 >> v2:172.21.15.174:3300/0 conn(0x7f0550154e50 0x7f05500cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f054ffff640 1 -- 172.21.15.172:0/3133346876 >> v2:172.21.15.174:3300/0 conn(0x7f0550154e50 msgr2=0x7f05500cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:42.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f054ffff640 1 --2- 172.21.15.172:0/3133346876 >> v2:172.21.15.174:3300/0 conn(0x7f0550154e50 0x7f05500cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f054ffff640 1 -- 172.21.15.172:0/3133346876 >> v1:172.21.15.172:6789/0 conn(0x7f05500caf40 legacy=0x7f055006b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f054f7fe640 1 -- 172.21.15.172:0/3133346876 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89526179 0 0) 0x7f055006b110 con 0x7f05500caf40 2024-03-16T19:30:42.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f054ffff640 1 -- 172.21.15.172:0/3133346876 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055015b030 con 0x7f05500b5f10 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f504bef4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f50441528b0 0x7f5044072410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.807+0000 7f504bef4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c89c0 0x7f50440ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f504bef4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504406ca10 con 0x7f50440c79d0 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f504bef4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504406cc40 con 0x7f50441528b0 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f504bef4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504415bd30 con 0x7f50440c89c0 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f5049c90640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f50441528b0 0x7f5044072410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f504948f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c89c0 0x7f50440ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f5049c90640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f50441528b0 0x7f5044072410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40542/0 (socket says 172.21.15.172:40542) 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f504a491640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f50440c79d0 0x7f50441586c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:43194/0 (socket says 172.21.15.172:43194) 2024-03-16T19:30:42.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f5049c90640 1 -- 172.21.15.172:0/4018293983 learned_addr learned my addr 172.21.15.172:0/4018293983 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:42.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f5049c90640 1 -- 172.21.15.172:0/4018293983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c89c0 msgr2=0x7f50440ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:42.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f5049c90640 1 --2- 172.21.15.172:0/4018293983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c89c0 0x7f50440ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f5049c90640 1 -- 172.21.15.172:0/4018293983 >> v1:172.21.15.174:6789/0 conn(0x7f50440c79d0 legacy=0x7f50441586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f5049c90640 1 -- 172.21.15.172:0/4018293983 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504415c660 con 0x7f50441528b0 2024-03-16T19:30:42.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f5048c8e640 1 -- 172.21.15.172:0/4018293983 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769711601 0 0) 0x7f504406ca10 con 0x7f50440c79d0 2024-03-16T19:30:42.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f054ffff640 1 --2- 172.21.15.172:0/3133346876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05500b5f10 0x7f05500cc170 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f055008e370 tx=0x7f053c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f161bf62cb3483c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:42.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f054f7fe640 1 -- 172.21.15.172:0/3133346876 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c030610 con 0x7f05500b5f10 2024-03-16T19:30:42.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f054f7fe640 1 -- 172.21.15.172:0/3133346876 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c030770 con 0x7f05500b5f10 2024-03-16T19:30:42.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.808+0000 7f054f7fe640 1 -- 172.21.15.172:0/3133346876 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c030b00 con 0x7f05500b5f10 2024-03-16T19:30:42.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f5049c90640 1 --2- 172.21.15.172:0/4018293983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50441528b0 0x7f5044072410 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5040009870 tx=0x7f504002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a42cf237f4f361c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:42.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f5048c8e640 1 -- 172.21.15.172:0/4018293983 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040030840 con 0x7f50441528b0 2024-03-16T19:30:42.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f5048c8e640 1 -- 172.21.15.172:0/4018293983 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50400309a0 con 0x7f50441528b0 2024-03-16T19:30:42.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f5048c8e640 1 -- 172.21.15.172:0/4018293983 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040030d30 con 0x7f50441528b0 2024-03-16T19:30:42.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f0556b3e640 1 -- 172.21.15.172:0/3133346876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05500b5f10 msgr2=0x7f05500cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f0556b3e640 1 --2- 172.21.15.172:0/3133346876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05500b5f10 0x7f05500cc170 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f055008e370 tx=0x7f053c02b960 comp rx=0 tx=0).stop 2024-03-16T19:30:42.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f0556b3e640 1 -- 172.21.15.172:0/3133346876 shutdown_connections 2024-03-16T19:30:42.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f0556b3e640 1 --2- 172.21.15.172:0/3133346876 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f05500b5f10 0x7f05500cc170 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f0556b3e640 1 --2- 172.21.15.172:0/3133346876 >> v2:172.21.15.174:3300/0 conn(0x7f0550154e50 0x7f05500cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f0556b3e640 1 -- 172.21.15.172:0/3133346876 >> 172.21.15.172:0/3133346876 conn(0x7f0550066c20 msgr2=0x7f0550067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:42.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f0556b3e640 1 -- 172.21.15.172:0/3133346876 shutdown_connections 2024-03-16T19:30:42.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.809+0000 7f0556b3e640 1 -- 172.21.15.172:0/3133346876 wait complete. 2024-03-16T19:30:42.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.810+0000 7f504bef4640 1 -- 172.21.15.172:0/4018293983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50441528b0 msgr2=0x7f5044072410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.810+0000 7f504bef4640 1 --2- 172.21.15.172:0/4018293983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50441528b0 0x7f5044072410 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5040009870 tx=0x7f504002b980 comp rx=0 tx=0).stop 2024-03-16T19:30:42.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.810+0000 7f504bef4640 1 -- 172.21.15.172:0/4018293983 shutdown_connections 2024-03-16T19:30:42.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.810+0000 7f504bef4640 1 --2- 172.21.15.172:0/4018293983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c89c0 0x7f50440ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.810+0000 7f504bef4640 1 --2- 172.21.15.172:0/4018293983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50441528b0 0x7f5044072410 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.810+0000 7f504bef4640 1 -- 172.21.15.172:0/4018293983 >> 172.21.15.172:0/4018293983 conn(0x7f5044066c20 msgr2=0x7f5044067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:42.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.810+0000 7f504bef4640 1 -- 172.21.15.172:0/4018293983 shutdown_connections 2024-03-16T19:30:42.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.810+0000 7f504bef4640 1 -- 172.21.15.172:0/4018293983 wait complete. 2024-03-16T19:30:42.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.811+0000 7f0556b3e640 1 Processor -- start 2024-03-16T19:30:42.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.811+0000 7f0556b3e640 1 -- start start 2024-03-16T19:30:42.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f504bef4640 1 Processor -- start 2024-03-16T19:30:42.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f504bef4640 1 -- start start 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f0556b3e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05500b5f10 0x7f0550157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f0556b3e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05500caf40 0x7f0550147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f0556b3e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0550154e50 0x7f0550147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f0556b3e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0550156ec0 con 0x7f0550154e50 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f0556b3e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05501f14b0 con 0x7f05500b5f10 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f0556b3e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05501f0f30 con 0x7f05500caf40 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f05550db640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0550154e50 0x7f0550147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f054ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05500caf40 0x7f0550147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f05550db640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0550154e50 0x7f0550147d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57470/0 (socket says 172.21.15.172:57470) 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f05548da640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05500b5f10 0x7f0550157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.812+0000 7f05550db640 1 -- 172.21.15.172:0/2686931736 learned_addr learned my addr 172.21.15.172:0/2686931736 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f05550db640 1 -- 172.21.15.172:0/2686931736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05500caf40 msgr2=0x7f0550147800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f05550db640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05500caf40 0x7f0550147800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f05550db640 1 -- 172.21.15.172:0/2686931736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05500b5f10 msgr2=0x7f0550157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f05550db640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05500b5f10 0x7f0550157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f05550db640 1 -- 172.21.15.172:0/2686931736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c009400 con 0x7f0550154e50 2024-03-16T19:30:42.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f05548da640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05500b5f10 0x7f0550157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:42.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504bef4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c79d0 0x7f504415b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f05550db640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0550154e50 0x7f0550147d40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f054800c7a0 tx=0x7f054800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:42.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504bef4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50440c89c0 0x7f50440c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504bef4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50441528b0 0x7f5044159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504bef4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50440cc0a0 con 0x7f50441528b0 2024-03-16T19:30:42.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504bef4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50440c4090 con 0x7f50440c89c0 2024-03-16T19:30:42.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504bef4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504415b220 con 0x7f50440c79d0 2024-03-16T19:30:42.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f5049c90640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c79d0 0x7f504415b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504948f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50440c89c0 0x7f50440c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f054d7fa640 1 -- 172.21.15.172:0/2686931736 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054800a640 con 0x7f0550154e50 2024-03-16T19:30:42.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504948f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50440c89c0 0x7f50440c3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40552/0 (socket says 172.21.15.172:40552) 2024-03-16T19:30:42.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504948f640 1 -- 172.21.15.172:0/880127093 learned_addr learned my addr 172.21.15.172:0/880127093 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:42.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f5049c90640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c79d0 0x7f504415b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48090/0 (socket says 172.21.15.172:48090) 2024-03-16T19:30:42.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504a491640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50441528b0 0x7f5044159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055015ab20 con 0x7f0550154e50 2024-03-16T19:30:42.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f054d7fa640 1 -- 172.21.15.172:0/2686931736 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054800a7a0 con 0x7f0550154e50 2024-03-16T19:30:42.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f054d7fa640 1 -- 172.21.15.172:0/2686931736 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054800a640 con 0x7f0550154e50 2024-03-16T19:30:42.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504948f640 1 -- 172.21.15.172:0/880127093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c79d0 msgr2=0x7f504415b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504948f640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c79d0 0x7f504415b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504948f640 1 -- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50441528b0 msgr2=0x7f5044159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:42.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504948f640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50441528b0 0x7f5044159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:42.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.813+0000 7f504948f640 1 -- 172.21.15.172:0/880127093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50400094d0 con 0x7f50440c89c0 2024-03-16T19:30:42.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.814+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05501570c0 con 0x7f0550154e50 2024-03-16T19:30:42.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.814+0000 7f504948f640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50440c89c0 0x7f50440c3b50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f503400e9e0 tx=0x7f503400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:42.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.814+0000 7f503a7fc640 1 -- 172.21.15.172:0/880127093 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034009670 con 0x7f50440c89c0 2024-03-16T19:30:42.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.814+0000 7f503a7fc640 1 -- 172.21.15.172:0/880127093 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503400c9d0 con 0x7f50440c89c0 2024-03-16T19:30:42.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.814+0000 7f503a7fc640 1 -- 172.21.15.172:0/880127093 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503400cd60 con 0x7f50440c89c0 2024-03-16T19:30:42.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.814+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50440c61c0 con 0x7f50440c89c0 2024-03-16T19:30:42.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.814+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50440c3880 con 0x7f50440c89c0 2024-03-16T19:30:42.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.815+0000 7f054d7fa640 1 -- 172.21.15.172:0/2686931736 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f054800a950 con 0x7f0550154e50 2024-03-16T19:30:42.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.815+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05500c9ab0 con 0x7f0550154e50 2024-03-16T19:30:42.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.815+0000 7f054d7fa640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f053406b440 0x7f053406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.815+0000 7f054d7fa640 1 -- 172.21.15.172:0/2686931736 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f0548088df0 con 0x7f0550154e50 2024-03-16T19:30:42.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.815+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504415cad0 con 0x7f50440c89c0 2024-03-16T19:30:42.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.815+0000 7f503a7fc640 1 -- 172.21.15.172:0/880127093 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5034013470 con 0x7f50440c89c0 2024-03-16T19:30:42.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.815+0000 7f05548da640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f053406b440 0x7f053406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.816+0000 7f503a7fc640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f502806b3b0 0x7f502806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:42.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.816+0000 7f503a7fc640 1 -- 172.21.15.172:0/880127093 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f5034089ea0 con 0x7f50440c89c0 2024-03-16T19:30:42.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.816+0000 7f05548da640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f053406b440 0x7f053406d900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0544009e90 tx=0x7f0544009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:42.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.816+0000 7f5049c90640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f502806b3b0 0x7f502806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:42.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.816+0000 7f5049c90640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f502806b3b0 0x7f502806d870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5040036000 tx=0x7f5040008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:42.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.821+0000 7f054d7fa640 1 -- 172.21.15.172:0/2686931736 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0548053a00 con 0x7f0550154e50 2024-03-16T19:30:42.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:42.825+0000 7f503a7fc640 1 -- 172.21.15.172:0/880127093 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f504415cad0 con 0x7f50440c89c0 2024-03-16T19:30:42.850 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:42.870+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:42.909 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:42.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:42.909 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:42.909 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:42.909 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:42.917 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:42.944 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:42.963+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:42.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.010+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f0550075a70 con 0x7f0550154e50 2024-03-16T19:30:43.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.024+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f504415cad0 con 0x7f50440c89c0 2024-03-16T19:30:43.045 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:43.065+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.727286879000005, 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:43.065 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:43.070 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.726059457000005, 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.725656751000002, 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:43.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.077 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.725048000000001, 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:43.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.724604131000007, 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:43.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:43.085 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.724461355000003, 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.724446814000004, 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.092 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.721569118999994, 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:43.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:43.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.096 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:43.097 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:43.146 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:43.165+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:43.244 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:43.245 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:43.245 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:43.245 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:43.247 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:43.266+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:43.253 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:43.348 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:43.367+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:43.369 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:43.369 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:43.369 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:43.369 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:43.369 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:43.378 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:43.449 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:43.468+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.146796375999998, 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.485 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:30:43.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.146717257999995, 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.146103134000001, 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:43.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:30:43.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.145663064999994, 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.500 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.145421475000006, 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:30:43.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.504 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.505 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.505 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.505 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:43.505 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:43.549 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:43.569+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:43.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.625+0000 7f054d7fa640 1 -- 172.21.15.172:0/2686931736 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v55) v1 ==== 87+0+0 (secure 0 0 0) 0x7f0548018090 con 0x7f0550154e50 2024-03-16T19:30:43.607 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:30:43.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f053406b440 msgr2=0x7f053406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:43.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f053406b440 0x7f053406d900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0544009e90 tx=0x7f0544009190 comp rx=0 tx=0).stop 2024-03-16T19:30:43.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0550154e50 msgr2=0x7f0550147d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:43.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0550154e50 0x7f0550147d40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f054800c7a0 tx=0x7f054800cc60 comp rx=0 tx=0).stop 2024-03-16T19:30:43.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f503a7fc640 1 -- 172.21.15.172:0/880127093 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v55) v1 ==== 113+0+0 (secure 0 0 0) 0x7f504415cad0 con 0x7f50440c89c0 2024-03-16T19:30:43.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 shutdown_connections 2024-03-16T19:30:43.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0550154e50 0x7f0550147d40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f053406b440 0x7f053406d900 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05500caf40 0x7f0550147800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 --2- 172.21.15.172:0/2686931736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05500b5f10 0x7f0550157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 >> 172.21.15.172:0/2686931736 conn(0x7f0550066c20 msgr2=0x7f05500cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:43.611 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.6 to 1 (10000) 2024-03-16T19:30:43.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 shutdown_connections 2024-03-16T19:30:43.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.628+0000 7f0556b3e640 1 -- 172.21.15.172:0/2686931736 wait complete. 2024-03-16T19:30:43.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f502806b3b0 msgr2=0x7f502806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:43.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f502806b3b0 0x7f502806d870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5040036000 tx=0x7f5040008030 comp rx=0 tx=0).stop 2024-03-16T19:30:43.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50440c89c0 msgr2=0x7f50440c3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:43.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50440c89c0 0x7f50440c3b50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f503400e9e0 tx=0x7f503400eea0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 shutdown_connections 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50441528b0 0x7f5044159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50440c89c0 0x7f50440c3b50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f502806b3b0 0x7f502806d870 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 --2- 172.21.15.172:0/880127093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50440c79d0 0x7f504415b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 >> 172.21.15.172:0/880127093 conn(0x7f5044066c20 msgr2=0x7f50440cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 shutdown_connections 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.631+0000 7f504bef4640 1 -- 172.21.15.172:0/880127093 wait complete. 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:43.613 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:43.622 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:43.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-16T19:30:43.665 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:43.684+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:43.751 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:43.771+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:43.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.787+0000 7fbbd9c9d640 1 Processor -- start 2024-03-16T19:30:43.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.787+0000 7fbbd9c9d640 1 -- start start 2024-03-16T19:30:43.769 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:43.769 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:43.769 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:43.769 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:43.769 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:43.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.787+0000 7fbbd9c9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd4152580 0x7fbbd4159090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:43.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.787+0000 7fbbd9c9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbbd40c9aa0 0x7fbbd40c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:43.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd9c9d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd40cd4c0 con 0x7fbbd406b140 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd9c9d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd41595d0 con 0x7fbbd40c9aa0 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd9c9d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd40cd760 con 0x7fbbd4152580 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd37fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd4152580 0x7fbbd4159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd2ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbbd40c9aa0 0x7fbbd40c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd3fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fbbd406b140 0x7fbbd415bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:43204/0 (socket says 172.21.15.172:43204) 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd3fff640 1 -- 172.21.15.172:0/810438397 learned_addr learned my addr 172.21.15.172:0/810438397 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd2ffd640 1 --2- 172.21.15.172:0/810438397 >> v2:172.21.15.174:3300/0 conn(0x7fbbd40c9aa0 0x7fbbd40c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40556/0 (socket says 172.21.15.172:40556) 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd37fe640 1 --2- 172.21.15.172:0/810438397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd4152580 0x7fbbd4159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48104/0 (socket says 172.21.15.172:48104) 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd2ffd640 1 -- 172.21.15.172:0/810438397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd4152580 msgr2=0x7fbbd4159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd2ffd640 1 --2- 172.21.15.172:0/810438397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd4152580 0x7fbbd4159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd2ffd640 1 -- 172.21.15.172:0/810438397 >> v1:172.21.15.174:6789/0 conn(0x7fbbd406b140 legacy=0x7fbbd415bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd2ffd640 1 -- 172.21.15.172:0/810438397 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd40cda20 con 0x7fbbd40c9aa0 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd27fc640 1 -- 172.21.15.172:0/810438397 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157004168 0 0) 0x7fbbd40cd4c0 con 0x7fbbd406b140 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.788+0000 7fbbd37fe640 1 --2- 172.21.15.172:0/810438397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd4152580 0x7fbbd4159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:43.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.789+0000 7fbbd2ffd640 1 --2- 172.21.15.172:0/810438397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd40c9aa0 0x7fbbd40c9ec0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80097a0 tx=0x7fbbc802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e51ae8f4ff0072f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:43.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.789+0000 7fbbd27fc640 1 -- 172.21.15.172:0/810438397 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8030610 con 0x7fbbd40c9aa0 2024-03-16T19:30:43.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.789+0000 7fbbd27fc640 1 -- 172.21.15.172:0/810438397 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc8030770 con 0x7fbbd40c9aa0 2024-03-16T19:30:43.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.789+0000 7fbbd27fc640 1 -- 172.21.15.172:0/810438397 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8030b00 con 0x7fbbd40c9aa0 2024-03-16T19:30:43.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.790+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/810438397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd40c9aa0 msgr2=0x7fbbd40c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:43.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.790+0000 7fbbd9c9d640 1 --2- 172.21.15.172:0/810438397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd40c9aa0 0x7fbbd40c9ec0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80097a0 tx=0x7fbbc802b960 comp rx=0 tx=0).stop 2024-03-16T19:30:43.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.790+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/810438397 shutdown_connections 2024-03-16T19:30:43.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.790+0000 7fbbd9c9d640 1 --2- 172.21.15.172:0/810438397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd40c9aa0 0x7fbbd40c9ec0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.790+0000 7fbbd9c9d640 1 --2- 172.21.15.172:0/810438397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd4152580 0x7fbbd4159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.790+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/810438397 >> 172.21.15.172:0/810438397 conn(0x7fbbd406bfe0 msgr2=0x7fbbd406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:43.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.790+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/810438397 shutdown_connections 2024-03-16T19:30:43.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.790+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/810438397 wait complete. 2024-03-16T19:30:43.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.791+0000 7fbbd9c9d640 1 Processor -- start 2024-03-16T19:30:43.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.793+0000 7fbbd9c9d640 1 -- start start 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.793+0000 7fbbd9c9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbd406b140 0x7fbbd40c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.793+0000 7fbbd9c9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd40c9aa0 0x7fbbd41edc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.793+0000 7fbbd9c9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4152580 0x7fbbd41580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.793+0000 7fbbd9c9d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd41ee160 con 0x7fbbd406b140 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.793+0000 7fbbd9c9d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd414ec80 con 0x7fbbd4152580 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.793+0000 7fbbd9c9d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd40c4b00 con 0x7fbbd40c9aa0 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbd406b140 0x7fbbd40c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbd406b140 0x7fbbd40c6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57480/0 (socket says 172.21.15.172:57480) 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd37fe640 1 -- 172.21.15.172:0/657956009 learned_addr learned my addr 172.21.15.172:0/657956009 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd3fff640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4152580 0x7fbbd41580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd2ffd640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd40c9aa0 0x7fbbd41edc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd37fe640 1 -- 172.21.15.172:0/657956009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd40c9aa0 msgr2=0x7fbbd41edc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd37fe640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd40c9aa0 0x7fbbd41edc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd37fe640 1 -- 172.21.15.172:0/657956009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4152580 msgr2=0x7fbbd41580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd37fe640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4152580 0x7fbbd41580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd37fe640 1 -- 172.21.15.172:0/657956009 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc4009400 con 0x7fbbd406b140 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd37fe640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbd406b140 0x7fbbd40c6170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400ebe0 tx=0x7fbbc400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbbbfff640 1 -- 172.21.15.172:0/657956009 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4009760 con 0x7fbbd406b140 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbbbfff640 1 -- 172.21.15.172:0/657956009 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc400cc30 con 0x7fbbd406b140 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc8009400 con 0x7fbbd406b140 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbbbfff640 1 -- 172.21.15.172:0/657956009 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc400eee0 con 0x7fbbd406b140 2024-03-16T19:30:43.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.794+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd40c7390 con 0x7fbbd406b140 2024-03-16T19:30:43.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.796+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd40688d0 con 0x7fbbd406b140 2024-03-16T19:30:43.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.796+0000 7fbbbbfff640 1 -- 172.21.15.172:0/657956009 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbbc4013470 con 0x7fbbd406b140 2024-03-16T19:30:43.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.796+0000 7fbbbbfff640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbbc006b3b0 0x7fbbc006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:43.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.796+0000 7fbbbbfff640 1 -- 172.21.15.172:0/657956009 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fbbc408a270 con 0x7fbbd406b140 2024-03-16T19:30:43.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.796+0000 7fbbd2ffd640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbbc006b3b0 0x7fbbc006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:43.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.797+0000 7fbbd2ffd640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbbc006b3b0 0x7fbbc006d870 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8036000 tx=0x7fbbc8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:43.780 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:43.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.800+0000 7fbbbbfff640 1 -- 172.21.15.172:0/657956009 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbc4054f00 con 0x7fbbd406b140 2024-03-16T19:30:43.852 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:43.871+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:43.928 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:43.928 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:43.928 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:43.928 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:43.928 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.928 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:43.928 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:43.928 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.589794877000003, 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.929 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.930 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:43.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.932 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.589770704000003, 2024-03-16T19:30:43.933 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.934 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:43.936 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:30:43.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.589764876999993, 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:30:43.938 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.939 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.940 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.941 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.589760964999996, 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:30:43.943 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:43.944 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.946 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.947 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.589754983000006, 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.948 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:30:43.949 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:30:43.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:30:43.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.589751316999994, 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.953 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.954 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:43.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:43.958 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.589740656999993, 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.960 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.589736767999995, 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.962 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.586220076999993, 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.966 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:43.969 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.586149641999995, 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:43.970 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:30:43.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:30:43.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:43.973 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.585767294999997, 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.974 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.975 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.584001012000002, 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:30:43.977 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.582188090000002, 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:30:43.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.581332341000007, 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.984 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 67.575471574999995, 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.990 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:30:43.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 64.142920367000002, 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:43.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:30:43.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 62.575426661000002, 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:43.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:30:43.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:30:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.067814409999997, 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:30:44.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.570951184999998, 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:44.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:30:44.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:30:44.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.011 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.570415269000002, 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:30:44.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:30:44.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.015 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:44.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:44.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:43.991+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7fbbd406bcb0 con 0x7fbbd406b140 2024-03-16T19:30:44.017 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:44.035+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:44.018 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:44.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.135+0000 7fdea926f640 1 Processor -- start 2024-03-16T19:30:44.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.136+0000 7fdea926f640 1 -- start start 2024-03-16T19:30:44.117 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:44.137+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:44.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.136+0000 7fdea926f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdea40c8ba0 0x7fdea415e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.136+0000 7fdea926f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea406fde0 con 0x7fdea40c4040 2024-03-16T19:30:44.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.136+0000 7fdea926f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea40c31c0 con 0x7fdea4154bb0 2024-03-16T19:30:44.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.136+0000 7fdea926f640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea4070310 con 0x7fdea40c8ba0 2024-03-16T19:30:44.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.136+0000 7fdea37fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdea40c8ba0 0x7fdea415e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.136+0000 7fdea27fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fdea40c4040 0x7fdea415de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45104/0 (socket says 172.21.15.172:45104) 2024-03-16T19:30:44.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.137+0000 7fdea27fc640 1 -- 172.21.15.172:0/3914981664 learned_addr learned my addr 172.21.15.172:0/3914981664 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:44.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.137+0000 7fdea1ffb640 1 -- 172.21.15.172:0/3914981664 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394929457 0 0) 0x7fdea406fde0 con 0x7fdea40c4040 2024-03-16T19:30:44.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.137+0000 7fdea1ffb640 1 -- 172.21.15.172:0/3914981664 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde8c003520 con 0x7fdea40c4040 2024-03-16T19:30:44.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.137+0000 7fdea37fe640 1 -- 172.21.15.172:0/3914981664 >> v1:172.21.15.174:6789/0 conn(0x7fdea4154bb0 legacy=0x7fdea40c3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.137+0000 7fdea1ffb640 1 -- 172.21.15.172:0/3914981664 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947045309 0 0) 0x7fdea40c31c0 con 0x7fdea4154bb0 2024-03-16T19:30:44.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.137+0000 7fdea37fe640 1 -- 172.21.15.172:0/3914981664 >> v1:172.21.15.172:6789/0 conn(0x7fdea40c4040 legacy=0x7fdea415de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.137+0000 7fdea37fe640 1 -- 172.21.15.172:0/3914981664 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea4070030 con 0x7fdea40c8ba0 2024-03-16T19:30:44.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.138+0000 7fdea37fe640 1 --2- 172.21.15.172:0/3914981664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 0x7fdea415e520 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fde9c009d90 tx=0x7fde9c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359a21125d7ac121 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.138+0000 7fdea1ffb640 1 -- 172.21.15.172:0/3914981664 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c02cd80 con 0x7fdea40c8ba0 2024-03-16T19:30:44.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.138+0000 7fdea1ffb640 1 -- 172.21.15.172:0/3914981664 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9c034440 con 0x7fdea40c8ba0 2024-03-16T19:30:44.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.138+0000 7fdea1ffb640 1 -- 172.21.15.172:0/3914981664 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c0347d0 con 0x7fdea40c8ba0 2024-03-16T19:30:44.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.139+0000 7fdea926f640 1 -- 172.21.15.172:0/3914981664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 msgr2=0x7fdea415e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.139+0000 7fdea926f640 1 --2- 172.21.15.172:0/3914981664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 0x7fdea415e520 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fde9c009d90 tx=0x7fde9c02bfd0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.139+0000 7fdea926f640 1 -- 172.21.15.172:0/3914981664 shutdown_connections 2024-03-16T19:30:44.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.139+0000 7fdea926f640 1 --2- 172.21.15.172:0/3914981664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 0x7fdea415e520 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.139+0000 7fdea926f640 1 -- 172.21.15.172:0/3914981664 >> 172.21.15.172:0/3914981664 conn(0x7fdea40708a0 msgr2=0x7fdea4070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:44.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.139+0000 7fdea926f640 1 -- 172.21.15.172:0/3914981664 shutdown_connections 2024-03-16T19:30:44.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.139+0000 7fdea926f640 1 -- 172.21.15.172:0/3914981664 wait complete. 2024-03-16T19:30:44.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.141+0000 7fdea926f640 1 Processor -- start 2024-03-16T19:30:44.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.141+0000 7fdea926f640 1 -- start start 2024-03-16T19:30:44.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea926f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea40c4040 0x7fdea4157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea926f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 0x7fdea41f0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea926f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdea4154bb0 0x7fdea41f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea926f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41f0040 con 0x7fdea40c8ba0 2024-03-16T19:30:44.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea926f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41f3790 con 0x7fdea40c4040 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea926f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41f3a70 con 0x7fdea4154bb0 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea27fc640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 0x7fdea41f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea27fc640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 0x7fdea41f0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57502/0 (socket says 172.21.15.172:57502) 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea40c4040 0x7fdea4157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea27fc640 1 -- 172.21.15.172:0/601912790 learned_addr learned my addr 172.21.15.172:0/601912790 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdea4154bb0 0x7fdea41f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea27fc640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdea4154bb0 msgr2=0x7fdea41f0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea27fc640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdea4154bb0 0x7fdea41f0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea27fc640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea40c4040 msgr2=0x7fdea4157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea27fc640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea40c4040 0x7fdea4157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea27fc640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c009a70 con 0x7fdea40c8ba0 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.142+0000 7fdea27fc640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 0x7fdea41f0520 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fde9800c430 tx=0x7fde9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.143+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98005ce0 con 0x7fdea40c8ba0 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.143+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde98005e40 con 0x7fdea40c8ba0 2024-03-16T19:30:44.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.143+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde980072f0 con 0x7fdea40c8ba0 2024-03-16T19:30:44.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.143+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea41f3d50 con 0x7fdea40c8ba0 2024-03-16T19:30:44.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.143+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea41f34a0 con 0x7fdea40c8ba0 2024-03-16T19:30:44.126 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:44.126 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:44.126 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:44.126 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.144+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdea40681c0 con 0x7fdea40c8ba0 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.145+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fde98007450 con 0x7fdea40c8ba0 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.145+0000 7fde8b7fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fde9006b3b0 0x7fde9006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.145+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fde98089360 con 0x7fdea40c8ba0 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.145+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fde9006b3b0 0x7fde9006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.145+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v55) v1 ==== 54+0+16 (secure 0 0 0) 0x7fde98090070 con 0x7fdea40c8ba0 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.145+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fdea4074670 0x7fdea40ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.145+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdea406c460 con 0x7fdea4074670 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.146+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fde9006b3b0 0x7fde9006d870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fde94000f60 tx=0x7fde94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.146+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fdea4074670 0x7fdea40ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.146+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fdea4074670 0x7fdea40ce1a0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.146+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fde9c02cb80 con 0x7fdea4074670 2024-03-16T19:30:44.134 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:44.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.161+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdea4154f90 con 0x7fdea4074670 2024-03-16T19:30:44.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.163+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fde9c0345b0 con 0x7fdea4074670 2024-03-16T19:30:44.144 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:44.145 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:44.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.163+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fdea415b7a0 0x7fdea414fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.163+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdea406fbc0 con 0x7fdea415b7a0 2024-03-16T19:30:44.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.163+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fdea415b7a0 0x7fdea414fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.163+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fdea415b7a0 0x7fdea414fe00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.164+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdea406fbc0 con 0x7fdea415b7a0 2024-03-16T19:30:44.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.180+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdea406d1a0 con 0x7fdea415b7a0 2024-03-16T19:30:44.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.182+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdea406d1a0 con 0x7fdea415b7a0 2024-03-16T19:30:44.163 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:44.164 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:44.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.182+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fdea4158c20 0x7fdea4159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.182+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdea4159540 con 0x7fdea4158c20 2024-03-16T19:30:44.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.182+0000 7fdea27fc640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fdea4158c20 0x7fdea4159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.183+0000 7fdea27fc640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fdea4158c20 0x7fdea4159000 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.183+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdea4159540 con 0x7fdea4158c20 2024-03-16T19:30:44.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.197+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdea4061c00 con 0x7fdea4158c20 2024-03-16T19:30:44.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.199+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdea4061c00 con 0x7fdea4158c20 2024-03-16T19:30:44.180 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:44.181 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:44.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.199+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdea40626a0 0x7fdea41553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.199+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdea4155930 con 0x7fdea40626a0 2024-03-16T19:30:44.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.199+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdea40626a0 0x7fdea41553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.200+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdea40626a0 0x7fdea41553f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.200+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdea4155930 con 0x7fdea40626a0 2024-03-16T19:30:44.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.214+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdea4155e00 con 0x7fdea40626a0 2024-03-16T19:30:44.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.216+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdea4155e00 con 0x7fdea40626a0 2024-03-16T19:30:44.197 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:44.198 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:44.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.216+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 0x7fdea402c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.216+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdea402c760 con 0x7fdea402bde0 2024-03-16T19:30:44.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.216+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 0x7fdea402c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.216+0000 7fdea2ffd640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 msgr2=0x7fdea402c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 25 2024-03-16T19:30:44.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.216+0000 7fdea2ffd640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 msgr2=0x7fdea402c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:30:44.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.216+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 0x7fdea402c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:30:44.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.217+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 0x7fdea402c220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:30:44.218 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:44.237+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:44.281 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:44.281 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:44.281 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:44.281 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:44.281 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:44.290 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:44.322 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:44.338+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:44.399 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.418+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 0x7fdea402c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.399 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.418+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 0x7fdea402c220 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.400 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.419+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdea402c760 con 0x7fdea402bde0 2024-03-16T19:30:44.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.434+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdea402c760 con 0x7fdea402bde0 2024-03-16T19:30:44.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.436+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdea402c760 con 0x7fdea402bde0 2024-03-16T19:30:44.417 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:44.418 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:44.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.436+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fdea402ce70 0x7fdea4206830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.436+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdea402d2f0 con 0x7fdea402ce70 2024-03-16T19:30:44.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.436+0000 7fdea27fc640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fdea402ce70 0x7fdea4206830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.437+0000 7fdea27fc640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fdea402ce70 0x7fdea4206830 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.437+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdea402d2f0 con 0x7fdea402ce70 2024-03-16T19:30:44.419 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:44.438+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:44.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.452+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdea402d2f0 con 0x7fdea402ce70 2024-03-16T19:30:44.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.453+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdea402d2f0 con 0x7fdea402ce70 2024-03-16T19:30:44.435 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:44.436 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:44.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.453+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdea4207280 0x7fdea4209670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.453+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdea420c440 con 0x7fdea4207280 2024-03-16T19:30:44.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.453+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdea4207280 0x7fdea4209670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdea4207280 0x7fdea4209670 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fdea37fe640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdea4207280 msgr2=0x7fdea4209670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 27 2024-03-16T19:30:44.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fdea37fe640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdea4207280 msgr2=0x7fdea4209670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:30:44.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdea4207280 0x7fdea4209670 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:30:44.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdea4207280 0x7fdea4209670 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdea4207280 msgr2=0x7fdea4209670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:30:44.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fde8b7fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdea4207280 0x7fdea4209670 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fde8b7fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fde900755a0 0x7fde90077990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdea4155930 con 0x7fde900755a0 2024-03-16T19:30:44.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=56}) v3 -- 0x7fde90077ed0 con 0x7fdea40c8ba0 2024-03-16T19:30:44.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.454+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fde900755a0 0x7fde90077990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.455+0000 7fdea37fe640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fde900755a0 0x7fde90077990 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.455+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdea4155930 con 0x7fde900755a0 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 88.233100102999998, 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:30:44.444 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:44.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:30:44.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:30:44.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 87.856404556000001, 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.453 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 87.080428112000007, 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.454 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:30:44.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:30:44.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:30:44.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:30:44.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 87.080343698999997, 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:30:44.459 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:44.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:30:44.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:30:44.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:30:44.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.464 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 87.080336342999999, 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:30:44.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.887797151000001, 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.470 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:30:44.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.887791343999993, 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.475 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:30:44.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:30:44.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:30:44.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.887673211000006, 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.480 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:30:44.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:30:44.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.887578013999999, 2024-03-16T19:30:44.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:30:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:30:44.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.887574002999997, 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:30:44.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:30:44.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.468642129000003, 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:30:44.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:30:44.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:30:44.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:30:44.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:30:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.468592760000007, 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:30:44.501 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:44.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:30:44.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:30:44.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.468513927999993, 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.468510717000001, 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.512 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:30:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:30:44.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.468507763999995, 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.517 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:44.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:30:44.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:30:44.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.522 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.468392789999996, 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:30:44.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:30:44.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.527 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.468389438000003, 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:30:44.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:30:44.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.532 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.468386322000001, 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.533 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:44.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:30:44.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:30:44.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.468325852999996, 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:30:44.538 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:30:44.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.543 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.859645803000006, 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.548 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:44.549 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:44.549 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:44.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.470+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdea4207660 con 0x7fde900755a0 2024-03-16T19:30:44.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.472+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdea4207660 con 0x7fde900755a0 2024-03-16T19:30:44.550 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:44.551 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:44.569+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:44.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.472+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fdea4207ce0 0x7fdea420a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.472+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdea420bd20 con 0x7fdea4207ce0 2024-03-16T19:30:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.472+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fdea4207ce0 0x7fdea420a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.472+0000 7fdea2ffd640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fdea4207ce0 0x7fdea420a0d0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.473+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdea420bd20 con 0x7fdea4207ce0 2024-03-16T19:30:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.488+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdea420bf20 con 0x7fdea4207ce0 2024-03-16T19:30:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fde8b7fe640 1 -- 172.21.15.172:0/601912790 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdea420bf20 con 0x7fdea4207ce0 2024-03-16T19:30:44.553 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fdea4074670 msgr2=0x7fdea40ce1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fdea4074670 0x7fdea40ce1a0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fdea415b7a0 msgr2=0x7fdea414fe00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fdea415b7a0 0x7fdea414fe00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fdea4158c20 msgr2=0x7fdea4159000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fdea4158c20 0x7fdea4159000 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdea40626a0 msgr2=0x7fdea41553f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdea40626a0 0x7fdea41553f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 msgr2=0x7fdea402c220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdea402bde0 0x7fdea402c220 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea37fe640 1 -- 172.21.15.172:0/601912790 reap_dead start 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea37fe640 1 -- 172.21.15.172:0/601912790 reap_dead start 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fdea402ce70 msgr2=0x7fdea4206830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fdea402ce70 0x7fdea4206830 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fde900755a0 msgr2=0x7fde90077990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fde900755a0 0x7fde90077990 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fdea4207ce0 msgr2=0x7fdea420a0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fdea4207ce0 0x7fdea420a0d0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fde9006b3b0 msgr2=0x7fde9006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fde9006b3b0 0x7fde9006d870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fde94000f60 tx=0x7fde94009040 comp rx=0 tx=0).stop 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 msgr2=0x7fdea41f0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 0x7fdea41f0520 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fde9800c430 tx=0x7fde9800c8f0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea37fe640 1 -- 172.21.15.172:0/601912790 reap_dead start 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 shutdown_connections 2024-03-16T19:30:44.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 --2- 172.21.15.172:0/601912790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdea40c8ba0 0x7fdea41f0520 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 >> 172.21.15.172:0/601912790 conn(0x7fdea40708a0 msgr2=0x7fdea40cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:44.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 shutdown_connections 2024-03-16T19:30:44.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.490+0000 7fdea926f640 1 -- 172.21.15.172:0/601912790 wait complete. 2024-03-16T19:30:44.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.627+0000 7fbbbbfff640 1 -- 172.21.15.172:0/657956009 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v56) v1 ==== 113+0+0 (secure 0 0 0) 0x7fbbc40585c0 con 0x7fbbd406b140 2024-03-16T19:30:44.608 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.7 to 1 (10000) 2024-03-16T19:30:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.629+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbbc006b3b0 msgr2=0x7fbbc006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.629+0000 7fbbd9c9d640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbbc006b3b0 0x7fbbc006d870 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8036000 tx=0x7fbbc8008030 comp rx=0 tx=0).stop 2024-03-16T19:30:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.629+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbd406b140 msgr2=0x7fbbd40c6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.629+0000 7fbbd9c9d640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbd406b140 0x7fbbd40c6170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400ebe0 tx=0x7fbbc400c520 comp rx=0 tx=0).stop 2024-03-16T19:30:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.630+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 shutdown_connections 2024-03-16T19:30:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.630+0000 7fbbd9c9d640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4152580 0x7fbbd41580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.630+0000 7fbbd9c9d640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbbc006b3b0 0x7fbbc006d870 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.630+0000 7fbbd9c9d640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbd40c9aa0 0x7fbbd41edc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.630+0000 7fbbd9c9d640 1 --2- 172.21.15.172:0/657956009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbbd406b140 0x7fbbd40c6170 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.630+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 >> 172.21.15.172:0/657956009 conn(0x7fbbd406bfe0 msgr2=0x7fbbd41e4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:44.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.630+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 shutdown_connections 2024-03-16T19:30:44.612 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.630+0000 7fbbd9c9d640 1 -- 172.21.15.172:0/657956009 wait complete. 2024-03-16T19:30:44.624 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-16T19:30:44.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:30:44.666 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:44.685+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:44.714 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:44.714 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:44.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:44.768 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:44.787+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:44.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.795+0000 7f61c8a76640 1 Processor -- start 2024-03-16T19:30:44.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.795+0000 7f61c8a76640 1 -- start start 2024-03-16T19:30:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.796+0000 7f61c8a76640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f61c4154cb0 0x7f61c406dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.796+0000 7f61c8a76640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c415de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.796+0000 7f61c8a76640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c415e340 con 0x7f61c40c8ba0 2024-03-16T19:30:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.796+0000 7f61c8a76640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c40cbba0 con 0x7f61c4154cb0 2024-03-16T19:30:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.796+0000 7f61c8a76640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c40c91f0 con 0x7f61c406fe10 2024-03-16T19:30:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.796+0000 7f61c2d9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f61c40c8ba0 0x7f61c40cabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45112/0 (socket says 172.21.15.172:45112) 2024-03-16T19:30:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.796+0000 7f61c2d9d640 1 -- 172.21.15.172:0/3047711741 learned_addr learned my addr 172.21.15.172:0/3047711741 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.796+0000 7f61c1d9b640 1 --2- 172.21.15.172:0/3047711741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c415de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.796+0000 7f61c259c640 1 --2- 172.21.15.172:0/3047711741 >> v2:172.21.15.174:3300/0 conn(0x7f61c4154cb0 0x7f61c406dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.797+0000 7f61c159a640 1 -- 172.21.15.172:0/3047711741 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902956798 0 0) 0x7f61c415e340 con 0x7f61c40c8ba0 2024-03-16T19:30:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.797+0000 7f61c159a640 1 -- 172.21.15.172:0/3047711741 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac003520 con 0x7f61c40c8ba0 2024-03-16T19:30:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.797+0000 7f61c259c640 1 -- 172.21.15.172:0/3047711741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 msgr2=0x7f61c415de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.797+0000 7f61c259c640 1 --2- 172.21.15.172:0/3047711741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c415de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.797+0000 7f61c259c640 1 -- 172.21.15.172:0/3047711741 >> v1:172.21.15.172:6789/0 conn(0x7f61c40c8ba0 legacy=0x7f61c40cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.797+0000 7f61c259c640 1 -- 172.21.15.172:0/3047711741 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c406c180 con 0x7f61c4154cb0 2024-03-16T19:30:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.797+0000 7f61c1d9b640 1 --2- 172.21.15.172:0/3047711741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c415de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.798+0000 7f61c259c640 1 --2- 172.21.15.172:0/3047711741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 0x7f61c406dc20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f61b40097a0 tx=0x7f61b402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=264707136927b0b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.798+0000 7f61c159a640 1 -- 172.21.15.172:0/3047711741 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b402f050 con 0x7f61c4154cb0 2024-03-16T19:30:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.798+0000 7f61c159a640 1 -- 172.21.15.172:0/3047711741 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b4031700 con 0x7f61c4154cb0 2024-03-16T19:30:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.798+0000 7f61c159a640 1 -- 172.21.15.172:0/3047711741 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b4031a90 con 0x7f61c4154cb0 2024-03-16T19:30:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.798+0000 7f61c8a76640 1 -- 172.21.15.172:0/3047711741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 msgr2=0x7f61c406dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.798+0000 7f61c8a76640 1 --2- 172.21.15.172:0/3047711741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 0x7f61c406dc20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f61b40097a0 tx=0x7f61b402b7a0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.799+0000 7f61c8a76640 1 -- 172.21.15.172:0/3047711741 shutdown_connections 2024-03-16T19:30:44.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.799+0000 7f61c8a76640 1 --2- 172.21.15.172:0/3047711741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c415de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.799+0000 7f61c8a76640 1 --2- 172.21.15.172:0/3047711741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 0x7f61c406dc20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.799+0000 7f61c8a76640 1 -- 172.21.15.172:0/3047711741 >> 172.21.15.172:0/3047711741 conn(0x7f61c4070860 msgr2=0x7f61c4070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:44.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.799+0000 7f61c8a76640 1 -- 172.21.15.172:0/3047711741 shutdown_connections 2024-03-16T19:30:44.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.799+0000 7f61c8a76640 1 -- 172.21.15.172:0/3047711741 wait complete. 2024-03-16T19:30:44.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.800+0000 7f61c8a76640 1 Processor -- start 2024-03-16T19:30:44.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.800+0000 7f61c8a76640 1 -- start start 2024-03-16T19:30:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c8a76640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c41f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c8a76640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61c40c8ba0 0x7f61c406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c8a76640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 0x7f61c406f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c8a76640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c41f3840 con 0x7f61c40c8ba0 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c8a76640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c40cdfe0 con 0x7f61c4154cb0 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c8a76640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c41e73b0 con 0x7f61c406fe10 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c1d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61c40c8ba0 0x7f61c406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c1d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61c40c8ba0 0x7f61c406eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57510/0 (socket says 172.21.15.172:57510) 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c41f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c1d9b640 1 -- 172.21.15.172:0/2085309881 learned_addr learned my addr 172.21.15.172:0/2085309881 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c2d9d640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 0x7f61c406f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c1d9b640 1 -- 172.21.15.172:0/2085309881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 msgr2=0x7f61c41f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c1d9b640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c41f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c1d9b640 1 -- 172.21.15.172:0/2085309881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 msgr2=0x7f61c406f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c1d9b640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 0x7f61c406f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c1d9b640 1 -- 172.21.15.172:0/2085309881 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b8009400 con 0x7f61c40c8ba0 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c2d9d640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 0x7f61c406f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c259c640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c41f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:44.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.801+0000 7f61c1d9b640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61c40c8ba0 0x7f61c406eed0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f61b800e9b0 tx=0x7f61b800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.802+0000 7f61b2ffd640 1 -- 172.21.15.172:0/2085309881 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b800ca00 con 0x7f61c40c8ba0 2024-03-16T19:30:44.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.802+0000 7f61b2ffd640 1 -- 172.21.15.172:0/2085309881 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b800cb60 con 0x7f61c40c8ba0 2024-03-16T19:30:44.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.802+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b4009400 con 0x7f61c40c8ba0 2024-03-16T19:30:44.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.802+0000 7f61b2ffd640 1 -- 172.21.15.172:0/2085309881 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b8013470 con 0x7f61c40c8ba0 2024-03-16T19:30:44.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.802+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c41f0030 con 0x7f61c40c8ba0 2024-03-16T19:30:44.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.803+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c41f3980 con 0x7f61c40c8ba0 2024-03-16T19:30:44.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.803+0000 7f61b2ffd640 1 -- 172.21.15.172:0/2085309881 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f61b80135d0 con 0x7f61c40c8ba0 2024-03-16T19:30:44.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.804+0000 7f61b2ffd640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f61a406b3b0 0x7f61a406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:44.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.804+0000 7f61b2ffd640 1 -- 172.21.15.172:0/2085309881 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f61b8089fb0 con 0x7f61c40c8ba0 2024-03-16T19:30:44.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.804+0000 7f61c259c640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f61a406b3b0 0x7f61a406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:44.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.804+0000 7f61c259c640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f61a406b3b0 0x7f61a406d870 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f61b4035000 tx=0x7f61b4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:44.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:44.808+0000 7f61b2ffd640 1 -- 172.21.15.172:0/2085309881 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61b80bde10 con 0x7f61c40c8ba0 2024-03-16T19:30:44.839 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:44.839 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:44.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:44.852 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:44.872+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:44.952 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:44.972+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:44.998 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:44.999 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:45.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.023+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61c40c94b0 con 0x7f61a406b3b0 2024-03-16T19:30:45.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:45.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.031+0000 7f61b2ffd640 1 -- 172.21.15.172:0/2085309881 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251879 (secure 0 0 0) 0x7f61b4041070 con 0x7f61a406b3b0 2024-03-16T19:30:45.015 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:30:45.016 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:30:45.017 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f61a406b3b0 msgr2=0x7f61a406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f61a406b3b0 0x7f61a406d870 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f61b4035000 tx=0x7f61b4008030 comp rx=0 tx=0).stop 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61c40c8ba0 msgr2=0x7f61c406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61c40c8ba0 0x7f61c406eed0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f61b800e9b0 tx=0x7f61b800ee70 comp rx=0 tx=0).stop 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 shutdown_connections 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4154cb0 0x7f61c406f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61c40c8ba0 0x7f61c406eed0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f61a406b3b0 0x7f61a406d870 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 --2- 172.21.15.172:0/2085309881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c406fe10 0x7f61c41f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 >> 172.21.15.172:0/2085309881 conn(0x7f61c4070860 msgr2=0x7f61c40cc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 shutdown_connections 2024-03-16T19:30:45.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.035+0000 7f61c8a76640 1 -- 172.21.15.172:0/2085309881 wait complete. 2024-03-16T19:30:45.028 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-03-16T19:30:44.629439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597828,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":216,"num_write_kb":2441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164,"ondisk_log_size":164,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222252,"kb_used_data":6048,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752468,"statfs":{"total":773094113280,"available":772866527232,"internally_reserved":0,"allocated":6193152,"data_stored":3365168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014294"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626153+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:30:42.626153+0000","last_peered":"2024-03-16T19:30:42.626153+0000","last_clean":"2024-03-16T19:30:42.626153+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:30:42.626153+0000","last_undegraded":"2024-03-16T19:30:42.626153+0000","last_fullsized":"2024-03-16T19:30:42.626153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625194+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:30:42.625194+0000","last_peered":"2024-03-16T19:30:42.625194+0000","last_clean":"2024-03-16T19:30:42.625194+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:30:42.625194+0000","last_undegraded":"2024-03-16T19:30:42.625194+0000","last_fullsized":"2024-03-16T19:30:42.625194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":70,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154027+0000","last_change":"2024-03-16T19:29:44.412537+0000","last_active":"2024-03-16T19:30:43.154027+0000","last_peered":"2024-03-16T19:30:43.154027+0000","last_clean":"2024-03-16T19:30:43.154027+0000","last_became_active":"2024-03-16T19:29:44.412236+0000","last_became_peered":"2024-03-16T19:29:44.412236+0000","last_unstale":"2024-03-16T19:30:43.154027+0000","last_undegraded":"2024-03-16T19:30:43.154027+0000","last_fullsized":"2024-03-16T19:30:43.154027+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:50:57.180910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625952+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:30:42.625952+0000","last_peered":"2024-03-16T19:30:42.625952+0000","last_clean":"2024-03-16T19:30:42.625952+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:30:42.625952+0000","last_undegraded":"2024-03-16T19:30:42.625952+0000","last_fullsized":"2024-03-16T19:30:42.625952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625531+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:30:42.625531+0000","last_peered":"2024-03-16T19:30:42.625531+0000","last_clean":"2024-03-16T19:30:42.625531+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:30:42.625531+0000","last_undegraded":"2024-03-16T19:30:42.625531+0000","last_fullsized":"2024-03-16T19:30:42.625531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625348+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:30:42.625348+0000","last_peered":"2024-03-16T19:30:42.625348+0000","last_clean":"2024-03-16T19:30:42.625348+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:30:42.625348+0000","last_undegraded":"2024-03-16T19:30:42.625348+0000","last_fullsized":"2024-03-16T19:30:42.625348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625186+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:30:42.625186+0000","last_peered":"2024-03-16T19:30:42.625186+0000","last_clean":"2024-03-16T19:30:42.625186+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:30:42.625186+0000","last_undegraded":"2024-03-16T19:30:42.625186+0000","last_fullsized":"2024-03-16T19:30:42.625186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154267+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:30:43.154267+0000","last_peered":"2024-03-16T19:30:43.154267+0000","last_clean":"2024-03-16T19:30:43.154267+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:30:43.154267+0000","last_undegraded":"2024-03-16T19:30:43.154267+0000","last_fullsized":"2024-03-16T19:30:43.154267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626201+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:30:42.626201+0000","last_peered":"2024-03-16T19:30:42.626201+0000","last_clean":"2024-03-16T19:30:42.626201+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:30:42.626201+0000","last_undegraded":"2024-03-16T19:30:42.626201+0000","last_fullsized":"2024-03-16T19:30:42.626201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154122+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:30:43.154122+0000","last_peered":"2024-03-16T19:30:43.154122+0000","last_clean":"2024-03-16T19:30:43.154122+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:30:43.154122+0000","last_undegraded":"2024-03-16T19:30:43.154122+0000","last_fullsized":"2024-03-16T19:30:43.154122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154075+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:30:43.154075+0000","last_peered":"2024-03-16T19:30:43.154075+0000","last_clean":"2024-03-16T19:30:43.154075+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:30:43.154075+0000","last_undegraded":"2024-03-16T19:30:43.154075+0000","last_fullsized":"2024-03-16T19:30:43.154075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153964+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:30:43.153964+0000","last_peered":"2024-03-16T19:30:43.153964+0000","last_clean":"2024-03-16T19:30:43.153964+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:30:43.153964+0000","last_undegraded":"2024-03-16T19:30:43.153964+0000","last_fullsized":"2024-03-16T19:30:43.153964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154374+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:30:43.154374+0000","last_peered":"2024-03-16T19:30:43.154374+0000","last_clean":"2024-03-16T19:30:43.154374+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:30:43.154374+0000","last_undegraded":"2024-03-16T19:30:43.154374+0000","last_fullsized":"2024-03-16T19:30:43.154374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624981+0000","last_change":"2024-03-16T19:29:44.413594+0000","last_active":"2024-03-16T19:30:42.624981+0000","last_peered":"2024-03-16T19:30:42.624981+0000","last_clean":"2024-03-16T19:30:42.624981+0000","last_became_active":"2024-03-16T19:29:44.413235+0000","last_became_peered":"2024-03-16T19:29:44.413235+0000","last_unstale":"2024-03-16T19:30:42.624981+0000","last_undegraded":"2024-03-16T19:30:42.624981+0000","last_fullsized":"2024-03-16T19:30:42.624981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:35.778808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624886+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:30:42.624886+0000","last_peered":"2024-03-16T19:30:42.624886+0000","last_clean":"2024-03-16T19:30:42.624886+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:30:42.624886+0000","last_undegraded":"2024-03-16T19:30:42.624886+0000","last_fullsized":"2024-03-16T19:30:42.624886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624933+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:30:42.624933+0000","last_peered":"2024-03-16T19:30:42.624933+0000","last_clean":"2024-03-16T19:30:42.624933+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:30:42.624933+0000","last_undegraded":"2024-03-16T19:30:42.624933+0000","last_fullsized":"2024-03-16T19:30:42.624933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625681+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:30:42.625681+0000","last_peered":"2024-03-16T19:30:42.625681+0000","last_clean":"2024-03-16T19:30:42.625681+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:30:42.625681+0000","last_undegraded":"2024-03-16T19:30:42.625681+0000","last_fullsized":"2024-03-16T19:30:42.625681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153942+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:30:43.153942+0000","last_peered":"2024-03-16T19:30:43.153942+0000","last_clean":"2024-03-16T19:30:43.153942+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:30:43.153942+0000","last_undegraded":"2024-03-16T19:30:43.153942+0000","last_fullsized":"2024-03-16T19:30:43.153942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626177+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:30:42.626177+0000","last_peered":"2024-03-16T19:30:42.626177+0000","last_clean":"2024-03-16T19:30:42.626177+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:30:42.626177+0000","last_undegraded":"2024-03-16T19:30:42.626177+0000","last_fullsized":"2024-03-16T19:30:42.626177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153907+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:30:43.153907+0000","last_peered":"2024-03-16T19:30:43.153907+0000","last_clean":"2024-03-16T19:30:43.153907+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:30:43.153907+0000","last_undegraded":"2024-03-16T19:30:43.153907+0000","last_fullsized":"2024-03-16T19:30:43.153907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625439+0000","last_change":"2024-03-16T19:29:44.414141+0000","last_active":"2024-03-16T19:30:42.625439+0000","last_peered":"2024-03-16T19:30:42.625439+0000","last_clean":"2024-03-16T19:30:42.625439+0000","last_became_active":"2024-03-16T19:29:44.413873+0000","last_became_peered":"2024-03-16T19:29:44.413873+0000","last_unstale":"2024-03-16T19:30:42.625439+0000","last_undegraded":"2024-03-16T19:30:42.625439+0000","last_fullsized":"2024-03-16T19:30:42.625439+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:03:03.291206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":51,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153869+0000","last_change":"2024-03-16T19:30:07.591770+0000","last_active":"2024-03-16T19:30:43.153869+0000","last_peered":"2024-03-16T19:30:43.153869+0000","last_clean":"2024-03-16T19:30:43.153869+0000","last_became_active":"2024-03-16T19:30:07.591482+0000","last_became_peered":"2024-03-16T19:30:07.591482+0000","last_unstale":"2024-03-16T19:30:43.153869+0000","last_undegraded":"2024-03-16T19:30:43.153869+0000","last_fullsized":"2024-03-16T19:30:43.153869+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:02.003669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154268+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:30:43.154268+0000","last_peered":"2024-03-16T19:30:43.154268+0000","last_clean":"2024-03-16T19:30:43.154268+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:30:43.154268+0000","last_undegraded":"2024-03-16T19:30:43.154268+0000","last_fullsized":"2024-03-16T19:30:43.154268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624999+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:30:42.624999+0000","last_peered":"2024-03-16T19:30:42.624999+0000","last_clean":"2024-03-16T19:30:42.624999+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:30:42.624999+0000","last_undegraded":"2024-03-16T19:30:42.624999+0000","last_fullsized":"2024-03-16T19:30:42.624999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154707+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:30:43.154707+0000","last_peered":"2024-03-16T19:30:43.154707+0000","last_clean":"2024-03-16T19:30:43.154707+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:30:43.154707+0000","last_undegraded":"2024-03-16T19:30:43.154707+0000","last_fullsized":"2024-03-16T19:30:43.154707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154184+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:30:43.154184+0000","last_peered":"2024-03-16T19:30:43.154184+0000","last_clean":"2024-03-16T19:30:43.154184+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:30:43.154184+0000","last_undegraded":"2024-03-16T19:30:43.154184+0000","last_fullsized":"2024-03-16T19:30:43.154184+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625049+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:30:42.625049+0000","last_peered":"2024-03-16T19:30:42.625049+0000","last_clean":"2024-03-16T19:30:42.625049+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:30:42.625049+0000","last_undegraded":"2024-03-16T19:30:42.625049+0000","last_fullsized":"2024-03-16T19:30:42.625049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154678+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:30:43.154678+0000","last_peered":"2024-03-16T19:30:43.154678+0000","last_clean":"2024-03-16T19:30:43.154678+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:30:43.154678+0000","last_undegraded":"2024-03-16T19:30:43.154678+0000","last_fullsized":"2024-03-16T19:30:43.154678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":70,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154261+0000","last_change":"2024-03-16T19:29:44.412475+0000","last_active":"2024-03-16T19:30:43.154261+0000","last_peered":"2024-03-16T19:30:43.154261+0000","last_clean":"2024-03-16T19:30:43.154261+0000","last_became_active":"2024-03-16T19:29:44.412176+0000","last_became_peered":"2024-03-16T19:29:44.412176+0000","last_unstale":"2024-03-16T19:30:43.154261+0000","last_undegraded":"2024-03-16T19:30:43.154261+0000","last_fullsized":"2024-03-16T19:30:43.154261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:29:34.284744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625413+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:30:42.625413+0000","last_peered":"2024-03-16T19:30:42.625413+0000","last_clean":"2024-03-16T19:30:42.625413+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:30:42.625413+0000","last_undegraded":"2024-03-16T19:30:42.625413+0000","last_fullsized":"2024-03-16T19:30:42.625413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153955+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:30:43.153955+0000","last_peered":"2024-03-16T19:30:43.153955+0000","last_clean":"2024-03-16T19:30:43.153955+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:30:43.153955+0000","last_undegraded":"2024-03-16T19:30:43.153955+0000","last_fullsized":"2024-03-16T19:30:43.153955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153558+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:30:43.153558+0000","last_peered":"2024-03-16T19:30:43.153558+0000","last_clean":"2024-03-16T19:30:43.153558+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:30:43.153558+0000","last_undegraded":"2024-03-16T19:30:43.153558+0000","last_fullsized":"2024-03-16T19:30:43.153558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153694+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:30:43.153694+0000","last_peered":"2024-03-16T19:30:43.153694+0000","last_clean":"2024-03-16T19:30:43.153694+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:30:43.153694+0000","last_undegraded":"2024-03-16T19:30:43.153694+0000","last_fullsized":"2024-03-16T19:30:43.153694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154491+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:30:43.154491+0000","last_peered":"2024-03-16T19:30:43.154491+0000","last_clean":"2024-03-16T19:30:43.154491+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:30:43.154491+0000","last_undegraded":"2024-03-16T19:30:43.154491+0000","last_fullsized":"2024-03-16T19:30:43.154491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153829+0000","last_change":"2024-03-16T19:29:44.412541+0000","last_active":"2024-03-16T19:30:43.153829+0000","last_peered":"2024-03-16T19:30:43.153829+0000","last_clean":"2024-03-16T19:30:43.153829+0000","last_became_active":"2024-03-16T19:29:44.412194+0000","last_became_peered":"2024-03-16T19:29:44.412194+0000","last_unstale":"2024-03-16T19:30:43.153829+0000","last_undegraded":"2024-03-16T19:30:43.153829+0000","last_fullsized":"2024-03-16T19:30:43.153829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:51:50.130592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153796+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:30:43.153796+0000","last_peered":"2024-03-16T19:30:43.153796+0000","last_clean":"2024-03-16T19:30:43.153796+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:30:43.153796+0000","last_undegraded":"2024-03-16T19:30:43.153796+0000","last_fullsized":"2024-03-16T19:30:43.153796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154395+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:30:43.154395+0000","last_peered":"2024-03-16T19:30:43.154395+0000","last_clean":"2024-03-16T19:30:43.154395+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:30:43.154395+0000","last_undegraded":"2024-03-16T19:30:43.154395+0000","last_fullsized":"2024-03-16T19:30:43.154395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154304+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:30:43.154304+0000","last_peered":"2024-03-16T19:30:43.154304+0000","last_clean":"2024-03-16T19:30:43.154304+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:30:43.154304+0000","last_undegraded":"2024-03-16T19:30:43.154304+0000","last_fullsized":"2024-03-16T19:30:43.154304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625033+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:30:42.625033+0000","last_peered":"2024-03-16T19:30:42.625033+0000","last_clean":"2024-03-16T19:30:42.625033+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:30:42.625033+0000","last_undegraded":"2024-03-16T19:30:42.625033+0000","last_fullsized":"2024-03-16T19:30:42.625033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624775+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:30:42.624775+0000","last_peered":"2024-03-16T19:30:42.624775+0000","last_clean":"2024-03-16T19:30:42.624775+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:30:42.624775+0000","last_undegraded":"2024-03-16T19:30:42.624775+0000","last_fullsized":"2024-03-16T19:30:42.624775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":79,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153949+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:30:43.153949+0000","last_peered":"2024-03-16T19:30:43.153949+0000","last_clean":"2024-03-16T19:30:43.153949+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:30:43.153949+0000","last_undegraded":"2024-03-16T19:30:43.153949+0000","last_fullsized":"2024-03-16T19:30:43.153949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625329+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:30:42.625329+0000","last_peered":"2024-03-16T19:30:42.625329+0000","last_clean":"2024-03-16T19:30:42.625329+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:30:42.625329+0000","last_undegraded":"2024-03-16T19:30:42.625329+0000","last_fullsized":"2024-03-16T19:30:42.625329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625404+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:30:42.625404+0000","last_peered":"2024-03-16T19:30:42.625404+0000","last_clean":"2024-03-16T19:30:42.625404+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:30:42.625404+0000","last_undegraded":"2024-03-16T19:30:42.625404+0000","last_fullsized":"2024-03-16T19:30:42.625404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625603+0000","last_change":"2024-03-16T19:29:44.413558+0000","last_active":"2024-03-16T19:30:42.625603+0000","last_peered":"2024-03-16T19:30:42.625603+0000","last_clean":"2024-03-16T19:30:42.625603+0000","last_became_active":"2024-03-16T19:29:44.413310+0000","last_became_peered":"2024-03-16T19:29:44.413310+0000","last_unstale":"2024-03-16T19:30:42.625603+0000","last_undegraded":"2024-03-16T19:30:42.625603+0000","last_fullsized":"2024-03-16T19:30:42.625603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:25:53.846575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":81,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625087+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:30:42.625087+0000","last_peered":"2024-03-16T19:30:42.625087+0000","last_clean":"2024-03-16T19:30:42.625087+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:30:42.625087+0000","last_undegraded":"2024-03-16T19:30:42.625087+0000","last_fullsized":"2024-03-16T19:30:42.625087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153983+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:30:43.153983+0000","last_peered":"2024-03-16T19:30:43.153983+0000","last_clean":"2024-03-16T19:30:43.153983+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:30:43.153983+0000","last_undegraded":"2024-03-16T19:30:43.153983+0000","last_fullsized":"2024-03-16T19:30:43.153983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"36'11","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153915+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:30:43.153915+0000","last_peered":"2024-03-16T19:30:43.153915+0000","last_clean":"2024-03-16T19:30:43.153915+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:30:43.153915+0000","last_undegraded":"2024-03-16T19:30:43.153915+0000","last_fullsized":"2024-03-16T19:30:43.153915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6662,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626092+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:30:42.626092+0000","last_peered":"2024-03-16T19:30:42.626092+0000","last_clean":"2024-03-16T19:30:42.626092+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:30:42.626092+0000","last_undegraded":"2024-03-16T19:30:42.626092+0000","last_fullsized":"2024-03-16T19:30:42.626092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":86,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153923+0000","last_change":"2024-03-16T19:29:44.412731+0000","last_active":"2024-03-16T19:30:43.153923+0000","last_peered":"2024-03-16T19:30:43.153923+0000","last_clean":"2024-03-16T19:30:43.153923+0000","last_became_active":"2024-03-16T19:29:44.412174+0000","last_became_peered":"2024-03-16T19:29:44.412174+0000","last_unstale":"2024-03-16T19:30:43.153923+0000","last_undegraded":"2024-03-16T19:30:43.153923+0000","last_fullsized":"2024-03-16T19:30:43.153923+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:57.962644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625309+0000","last_change":"2024-03-16T19:29:44.412524+0000","last_active":"2024-03-16T19:30:42.625309+0000","last_peered":"2024-03-16T19:30:42.625309+0000","last_clean":"2024-03-16T19:30:42.625309+0000","last_became_active":"2024-03-16T19:29:44.412200+0000","last_became_peered":"2024-03-16T19:29:44.412200+0000","last_unstale":"2024-03-16T19:30:42.625309+0000","last_undegraded":"2024-03-16T19:30:42.625309+0000","last_fullsized":"2024-03-16T19:30:42.625309+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:52.636929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625443+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:30:42.625443+0000","last_peered":"2024-03-16T19:30:42.625443+0000","last_clean":"2024-03-16T19:30:42.625443+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:30:42.625443+0000","last_undegraded":"2024-03-16T19:30:42.625443+0000","last_fullsized":"2024-03-16T19:30:42.625443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625621+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:30:42.625621+0000","last_peered":"2024-03-16T19:30:42.625621+0000","last_clean":"2024-03-16T19:30:42.625621+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:30:42.625621+0000","last_undegraded":"2024-03-16T19:30:42.625621+0000","last_fullsized":"2024-03-16T19:30:42.625621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":79,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153693+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:30:43.153693+0000","last_peered":"2024-03-16T19:30:43.153693+0000","last_clean":"2024-03-16T19:30:43.153693+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:30:43.153693+0000","last_undegraded":"2024-03-16T19:30:43.153693+0000","last_fullsized":"2024-03-16T19:30:43.153693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153595+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:30:43.153595+0000","last_peered":"2024-03-16T19:30:43.153595+0000","last_clean":"2024-03-16T19:30:43.153595+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:30:43.153595+0000","last_undegraded":"2024-03-16T19:30:43.153595+0000","last_fullsized":"2024-03-16T19:30:43.153595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":85,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153826+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:30:43.153826+0000","last_peered":"2024-03-16T19:30:43.153826+0000","last_clean":"2024-03-16T19:30:43.153826+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:30:43.153826+0000","last_undegraded":"2024-03-16T19:30:43.153826+0000","last_fullsized":"2024-03-16T19:30:43.153826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625278+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:30:42.625278+0000","last_peered":"2024-03-16T19:30:42.625278+0000","last_clean":"2024-03-16T19:30:42.625278+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:30:42.625278+0000","last_undegraded":"2024-03-16T19:30:42.625278+0000","last_fullsized":"2024-03-16T19:30:42.625278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":81,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154098+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:30:43.154098+0000","last_peered":"2024-03-16T19:30:43.154098+0000","last_clean":"2024-03-16T19:30:43.154098+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:30:43.154098+0000","last_undegraded":"2024-03-16T19:30:43.154098+0000","last_fullsized":"2024-03-16T19:30:43.154098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626148+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:30:42.626148+0000","last_peered":"2024-03-16T19:30:42.626148+0000","last_clean":"2024-03-16T19:30:42.626148+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:30:42.626148+0000","last_undegraded":"2024-03-16T19:30:42.626148+0000","last_fullsized":"2024-03-16T19:30:42.626148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":79,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154025+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:30:43.154025+0000","last_peered":"2024-03-16T19:30:43.154025+0000","last_clean":"2024-03-16T19:30:43.154025+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:30:43.154025+0000","last_undegraded":"2024-03-16T19:30:43.154025+0000","last_fullsized":"2024-03-16T19:30:43.154025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626246+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:30:42.626246+0000","last_peered":"2024-03-16T19:30:42.626246+0000","last_clean":"2024-03-16T19:30:42.626246+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:30:42.626246+0000","last_undegraded":"2024-03-16T19:30:42.626246+0000","last_fullsized":"2024-03-16T19:30:42.626246+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":79,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625140+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:30:42.625140+0000","last_peered":"2024-03-16T19:30:42.625140+0000","last_clean":"2024-03-16T19:30:42.625140+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:30:42.625140+0000","last_undegraded":"2024-03-16T19:30:42.625140+0000","last_fullsized":"2024-03-16T19:30:42.625140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625841+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:30:42.625841+0000","last_peered":"2024-03-16T19:30:42.625841+0000","last_clean":"2024-03-16T19:30:42.625841+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:30:42.625841+0000","last_undegraded":"2024-03-16T19:30:42.625841+0000","last_fullsized":"2024-03-16T19:30:42.625841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153929+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:30:43.153929+0000","last_peered":"2024-03-16T19:30:43.153929+0000","last_clean":"2024-03-16T19:30:43.153929+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:30:43.153929+0000","last_undegraded":"2024-03-16T19:30:43.153929+0000","last_fullsized":"2024-03-16T19:30:43.153929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":48,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626062+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:30:42.626062+0000","last_peered":"2024-03-16T19:30:42.626062+0000","last_clean":"2024-03-16T19:30:42.626062+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:30:42.626062+0000","last_undegraded":"2024-03-16T19:30:42.626062+0000","last_fullsized":"2024-03-16T19:30:42.626062+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":236,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626051+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:30:42.626051+0000","last_peered":"2024-03-16T19:30:42.626051+0000","last_clean":"2024-03-16T19:30:42.626051+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:30:42.626051+0000","last_undegraded":"2024-03-16T19:30:42.626051+0000","last_fullsized":"2024-03-16T19:30:42.626051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625170+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:30:42.625170+0000","last_peered":"2024-03-16T19:30:42.625170+0000","last_clean":"2024-03-16T19:30:42.625170+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:30:42.625170+0000","last_undegraded":"2024-03-16T19:30:42.625170+0000","last_fullsized":"2024-03-16T19:30:42.625170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154274+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:30:43.154274+0000","last_peered":"2024-03-16T19:30:43.154274+0000","last_clean":"2024-03-16T19:30:43.154274+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:30:43.154274+0000","last_undegraded":"2024-03-16T19:30:43.154274+0000","last_fullsized":"2024-03-16T19:30:43.154274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625946+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:30:42.625946+0000","last_peered":"2024-03-16T19:30:42.625946+0000","last_clean":"2024-03-16T19:30:42.625946+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:30:42.625946+0000","last_undegraded":"2024-03-16T19:30:42.625946+0000","last_fullsized":"2024-03-16T19:30:42.625946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153701+0000","last_change":"2024-03-16T19:29:44.412505+0000","last_active":"2024-03-16T19:30:43.153701+0000","last_peered":"2024-03-16T19:30:43.153701+0000","last_clean":"2024-03-16T19:30:43.153701+0000","last_became_active":"2024-03-16T19:29:44.412221+0000","last_became_peered":"2024-03-16T19:29:44.412221+0000","last_unstale":"2024-03-16T19:30:43.153701+0000","last_undegraded":"2024-03-16T19:30:43.153701+0000","last_fullsized":"2024-03-16T19:30:43.153701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:14.739638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154043+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:30:43.154043+0000","last_peered":"2024-03-16T19:30:43.154043+0000","last_clean":"2024-03-16T19:30:43.154043+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:30:43.154043+0000","last_undegraded":"2024-03-16T19:30:43.154043+0000","last_fullsized":"2024-03-16T19:30:43.154043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624818+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:30:42.624818+0000","last_peered":"2024-03-16T19:30:42.624818+0000","last_clean":"2024-03-16T19:30:42.624818+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:30:42.624818+0000","last_undegraded":"2024-03-16T19:30:42.624818+0000","last_fullsized":"2024-03-16T19:30:42.624818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625514+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:30:42.625514+0000","last_peered":"2024-03-16T19:30:42.625514+0000","last_clean":"2024-03-16T19:30:42.625514+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:30:42.625514+0000","last_undegraded":"2024-03-16T19:30:42.625514+0000","last_fullsized":"2024-03-16T19:30:42.625514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154015+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:30:43.154015+0000","last_peered":"2024-03-16T19:30:43.154015+0000","last_clean":"2024-03-16T19:30:43.154015+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:30:43.154015+0000","last_undegraded":"2024-03-16T19:30:43.154015+0000","last_fullsized":"2024-03-16T19:30:43.154015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624939+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:30:42.624939+0000","last_peered":"2024-03-16T19:30:42.624939+0000","last_clean":"2024-03-16T19:30:42.624939+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:30:42.624939+0000","last_undegraded":"2024-03-16T19:30:42.624939+0000","last_fullsized":"2024-03-16T19:30:42.624939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625869+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:30:42.625869+0000","last_peered":"2024-03-16T19:30:42.625869+0000","last_clean":"2024-03-16T19:30:42.625869+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:30:42.625869+0000","last_undegraded":"2024-03-16T19:30:42.625869+0000","last_fullsized":"2024-03-16T19:30:42.625869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626256+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:30:42.626256+0000","last_peered":"2024-03-16T19:30:42.626256+0000","last_clean":"2024-03-16T19:30:42.626256+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:30:42.626256+0000","last_undegraded":"2024-03-16T19:30:42.626256+0000","last_fullsized":"2024-03-16T19:30:42.626256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624847+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:30:42.624847+0000","last_peered":"2024-03-16T19:30:42.624847+0000","last_clean":"2024-03-16T19:30:42.624847+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:30:42.624847+0000","last_undegraded":"2024-03-16T19:30:42.624847+0000","last_fullsized":"2024-03-16T19:30:42.624847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"40'4","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625221+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:30:42.625221+0000","last_peered":"2024-03-16T19:30:42.625221+0000","last_clean":"2024-03-16T19:30:42.625221+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:30:42.625221+0000","last_undegraded":"2024-03-16T19:30:42.625221+0000","last_fullsized":"2024-03-16T19:30:42.625221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154796+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:30:43.154796+0000","last_peered":"2024-03-16T19:30:43.154796+0000","last_clean":"2024-03-16T19:30:43.154796+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:30:43.154796+0000","last_undegraded":"2024-03-16T19:30:43.154796+0000","last_fullsized":"2024-03-16T19:30:43.154796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":81,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154039+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:30:43.154039+0000","last_peered":"2024-03-16T19:30:43.154039+0000","last_clean":"2024-03-16T19:30:43.154039+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:30:43.154039+0000","last_undegraded":"2024-03-16T19:30:43.154039+0000","last_fullsized":"2024-03-16T19:30:43.154039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625098+0000","last_change":"2024-03-16T19:29:44.413626+0000","last_active":"2024-03-16T19:30:42.625098+0000","last_peered":"2024-03-16T19:30:42.625098+0000","last_clean":"2024-03-16T19:30:42.625098+0000","last_became_active":"2024-03-16T19:29:44.413174+0000","last_became_peered":"2024-03-16T19:29:44.413174+0000","last_unstale":"2024-03-16T19:30:42.625098+0000","last_undegraded":"2024-03-16T19:30:42.625098+0000","last_fullsized":"2024-03-16T19:30:42.625098+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:10:39.043849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624916+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:30:42.624916+0000","last_peered":"2024-03-16T19:30:42.624916+0000","last_clean":"2024-03-16T19:30:42.624916+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:30:42.624916+0000","last_undegraded":"2024-03-16T19:30:42.624916+0000","last_fullsized":"2024-03-16T19:30:42.624916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153787+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:30:43.153787+0000","last_peered":"2024-03-16T19:30:43.153787+0000","last_clean":"2024-03-16T19:30:43.153787+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:30:43.153787+0000","last_undegraded":"2024-03-16T19:30:43.153787+0000","last_fullsized":"2024-03-16T19:30:43.153787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624975+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:30:42.624975+0000","last_peered":"2024-03-16T19:30:42.624975+0000","last_clean":"2024-03-16T19:30:42.624975+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:30:42.624975+0000","last_undegraded":"2024-03-16T19:30:42.624975+0000","last_fullsized":"2024-03-16T19:30:42.624975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":51,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153750+0000","last_change":"2024-03-16T19:30:07.477317+0000","last_active":"2024-03-16T19:30:43.153750+0000","last_peered":"2024-03-16T19:30:43.153750+0000","last_clean":"2024-03-16T19:30:43.153750+0000","last_became_active":"2024-03-16T19:30:07.477037+0000","last_became_peered":"2024-03-16T19:30:07.477037+0000","last_unstale":"2024-03-16T19:30:43.153750+0000","last_undegraded":"2024-03-16T19:30:43.153750+0000","last_fullsized":"2024-03-16T19:30:43.153750+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:59:02.992861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625053+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:30:42.625053+0000","last_peered":"2024-03-16T19:30:42.625053+0000","last_clean":"2024-03-16T19:30:42.625053+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:30:42.625053+0000","last_undegraded":"2024-03-16T19:30:42.625053+0000","last_fullsized":"2024-03-16T19:30:42.625053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154478+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:30:43.154478+0000","last_peered":"2024-03-16T19:30:43.154478+0000","last_clean":"2024-03-16T19:30:43.154478+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:30:43.154478+0000","last_undegraded":"2024-03-16T19:30:43.154478+0000","last_fullsized":"2024-03-16T19:30:43.154478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625716+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:30:42.625716+0000","last_peered":"2024-03-16T19:30:42.625716+0000","last_clean":"2024-03-16T19:30:42.625716+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:30:42.625716+0000","last_undegraded":"2024-03-16T19:30:42.625716+0000","last_fullsized":"2024-03-16T19:30:42.625716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153695+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:30:43.153695+0000","last_peered":"2024-03-16T19:30:43.153695+0000","last_clean":"2024-03-16T19:30:43.153695+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:30:43.153695+0000","last_undegraded":"2024-03-16T19:30:43.153695+0000","last_fullsized":"2024-03-16T19:30:43.153695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625836+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:30:42.625836+0000","last_peered":"2024-03-16T19:30:42.625836+0000","last_clean":"2024-03-16T19:30:42.625836+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:30:42.625836+0000","last_undegraded":"2024-03-16T19:30:42.625836+0000","last_fullsized":"2024-03-16T19:30:42.625836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625943+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:30:42.625943+0000","last_peered":"2024-03-16T19:30:42.625943+0000","last_clean":"2024-03-16T19:30:42.625943+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:30:42.625943+0000","last_undegraded":"2024-03-16T19:30:42.625943+0000","last_fullsized":"2024-03-16T19:30:42.625943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625745+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:30:42.625745+0000","last_peered":"2024-03-16T19:30:42.625745+0000","last_clean":"2024-03-16T19:30:42.625745+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:30:42.625745+0000","last_undegraded":"2024-03-16T19:30:42.625745+0000","last_fullsized":"2024-03-16T19:30:42.625745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626158+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:30:42.626158+0000","last_peered":"2024-03-16T19:30:42.626158+0000","last_clean":"2024-03-16T19:30:42.626158+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:30:42.626158+0000","last_undegraded":"2024-03-16T19:30:42.626158+0000","last_fullsized":"2024-03-16T19:30:42.626158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":52,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625254+0000","last_change":"2024-03-16T19:30:07.684215+0000","last_active":"2024-03-16T19:30:42.625254+0000","last_peered":"2024-03-16T19:30:42.625254+0000","last_clean":"2024-03-16T19:30:42.625254+0000","last_became_active":"2024-03-16T19:30:07.683959+0000","last_became_peered":"2024-03-16T19:30:07.683959+0000","last_unstale":"2024-03-16T19:30:42.625254+0000","last_undegraded":"2024-03-16T19:30:42.625254+0000","last_fullsized":"2024-03-16T19:30:42.625254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:23:17.567774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154874+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:30:43.154874+0000","last_peered":"2024-03-16T19:30:43.154874+0000","last_clean":"2024-03-16T19:30:43.154874+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:30:43.154874+0000","last_undegraded":"2024-03-16T19:30:43.154874+0000","last_fullsized":"2024-03-16T19:30:43.154874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154783+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:30:43.154783+0000","last_peered":"2024-03-16T19:30:43.154783+0000","last_clean":"2024-03-16T19:30:43.154783+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:30:43.154783+0000","last_undegraded":"2024-03-16T19:30:43.154783+0000","last_fullsized":"2024-03-16T19:30:43.154783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":86,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626277+0000","last_change":"2024-03-16T19:29:44.414207+0000","last_active":"2024-03-16T19:30:42.626277+0000","last_peered":"2024-03-16T19:30:42.626277+0000","last_clean":"2024-03-16T19:30:42.626277+0000","last_became_active":"2024-03-16T19:29:44.413852+0000","last_became_peered":"2024-03-16T19:29:44.413852+0000","last_unstale":"2024-03-16T19:30:42.626277+0000","last_undegraded":"2024-03-16T19:30:42.626277+0000","last_fullsized":"2024-03-16T19:30:42.626277+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:25.285467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7460,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":14920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":1208320,"data_stored":859121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":274833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":274297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":634880,"data_stored":274333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":280869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":268171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":630784,"data_stored":274367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1216512,"data_stored":859177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":6662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":6662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:30:45.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:30:45.054 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:45.074+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:45.123 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:45.123 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:45.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:45.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.193+0000 7f77e604e640 1 Processor -- start 2024-03-16T19:30:45.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.193+0000 7f77e604e640 1 -- start start 2024-03-16T19:30:45.175 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:45.194+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:45.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77e604e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e0152a20 0x7f77e0159190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77e604e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77e0072dd0 0x7f77e00c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77e604e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f77e00c93c0 0x7f77e00c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77e604e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e0152e00 con 0x7f77e0072dd0 2024-03-16T19:30:45.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77e604e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e015c610 con 0x7f77e00c93c0 2024-03-16T19:30:45.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77e604e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e01596d0 con 0x7f77e0152a20 2024-03-16T19:30:45.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77deffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77e0072dd0 0x7f77e00c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77df7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e0152a20 0x7f77e0159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77deffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f77e0072dd0 0x7f77e00c8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57518/0 (socket says 172.21.15.172:57518) 2024-03-16T19:30:45.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77dffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f77e00c93c0 0x7f77e00c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.194+0000 7f77deffd640 1 -- 172.21.15.172:0/389615161 learned_addr learned my addr 172.21.15.172:0/389615161 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:45.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.195+0000 7f77dffff640 1 -- 172.21.15.172:0/389615161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e0152a20 msgr2=0x7f77e0159190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:45.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.195+0000 7f77dffff640 1 --2- 172.21.15.172:0/389615161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e0152a20 0x7f77e0159190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.195+0000 7f77dffff640 1 -- 172.21.15.172:0/389615161 >> v2:172.21.15.172:3300/0 conn(0x7f77e0072dd0 msgr2=0x7f77e00c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:45.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.195+0000 7f77dffff640 1 --2- 172.21.15.172:0/389615161 >> v2:172.21.15.172:3300/0 conn(0x7f77e0072dd0 0x7f77e00c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.195+0000 7f77dffff640 1 -- 172.21.15.172:0/389615161 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e00c9010 con 0x7f77e00c93c0 2024-03-16T19:30:45.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.195+0000 7f77dffff640 1 --2- 172.21.15.172:0/389615161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e00c93c0 0x7f77e00c7b40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f77e00706d0 tx=0x7f77d802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73cc30b175f2a6d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:45.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.196+0000 7f77de7fc640 1 -- 172.21.15.172:0/389615161 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d802cc60 con 0x7f77e00c93c0 2024-03-16T19:30:45.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.196+0000 7f77de7fc640 1 -- 172.21.15.172:0/389615161 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d802cdc0 con 0x7f77e00c93c0 2024-03-16T19:30:45.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.196+0000 7f77de7fc640 1 -- 172.21.15.172:0/389615161 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8034660 con 0x7f77e00c93c0 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.196+0000 7f77e604e640 1 -- 172.21.15.172:0/389615161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e00c93c0 msgr2=0x7f77e00c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.196+0000 7f77e604e640 1 --2- 172.21.15.172:0/389615161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e00c93c0 0x7f77e00c7b40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f77e00706d0 tx=0x7f77d802beb0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.197+0000 7f77e604e640 1 -- 172.21.15.172:0/389615161 shutdown_connections 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.197+0000 7f77e604e640 1 --2- 172.21.15.172:0/389615161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e00c93c0 0x7f77e00c7b40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.197+0000 7f77e604e640 1 --2- 172.21.15.172:0/389615161 >> v2:172.21.15.172:3300/0 conn(0x7f77e0072dd0 0x7f77e00c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.197+0000 7f77e604e640 1 --2- 172.21.15.172:0/389615161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e0152a20 0x7f77e0159190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.197+0000 7f77e604e640 1 -- 172.21.15.172:0/389615161 >> 172.21.15.172:0/389615161 conn(0x7f77e006f6f0 msgr2=0x7f77e006fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.197+0000 7f77e604e640 1 -- 172.21.15.172:0/389615161 shutdown_connections 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.197+0000 7f77e604e640 1 -- 172.21.15.172:0/389615161 wait complete. 2024-03-16T19:30:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.198+0000 7f77e604e640 1 Processor -- start 2024-03-16T19:30:45.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.199+0000 7f77e604e640 1 -- start start 2024-03-16T19:30:45.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77e604e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e0072dd0 0x7f77e0143f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77e604e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e00c93c0 0x7f77e01563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77e604e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e0152a20 0x7f77e00b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77e604e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e0144450 con 0x7f77e0152a20 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77e604e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e0156e90 con 0x7f77e0072dd0 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77e604e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e0155230 con 0x7f77e00c93c0 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77deffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e00c93c0 0x7f77e01563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77dffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e0152a20 0x7f77e00b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77df7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e0072dd0 0x7f77e0143f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77df7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e0072dd0 0x7f77e0143f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40640/0 (socket says 172.21.15.172:40640) 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77deffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e00c93c0 0x7f77e01563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48152/0 (socket says 172.21.15.172:48152) 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77df7fe640 1 -- 172.21.15.172:0/1462436868 learned_addr learned my addr 172.21.15.172:0/1462436868 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77deffd640 1 -- 172.21.15.172:0/1462436868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e0072dd0 msgr2=0x7f77e0143f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77deffd640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e0072dd0 0x7f77e0143f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77deffd640 1 -- 172.21.15.172:0/1462436868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e0152a20 msgr2=0x7f77e00b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77deffd640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e0152a20 0x7f77e00b1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77deffd640 1 -- 172.21.15.172:0/1462436868 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d8009a70 con 0x7f77e00c93c0 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.200+0000 7f77df7fe640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e0072dd0 0x7f77e0143f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:45.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.201+0000 7f77deffd640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e00c93c0 0x7f77e01563e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f77d400e850 tx=0x7f77d400ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:45.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.201+0000 7f77c7fff640 1 -- 172.21.15.172:0/1462436868 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d400c940 con 0x7f77e00c93c0 2024-03-16T19:30:45.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.201+0000 7f77c7fff640 1 -- 172.21.15.172:0/1462436868 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d400caa0 con 0x7f77e00c93c0 2024-03-16T19:30:45.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.201+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e0155510 con 0x7f77e00c93c0 2024-03-16T19:30:45.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.201+0000 7f77c7fff640 1 -- 172.21.15.172:0/1462436868 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d400ce10 con 0x7f77e00c93c0 2024-03-16T19:30:45.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.201+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e0155ad0 con 0x7f77e00c93c0 2024-03-16T19:30:45.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.202+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e00b0060 con 0x7f77e00c93c0 2024-03-16T19:30:45.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.202+0000 7f77c7fff640 1 -- 172.21.15.172:0/1462436868 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f77d4013470 con 0x7f77e00c93c0 2024-03-16T19:30:45.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.203+0000 7f77c7fff640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77cc06b3e0 0x7f77cc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.203+0000 7f77c7fff640 1 -- 172.21.15.172:0/1462436868 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f77d4089e20 con 0x7f77e00c93c0 2024-03-16T19:30:45.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.203+0000 7f77df7fe640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77cc06b3e0 0x7f77cc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.203+0000 7f77df7fe640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77cc06b3e0 0x7f77cc06d8a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009ed0 tx=0x7f77d0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:45.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.207+0000 7f77c7fff640 1 -- 172.21.15.172:0/1462436868 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77d4054a20 con 0x7f77e00c93c0 2024-03-16T19:30:45.250 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:45.250 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:45.256 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:45.276+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:45.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:45.356 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:45.376+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:45.373 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.391+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77e015c800 con 0x7f77cc06b3e0 2024-03-16T19:30:45.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.403+0000 7f77c7fff640 1 -- 172.21.15.172:0/1462436868 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251879 (secure 0 0 0) 0x7f77d0009b20 con 0x7f77cc06b3e0 2024-03-16T19:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:30:45.389 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:30:45.390 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77cc06b3e0 msgr2=0x7f77cc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.390 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77cc06b3e0 0x7f77cc06d8a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009ed0 tx=0x7f77d0009190 comp rx=0 tx=0).stop 2024-03-16T19:30:45.390 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e00c93c0 msgr2=0x7f77e01563e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e00c93c0 0x7f77e01563e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f77d400e850 tx=0x7f77d400ed10 comp rx=0 tx=0).stop 2024-03-16T19:30:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 shutdown_connections 2024-03-16T19:30:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77e0152a20 0x7f77e00b1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f77cc06b3e0 0x7f77cc06d8a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77e00c93c0 0x7f77e01563e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 --2- 172.21.15.172:0/1462436868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77e0072dd0 0x7f77e0143f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 >> 172.21.15.172:0/1462436868 conn(0x7f77e006f6f0 msgr2=0x7f77e0071a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.406+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 shutdown_connections 2024-03-16T19:30:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.407+0000 7f77e604e640 1 -- 172.21.15.172:0/1462436868 wait complete. 2024-03-16T19:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-03-16T19:30:44.629439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597828,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":216,"num_write_kb":2441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164,"ondisk_log_size":164,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222252,"kb_used_data":6048,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752468,"statfs":{"total":773094113280,"available":772866527232,"internally_reserved":0,"allocated":6193152,"data_stored":3365168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014294"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626153+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:30:42.626153+0000","last_peered":"2024-03-16T19:30:42.626153+0000","last_clean":"2024-03-16T19:30:42.626153+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:30:42.626153+0000","last_undegraded":"2024-03-16T19:30:42.626153+0000","last_fullsized":"2024-03-16T19:30:42.626153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625194+0000","last_change":"2024-03-16T19:29:26.348992+0000","last_active":"2024-03-16T19:30:42.625194+0000","last_peered":"2024-03-16T19:30:42.625194+0000","last_clean":"2024-03-16T19:30:42.625194+0000","last_became_active":"2024-03-16T19:29:26.348699+0000","last_became_peered":"2024-03-16T19:29:26.348699+0000","last_unstale":"2024-03-16T19:30:42.625194+0000","last_undegraded":"2024-03-16T19:30:42.625194+0000","last_fullsized":"2024-03-16T19:30:42.625194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:38.330510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":70,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154027+0000","last_change":"2024-03-16T19:29:44.412537+0000","last_active":"2024-03-16T19:30:43.154027+0000","last_peered":"2024-03-16T19:30:43.154027+0000","last_clean":"2024-03-16T19:30:43.154027+0000","last_became_active":"2024-03-16T19:29:44.412236+0000","last_became_peered":"2024-03-16T19:29:44.412236+0000","last_unstale":"2024-03-16T19:30:43.154027+0000","last_undegraded":"2024-03-16T19:30:43.154027+0000","last_fullsized":"2024-03-16T19:30:43.154027+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:50:57.180910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625952+0000","last_change":"2024-03-16T19:29:26.358107+0000","last_active":"2024-03-16T19:30:42.625952+0000","last_peered":"2024-03-16T19:30:42.625952+0000","last_clean":"2024-03-16T19:30:42.625952+0000","last_became_active":"2024-03-16T19:29:26.357555+0000","last_became_peered":"2024-03-16T19:29:26.357555+0000","last_unstale":"2024-03-16T19:30:42.625952+0000","last_undegraded":"2024-03-16T19:30:42.625952+0000","last_fullsized":"2024-03-16T19:30:42.625952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:51.046069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625531+0000","last_change":"2024-03-16T19:29:26.354929+0000","last_active":"2024-03-16T19:30:42.625531+0000","last_peered":"2024-03-16T19:30:42.625531+0000","last_clean":"2024-03-16T19:30:42.625531+0000","last_became_active":"2024-03-16T19:29:26.354646+0000","last_became_peered":"2024-03-16T19:29:26.354646+0000","last_unstale":"2024-03-16T19:30:42.625531+0000","last_undegraded":"2024-03-16T19:30:42.625531+0000","last_fullsized":"2024-03-16T19:30:42.625531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:46:09.397285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625348+0000","last_change":"2024-03-16T19:29:26.361324+0000","last_active":"2024-03-16T19:30:42.625348+0000","last_peered":"2024-03-16T19:30:42.625348+0000","last_clean":"2024-03-16T19:30:42.625348+0000","last_became_active":"2024-03-16T19:29:26.360418+0000","last_became_peered":"2024-03-16T19:29:26.360418+0000","last_unstale":"2024-03-16T19:30:42.625348+0000","last_undegraded":"2024-03-16T19:30:42.625348+0000","last_fullsized":"2024-03-16T19:30:42.625348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:29.974036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625186+0000","last_change":"2024-03-16T19:29:26.354747+0000","last_active":"2024-03-16T19:30:42.625186+0000","last_peered":"2024-03-16T19:30:42.625186+0000","last_clean":"2024-03-16T19:30:42.625186+0000","last_became_active":"2024-03-16T19:29:26.354410+0000","last_became_peered":"2024-03-16T19:29:26.354410+0000","last_unstale":"2024-03-16T19:30:42.625186+0000","last_undegraded":"2024-03-16T19:30:42.625186+0000","last_fullsized":"2024-03-16T19:30:42.625186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:46.497715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154267+0000","last_change":"2024-03-16T19:29:26.353214+0000","last_active":"2024-03-16T19:30:43.154267+0000","last_peered":"2024-03-16T19:30:43.154267+0000","last_clean":"2024-03-16T19:30:43.154267+0000","last_became_active":"2024-03-16T19:29:26.352961+0000","last_became_peered":"2024-03-16T19:29:26.352961+0000","last_unstale":"2024-03-16T19:30:43.154267+0000","last_undegraded":"2024-03-16T19:30:43.154267+0000","last_fullsized":"2024-03-16T19:30:43.154267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:08:56.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626201+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:30:42.626201+0000","last_peered":"2024-03-16T19:30:42.626201+0000","last_clean":"2024-03-16T19:30:42.626201+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:30:42.626201+0000","last_undegraded":"2024-03-16T19:30:42.626201+0000","last_fullsized":"2024-03-16T19:30:42.626201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154122+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:30:43.154122+0000","last_peered":"2024-03-16T19:30:43.154122+0000","last_clean":"2024-03-16T19:30:43.154122+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:30:43.154122+0000","last_undegraded":"2024-03-16T19:30:43.154122+0000","last_fullsized":"2024-03-16T19:30:43.154122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154075+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:30:43.154075+0000","last_peered":"2024-03-16T19:30:43.154075+0000","last_clean":"2024-03-16T19:30:43.154075+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:30:43.154075+0000","last_undegraded":"2024-03-16T19:30:43.154075+0000","last_fullsized":"2024-03-16T19:30:43.154075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153964+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:30:43.153964+0000","last_peered":"2024-03-16T19:30:43.153964+0000","last_clean":"2024-03-16T19:30:43.153964+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:30:43.153964+0000","last_undegraded":"2024-03-16T19:30:43.153964+0000","last_fullsized":"2024-03-16T19:30:43.153964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154374+0000","last_change":"2024-03-16T19:29:26.359593+0000","last_active":"2024-03-16T19:30:43.154374+0000","last_peered":"2024-03-16T19:30:43.154374+0000","last_clean":"2024-03-16T19:30:43.154374+0000","last_became_active":"2024-03-16T19:29:26.359331+0000","last_became_peered":"2024-03-16T19:29:26.359331+0000","last_unstale":"2024-03-16T19:30:43.154374+0000","last_undegraded":"2024-03-16T19:30:43.154374+0000","last_fullsized":"2024-03-16T19:30:43.154374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:48.141243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624981+0000","last_change":"2024-03-16T19:29:44.413594+0000","last_active":"2024-03-16T19:30:42.624981+0000","last_peered":"2024-03-16T19:30:42.624981+0000","last_clean":"2024-03-16T19:30:42.624981+0000","last_became_active":"2024-03-16T19:29:44.413235+0000","last_became_peered":"2024-03-16T19:29:44.413235+0000","last_unstale":"2024-03-16T19:30:42.624981+0000","last_undegraded":"2024-03-16T19:30:42.624981+0000","last_fullsized":"2024-03-16T19:30:42.624981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:35.778808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624886+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:30:42.624886+0000","last_peered":"2024-03-16T19:30:42.624886+0000","last_clean":"2024-03-16T19:30:42.624886+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:30:42.624886+0000","last_undegraded":"2024-03-16T19:30:42.624886+0000","last_fullsized":"2024-03-16T19:30:42.624886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624933+0000","last_change":"2024-03-16T19:29:26.352087+0000","last_active":"2024-03-16T19:30:42.624933+0000","last_peered":"2024-03-16T19:30:42.624933+0000","last_clean":"2024-03-16T19:30:42.624933+0000","last_became_active":"2024-03-16T19:29:26.351690+0000","last_became_peered":"2024-03-16T19:29:26.351690+0000","last_unstale":"2024-03-16T19:30:42.624933+0000","last_undegraded":"2024-03-16T19:30:42.624933+0000","last_fullsized":"2024-03-16T19:30:42.624933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:35:47.433247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625681+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:30:42.625681+0000","last_peered":"2024-03-16T19:30:42.625681+0000","last_clean":"2024-03-16T19:30:42.625681+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:30:42.625681+0000","last_undegraded":"2024-03-16T19:30:42.625681+0000","last_fullsized":"2024-03-16T19:30:42.625681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153942+0000","last_change":"2024-03-16T19:29:26.363630+0000","last_active":"2024-03-16T19:30:43.153942+0000","last_peered":"2024-03-16T19:30:43.153942+0000","last_clean":"2024-03-16T19:30:43.153942+0000","last_became_active":"2024-03-16T19:29:26.363334+0000","last_became_peered":"2024-03-16T19:29:26.363334+0000","last_unstale":"2024-03-16T19:30:43.153942+0000","last_undegraded":"2024-03-16T19:30:43.153942+0000","last_fullsized":"2024-03-16T19:30:43.153942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:25:25.308020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626177+0000","last_change":"2024-03-16T19:29:26.355389+0000","last_active":"2024-03-16T19:30:42.626177+0000","last_peered":"2024-03-16T19:30:42.626177+0000","last_clean":"2024-03-16T19:30:42.626177+0000","last_became_active":"2024-03-16T19:29:26.355120+0000","last_became_peered":"2024-03-16T19:29:26.355120+0000","last_unstale":"2024-03-16T19:30:42.626177+0000","last_undegraded":"2024-03-16T19:30:42.626177+0000","last_fullsized":"2024-03-16T19:30:42.626177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:44:07.659200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153907+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:30:43.153907+0000","last_peered":"2024-03-16T19:30:43.153907+0000","last_clean":"2024-03-16T19:30:43.153907+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:30:43.153907+0000","last_undegraded":"2024-03-16T19:30:43.153907+0000","last_fullsized":"2024-03-16T19:30:43.153907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625439+0000","last_change":"2024-03-16T19:29:44.414141+0000","last_active":"2024-03-16T19:30:42.625439+0000","last_peered":"2024-03-16T19:30:42.625439+0000","last_clean":"2024-03-16T19:30:42.625439+0000","last_became_active":"2024-03-16T19:29:44.413873+0000","last_became_peered":"2024-03-16T19:29:44.413873+0000","last_unstale":"2024-03-16T19:30:42.625439+0000","last_undegraded":"2024-03-16T19:30:42.625439+0000","last_fullsized":"2024-03-16T19:30:42.625439+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:03:03.291206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":51,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153869+0000","last_change":"2024-03-16T19:30:07.591770+0000","last_active":"2024-03-16T19:30:43.153869+0000","last_peered":"2024-03-16T19:30:43.153869+0000","last_clean":"2024-03-16T19:30:43.153869+0000","last_became_active":"2024-03-16T19:30:07.591482+0000","last_became_peered":"2024-03-16T19:30:07.591482+0000","last_unstale":"2024-03-16T19:30:43.153869+0000","last_undegraded":"2024-03-16T19:30:43.153869+0000","last_fullsized":"2024-03-16T19:30:43.153869+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:23:02.003669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154268+0000","last_change":"2024-03-16T19:29:26.358176+0000","last_active":"2024-03-16T19:30:43.154268+0000","last_peered":"2024-03-16T19:30:43.154268+0000","last_clean":"2024-03-16T19:30:43.154268+0000","last_became_active":"2024-03-16T19:29:26.357875+0000","last_became_peered":"2024-03-16T19:29:26.357875+0000","last_unstale":"2024-03-16T19:30:43.154268+0000","last_undegraded":"2024-03-16T19:30:43.154268+0000","last_fullsized":"2024-03-16T19:30:43.154268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:00.137458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624999+0000","last_change":"2024-03-16T19:29:26.360410+0000","last_active":"2024-03-16T19:30:42.624999+0000","last_peered":"2024-03-16T19:30:42.624999+0000","last_clean":"2024-03-16T19:30:42.624999+0000","last_became_active":"2024-03-16T19:29:26.358526+0000","last_became_peered":"2024-03-16T19:29:26.358526+0000","last_unstale":"2024-03-16T19:30:42.624999+0000","last_undegraded":"2024-03-16T19:30:42.624999+0000","last_fullsized":"2024-03-16T19:30:42.624999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:54.885893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154707+0000","last_change":"2024-03-16T19:29:26.354739+0000","last_active":"2024-03-16T19:30:43.154707+0000","last_peered":"2024-03-16T19:30:43.154707+0000","last_clean":"2024-03-16T19:30:43.154707+0000","last_became_active":"2024-03-16T19:29:26.354448+0000","last_became_peered":"2024-03-16T19:29:26.354448+0000","last_unstale":"2024-03-16T19:30:43.154707+0000","last_undegraded":"2024-03-16T19:30:43.154707+0000","last_fullsized":"2024-03-16T19:30:43.154707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:37.357954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154184+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:30:43.154184+0000","last_peered":"2024-03-16T19:30:43.154184+0000","last_clean":"2024-03-16T19:30:43.154184+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:30:43.154184+0000","last_undegraded":"2024-03-16T19:30:43.154184+0000","last_fullsized":"2024-03-16T19:30:43.154184+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625049+0000","last_change":"2024-03-16T19:29:26.345798+0000","last_active":"2024-03-16T19:30:42.625049+0000","last_peered":"2024-03-16T19:30:42.625049+0000","last_clean":"2024-03-16T19:30:42.625049+0000","last_became_active":"2024-03-16T19:29:26.345397+0000","last_became_peered":"2024-03-16T19:29:26.345397+0000","last_unstale":"2024-03-16T19:30:42.625049+0000","last_undegraded":"2024-03-16T19:30:42.625049+0000","last_fullsized":"2024-03-16T19:30:42.625049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:31:17.956976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154678+0000","last_change":"2024-03-16T19:29:26.351263+0000","last_active":"2024-03-16T19:30:43.154678+0000","last_peered":"2024-03-16T19:30:43.154678+0000","last_clean":"2024-03-16T19:30:43.154678+0000","last_became_active":"2024-03-16T19:29:26.350987+0000","last_became_peered":"2024-03-16T19:29:26.350987+0000","last_unstale":"2024-03-16T19:30:43.154678+0000","last_undegraded":"2024-03-16T19:30:43.154678+0000","last_fullsized":"2024-03-16T19:30:43.154678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:41:17.868783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":70,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154261+0000","last_change":"2024-03-16T19:29:44.412475+0000","last_active":"2024-03-16T19:30:43.154261+0000","last_peered":"2024-03-16T19:30:43.154261+0000","last_clean":"2024-03-16T19:30:43.154261+0000","last_became_active":"2024-03-16T19:29:44.412176+0000","last_became_peered":"2024-03-16T19:29:44.412176+0000","last_unstale":"2024-03-16T19:30:43.154261+0000","last_undegraded":"2024-03-16T19:30:43.154261+0000","last_fullsized":"2024-03-16T19:30:43.154261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:29:34.284744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625413+0000","last_change":"2024-03-16T19:29:26.357621+0000","last_active":"2024-03-16T19:30:42.625413+0000","last_peered":"2024-03-16T19:30:42.625413+0000","last_clean":"2024-03-16T19:30:42.625413+0000","last_became_active":"2024-03-16T19:29:26.356979+0000","last_became_peered":"2024-03-16T19:29:26.356979+0000","last_unstale":"2024-03-16T19:30:42.625413+0000","last_undegraded":"2024-03-16T19:30:42.625413+0000","last_fullsized":"2024-03-16T19:30:42.625413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:45:19.240060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153955+0000","last_change":"2024-03-16T19:29:26.355629+0000","last_active":"2024-03-16T19:30:43.153955+0000","last_peered":"2024-03-16T19:30:43.153955+0000","last_clean":"2024-03-16T19:30:43.153955+0000","last_became_active":"2024-03-16T19:29:26.355341+0000","last_became_peered":"2024-03-16T19:29:26.355341+0000","last_unstale":"2024-03-16T19:30:43.153955+0000","last_undegraded":"2024-03-16T19:30:43.153955+0000","last_fullsized":"2024-03-16T19:30:43.153955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:38.416237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153558+0000","last_change":"2024-03-16T19:29:26.359894+0000","last_active":"2024-03-16T19:30:43.153558+0000","last_peered":"2024-03-16T19:30:43.153558+0000","last_clean":"2024-03-16T19:30:43.153558+0000","last_became_active":"2024-03-16T19:29:26.359656+0000","last_became_peered":"2024-03-16T19:29:26.359656+0000","last_unstale":"2024-03-16T19:30:43.153558+0000","last_undegraded":"2024-03-16T19:30:43.153558+0000","last_fullsized":"2024-03-16T19:30:43.153558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:49.492780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153694+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:30:43.153694+0000","last_peered":"2024-03-16T19:30:43.153694+0000","last_clean":"2024-03-16T19:30:43.153694+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:30:43.153694+0000","last_undegraded":"2024-03-16T19:30:43.153694+0000","last_fullsized":"2024-03-16T19:30:43.153694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154491+0000","last_change":"2024-03-16T19:29:25.347807+0000","last_active":"2024-03-16T19:30:43.154491+0000","last_peered":"2024-03-16T19:30:43.154491+0000","last_clean":"2024-03-16T19:30:43.154491+0000","last_became_active":"2024-03-16T19:29:25.347483+0000","last_became_peered":"2024-03-16T19:29:25.347483+0000","last_unstale":"2024-03-16T19:30:43.154491+0000","last_undegraded":"2024-03-16T19:30:43.154491+0000","last_fullsized":"2024-03-16T19:30:43.154491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:37:18.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153829+0000","last_change":"2024-03-16T19:29:44.412541+0000","last_active":"2024-03-16T19:30:43.153829+0000","last_peered":"2024-03-16T19:30:43.153829+0000","last_clean":"2024-03-16T19:30:43.153829+0000","last_became_active":"2024-03-16T19:29:44.412194+0000","last_became_peered":"2024-03-16T19:29:44.412194+0000","last_unstale":"2024-03-16T19:30:43.153829+0000","last_undegraded":"2024-03-16T19:30:43.153829+0000","last_fullsized":"2024-03-16T19:30:43.153829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:51:50.130592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153796+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:30:43.153796+0000","last_peered":"2024-03-16T19:30:43.153796+0000","last_clean":"2024-03-16T19:30:43.153796+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:30:43.153796+0000","last_undegraded":"2024-03-16T19:30:43.153796+0000","last_fullsized":"2024-03-16T19:30:43.153796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154395+0000","last_change":"2024-03-16T19:29:25.341238+0000","last_active":"2024-03-16T19:30:43.154395+0000","last_peered":"2024-03-16T19:30:43.154395+0000","last_clean":"2024-03-16T19:30:43.154395+0000","last_became_active":"2024-03-16T19:29:25.340980+0000","last_became_peered":"2024-03-16T19:29:25.340980+0000","last_unstale":"2024-03-16T19:30:43.154395+0000","last_undegraded":"2024-03-16T19:30:43.154395+0000","last_fullsized":"2024-03-16T19:30:43.154395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:27.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154304+0000","last_change":"2024-03-16T19:29:26.349258+0000","last_active":"2024-03-16T19:30:43.154304+0000","last_peered":"2024-03-16T19:30:43.154304+0000","last_clean":"2024-03-16T19:30:43.154304+0000","last_became_active":"2024-03-16T19:29:26.349009+0000","last_became_peered":"2024-03-16T19:29:26.349009+0000","last_unstale":"2024-03-16T19:30:43.154304+0000","last_undegraded":"2024-03-16T19:30:43.154304+0000","last_fullsized":"2024-03-16T19:30:43.154304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:15:34.547112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625033+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:30:42.625033+0000","last_peered":"2024-03-16T19:30:42.625033+0000","last_clean":"2024-03-16T19:30:42.625033+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:30:42.625033+0000","last_undegraded":"2024-03-16T19:30:42.625033+0000","last_fullsized":"2024-03-16T19:30:42.625033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624775+0000","last_change":"2024-03-16T19:29:26.345710+0000","last_active":"2024-03-16T19:30:42.624775+0000","last_peered":"2024-03-16T19:30:42.624775+0000","last_clean":"2024-03-16T19:30:42.624775+0000","last_became_active":"2024-03-16T19:29:26.345372+0000","last_became_peered":"2024-03-16T19:29:26.345372+0000","last_unstale":"2024-03-16T19:30:42.624775+0000","last_undegraded":"2024-03-16T19:30:42.624775+0000","last_fullsized":"2024-03-16T19:30:42.624775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:44:56.590094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":79,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153949+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:30:43.153949+0000","last_peered":"2024-03-16T19:30:43.153949+0000","last_clean":"2024-03-16T19:30:43.153949+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:30:43.153949+0000","last_undegraded":"2024-03-16T19:30:43.153949+0000","last_fullsized":"2024-03-16T19:30:43.153949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625329+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:30:42.625329+0000","last_peered":"2024-03-16T19:30:42.625329+0000","last_clean":"2024-03-16T19:30:42.625329+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:30:42.625329+0000","last_undegraded":"2024-03-16T19:30:42.625329+0000","last_fullsized":"2024-03-16T19:30:42.625329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625404+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:30:42.625404+0000","last_peered":"2024-03-16T19:30:42.625404+0000","last_clean":"2024-03-16T19:30:42.625404+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:30:42.625404+0000","last_undegraded":"2024-03-16T19:30:42.625404+0000","last_fullsized":"2024-03-16T19:30:42.625404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625603+0000","last_change":"2024-03-16T19:29:44.413558+0000","last_active":"2024-03-16T19:30:42.625603+0000","last_peered":"2024-03-16T19:30:42.625603+0000","last_clean":"2024-03-16T19:30:42.625603+0000","last_became_active":"2024-03-16T19:29:44.413310+0000","last_became_peered":"2024-03-16T19:29:44.413310+0000","last_unstale":"2024-03-16T19:30:42.625603+0000","last_undegraded":"2024-03-16T19:30:42.625603+0000","last_fullsized":"2024-03-16T19:30:42.625603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:25:53.846575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":81,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625087+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:30:42.625087+0000","last_peered":"2024-03-16T19:30:42.625087+0000","last_clean":"2024-03-16T19:30:42.625087+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:30:42.625087+0000","last_undegraded":"2024-03-16T19:30:42.625087+0000","last_fullsized":"2024-03-16T19:30:42.625087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153983+0000","last_change":"2024-03-16T19:29:26.362327+0000","last_active":"2024-03-16T19:30:43.153983+0000","last_peered":"2024-03-16T19:30:43.153983+0000","last_clean":"2024-03-16T19:30:43.153983+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:30:43.153983+0000","last_undegraded":"2024-03-16T19:30:43.153983+0000","last_fullsized":"2024-03-16T19:30:43.153983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:08.316459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"36'11","reported_seq":90,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153915+0000","last_change":"2024-03-16T19:29:25.342090+0000","last_active":"2024-03-16T19:30:43.153915+0000","last_peered":"2024-03-16T19:30:43.153915+0000","last_clean":"2024-03-16T19:30:43.153915+0000","last_became_active":"2024-03-16T19:29:25.341648+0000","last_became_peered":"2024-03-16T19:29:25.341648+0000","last_unstale":"2024-03-16T19:30:43.153915+0000","last_undegraded":"2024-03-16T19:30:43.153915+0000","last_fullsized":"2024-03-16T19:30:43.153915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:21:13.730280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6662,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626092+0000","last_change":"2024-03-16T19:29:26.355995+0000","last_active":"2024-03-16T19:30:42.626092+0000","last_peered":"2024-03-16T19:30:42.626092+0000","last_clean":"2024-03-16T19:30:42.626092+0000","last_became_active":"2024-03-16T19:29:26.355730+0000","last_became_peered":"2024-03-16T19:29:26.355730+0000","last_unstale":"2024-03-16T19:30:42.626092+0000","last_undegraded":"2024-03-16T19:30:42.626092+0000","last_fullsized":"2024-03-16T19:30:42.626092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:24:37.749226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":86,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153923+0000","last_change":"2024-03-16T19:29:44.412731+0000","last_active":"2024-03-16T19:30:43.153923+0000","last_peered":"2024-03-16T19:30:43.153923+0000","last_clean":"2024-03-16T19:30:43.153923+0000","last_became_active":"2024-03-16T19:29:44.412174+0000","last_became_peered":"2024-03-16T19:29:44.412174+0000","last_unstale":"2024-03-16T19:30:43.153923+0000","last_undegraded":"2024-03-16T19:30:43.153923+0000","last_fullsized":"2024-03-16T19:30:43.153923+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:37:57.962644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625309+0000","last_change":"2024-03-16T19:29:44.412524+0000","last_active":"2024-03-16T19:30:42.625309+0000","last_peered":"2024-03-16T19:30:42.625309+0000","last_clean":"2024-03-16T19:30:42.625309+0000","last_became_active":"2024-03-16T19:29:44.412200+0000","last_became_peered":"2024-03-16T19:29:44.412200+0000","last_unstale":"2024-03-16T19:30:42.625309+0000","last_undegraded":"2024-03-16T19:30:42.625309+0000","last_fullsized":"2024-03-16T19:30:42.625309+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:49:52.636929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625443+0000","last_change":"2024-03-16T19:29:25.340860+0000","last_active":"2024-03-16T19:30:42.625443+0000","last_peered":"2024-03-16T19:30:42.625443+0000","last_clean":"2024-03-16T19:30:42.625443+0000","last_became_active":"2024-03-16T19:29:25.340584+0000","last_became_peered":"2024-03-16T19:29:25.340584+0000","last_unstale":"2024-03-16T19:30:42.625443+0000","last_undegraded":"2024-03-16T19:30:42.625443+0000","last_fullsized":"2024-03-16T19:30:42.625443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:48:40.360129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625621+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:30:42.625621+0000","last_peered":"2024-03-16T19:30:42.625621+0000","last_clean":"2024-03-16T19:30:42.625621+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:30:42.625621+0000","last_undegraded":"2024-03-16T19:30:42.625621+0000","last_fullsized":"2024-03-16T19:30:42.625621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":79,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153693+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:30:43.153693+0000","last_peered":"2024-03-16T19:30:43.153693+0000","last_clean":"2024-03-16T19:30:43.153693+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:30:43.153693+0000","last_undegraded":"2024-03-16T19:30:43.153693+0000","last_fullsized":"2024-03-16T19:30:43.153693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153595+0000","last_change":"2024-03-16T19:29:26.360016+0000","last_active":"2024-03-16T19:30:43.153595+0000","last_peered":"2024-03-16T19:30:43.153595+0000","last_clean":"2024-03-16T19:30:43.153595+0000","last_became_active":"2024-03-16T19:29:26.358826+0000","last_became_peered":"2024-03-16T19:29:26.358826+0000","last_unstale":"2024-03-16T19:30:43.153595+0000","last_undegraded":"2024-03-16T19:30:43.153595+0000","last_fullsized":"2024-03-16T19:30:43.153595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:36.071120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":85,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153826+0000","last_change":"2024-03-16T19:29:25.342186+0000","last_active":"2024-03-16T19:30:43.153826+0000","last_peered":"2024-03-16T19:30:43.153826+0000","last_clean":"2024-03-16T19:30:43.153826+0000","last_became_active":"2024-03-16T19:29:25.341899+0000","last_became_peered":"2024-03-16T19:29:25.341899+0000","last_unstale":"2024-03-16T19:30:43.153826+0000","last_undegraded":"2024-03-16T19:30:43.153826+0000","last_fullsized":"2024-03-16T19:30:43.153826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:04:01.324766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625278+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:30:42.625278+0000","last_peered":"2024-03-16T19:30:42.625278+0000","last_clean":"2024-03-16T19:30:42.625278+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:30:42.625278+0000","last_undegraded":"2024-03-16T19:30:42.625278+0000","last_fullsized":"2024-03-16T19:30:42.625278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":81,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154098+0000","last_change":"2024-03-16T19:29:25.337201+0000","last_active":"2024-03-16T19:30:43.154098+0000","last_peered":"2024-03-16T19:30:43.154098+0000","last_clean":"2024-03-16T19:30:43.154098+0000","last_became_active":"2024-03-16T19:29:25.336909+0000","last_became_peered":"2024-03-16T19:29:25.336909+0000","last_unstale":"2024-03-16T19:30:43.154098+0000","last_undegraded":"2024-03-16T19:30:43.154098+0000","last_fullsized":"2024-03-16T19:30:43.154098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:03.863247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626148+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:30:42.626148+0000","last_peered":"2024-03-16T19:30:42.626148+0000","last_clean":"2024-03-16T19:30:42.626148+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:30:42.626148+0000","last_undegraded":"2024-03-16T19:30:42.626148+0000","last_fullsized":"2024-03-16T19:30:42.626148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":79,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154025+0000","last_change":"2024-03-16T19:29:25.343086+0000","last_active":"2024-03-16T19:30:43.154025+0000","last_peered":"2024-03-16T19:30:43.154025+0000","last_clean":"2024-03-16T19:30:43.154025+0000","last_became_active":"2024-03-16T19:29:25.342825+0000","last_became_peered":"2024-03-16T19:29:25.342825+0000","last_unstale":"2024-03-16T19:30:43.154025+0000","last_undegraded":"2024-03-16T19:30:43.154025+0000","last_fullsized":"2024-03-16T19:30:43.154025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:40.561057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626246+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:30:42.626246+0000","last_peered":"2024-03-16T19:30:42.626246+0000","last_clean":"2024-03-16T19:30:42.626246+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:30:42.626246+0000","last_undegraded":"2024-03-16T19:30:42.626246+0000","last_fullsized":"2024-03-16T19:30:42.626246+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":79,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625140+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:30:42.625140+0000","last_peered":"2024-03-16T19:30:42.625140+0000","last_clean":"2024-03-16T19:30:42.625140+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:30:42.625140+0000","last_undegraded":"2024-03-16T19:30:42.625140+0000","last_fullsized":"2024-03-16T19:30:42.625140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625841+0000","last_change":"2024-03-16T19:29:26.358870+0000","last_active":"2024-03-16T19:30:42.625841+0000","last_peered":"2024-03-16T19:30:42.625841+0000","last_clean":"2024-03-16T19:30:42.625841+0000","last_became_active":"2024-03-16T19:29:26.358563+0000","last_became_peered":"2024-03-16T19:29:26.358563+0000","last_unstale":"2024-03-16T19:30:42.625841+0000","last_undegraded":"2024-03-16T19:30:42.625841+0000","last_fullsized":"2024-03-16T19:30:42.625841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:45.236809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153929+0000","last_change":"2024-03-16T19:29:25.339470+0000","last_active":"2024-03-16T19:30:43.153929+0000","last_peered":"2024-03-16T19:30:43.153929+0000","last_clean":"2024-03-16T19:30:43.153929+0000","last_became_active":"2024-03-16T19:29:25.339219+0000","last_became_peered":"2024-03-16T19:29:25.339219+0000","last_unstale":"2024-03-16T19:30:43.153929+0000","last_undegraded":"2024-03-16T19:30:43.153929+0000","last_fullsized":"2024-03-16T19:30:43.153929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:19.490866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":48,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626062+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:30:42.626062+0000","last_peered":"2024-03-16T19:30:42.626062+0000","last_clean":"2024-03-16T19:30:42.626062+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:30:42.626062+0000","last_undegraded":"2024-03-16T19:30:42.626062+0000","last_fullsized":"2024-03-16T19:30:42.626062+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":236,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626051+0000","last_change":"2024-03-16T19:29:16.379456+0000","last_active":"2024-03-16T19:30:42.626051+0000","last_peered":"2024-03-16T19:30:42.626051+0000","last_clean":"2024-03-16T19:30:42.626051+0000","last_became_active":"2024-03-16T19:29:16.379159+0000","last_became_peered":"2024-03-16T19:29:16.379159+0000","last_unstale":"2024-03-16T19:30:42.626051+0000","last_undegraded":"2024-03-16T19:30:42.626051+0000","last_fullsized":"2024-03-16T19:30:42.626051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:10.706858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625170+0000","last_change":"2024-03-16T19:29:25.335881+0000","last_active":"2024-03-16T19:30:42.625170+0000","last_peered":"2024-03-16T19:30:42.625170+0000","last_clean":"2024-03-16T19:30:42.625170+0000","last_became_active":"2024-03-16T19:29:25.335625+0000","last_became_peered":"2024-03-16T19:29:25.335625+0000","last_unstale":"2024-03-16T19:30:42.625170+0000","last_undegraded":"2024-03-16T19:30:42.625170+0000","last_fullsized":"2024-03-16T19:30:42.625170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:56.095006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154274+0000","last_change":"2024-03-16T19:29:26.363266+0000","last_active":"2024-03-16T19:30:43.154274+0000","last_peered":"2024-03-16T19:30:43.154274+0000","last_clean":"2024-03-16T19:30:43.154274+0000","last_became_active":"2024-03-16T19:29:26.362703+0000","last_became_peered":"2024-03-16T19:29:26.362703+0000","last_unstale":"2024-03-16T19:30:43.154274+0000","last_undegraded":"2024-03-16T19:30:43.154274+0000","last_fullsized":"2024-03-16T19:30:43.154274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:10.755635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625946+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:30:42.625946+0000","last_peered":"2024-03-16T19:30:42.625946+0000","last_clean":"2024-03-16T19:30:42.625946+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:30:42.625946+0000","last_undegraded":"2024-03-16T19:30:42.625946+0000","last_fullsized":"2024-03-16T19:30:42.625946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153701+0000","last_change":"2024-03-16T19:29:44.412505+0000","last_active":"2024-03-16T19:30:43.153701+0000","last_peered":"2024-03-16T19:30:43.153701+0000","last_clean":"2024-03-16T19:30:43.153701+0000","last_became_active":"2024-03-16T19:29:44.412221+0000","last_became_peered":"2024-03-16T19:29:44.412221+0000","last_unstale":"2024-03-16T19:30:43.153701+0000","last_undegraded":"2024-03-16T19:30:43.153701+0000","last_fullsized":"2024-03-16T19:30:43.153701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:03:14.739638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154043+0000","last_change":"2024-03-16T19:29:25.338586+0000","last_active":"2024-03-16T19:30:43.154043+0000","last_peered":"2024-03-16T19:30:43.154043+0000","last_clean":"2024-03-16T19:30:43.154043+0000","last_became_active":"2024-03-16T19:29:25.338268+0000","last_became_peered":"2024-03-16T19:29:25.338268+0000","last_unstale":"2024-03-16T19:30:43.154043+0000","last_undegraded":"2024-03-16T19:30:43.154043+0000","last_fullsized":"2024-03-16T19:30:43.154043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:37.664233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624818+0000","last_change":"2024-03-16T19:29:26.345908+0000","last_active":"2024-03-16T19:30:42.624818+0000","last_peered":"2024-03-16T19:30:42.624818+0000","last_clean":"2024-03-16T19:30:42.624818+0000","last_became_active":"2024-03-16T19:29:26.345572+0000","last_became_peered":"2024-03-16T19:29:26.345572+0000","last_unstale":"2024-03-16T19:30:42.624818+0000","last_undegraded":"2024-03-16T19:30:42.624818+0000","last_fullsized":"2024-03-16T19:30:42.624818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:21:59.558481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625514+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:30:42.625514+0000","last_peered":"2024-03-16T19:30:42.625514+0000","last_clean":"2024-03-16T19:30:42.625514+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:30:42.625514+0000","last_undegraded":"2024-03-16T19:30:42.625514+0000","last_fullsized":"2024-03-16T19:30:42.625514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154015+0000","last_change":"2024-03-16T19:29:26.350917+0000","last_active":"2024-03-16T19:30:43.154015+0000","last_peered":"2024-03-16T19:30:43.154015+0000","last_clean":"2024-03-16T19:30:43.154015+0000","last_became_active":"2024-03-16T19:29:26.350602+0000","last_became_peered":"2024-03-16T19:29:26.350602+0000","last_unstale":"2024-03-16T19:30:43.154015+0000","last_undegraded":"2024-03-16T19:30:43.154015+0000","last_fullsized":"2024-03-16T19:30:43.154015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:42:25.389957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624939+0000","last_change":"2024-03-16T19:29:25.339362+0000","last_active":"2024-03-16T19:30:42.624939+0000","last_peered":"2024-03-16T19:30:42.624939+0000","last_clean":"2024-03-16T19:30:42.624939+0000","last_became_active":"2024-03-16T19:29:25.338882+0000","last_became_peered":"2024-03-16T19:29:25.338882+0000","last_unstale":"2024-03-16T19:30:42.624939+0000","last_undegraded":"2024-03-16T19:30:42.624939+0000","last_fullsized":"2024-03-16T19:30:42.624939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:18.885894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625869+0000","last_change":"2024-03-16T19:29:26.361935+0000","last_active":"2024-03-16T19:30:42.625869+0000","last_peered":"2024-03-16T19:30:42.625869+0000","last_clean":"2024-03-16T19:30:42.625869+0000","last_became_active":"2024-03-16T19:29:26.357430+0000","last_became_peered":"2024-03-16T19:29:26.357430+0000","last_unstale":"2024-03-16T19:30:42.625869+0000","last_undegraded":"2024-03-16T19:30:42.625869+0000","last_fullsized":"2024-03-16T19:30:42.625869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:13.748179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626256+0000","last_change":"2024-03-16T19:29:25.348170+0000","last_active":"2024-03-16T19:30:42.626256+0000","last_peered":"2024-03-16T19:30:42.626256+0000","last_clean":"2024-03-16T19:30:42.626256+0000","last_became_active":"2024-03-16T19:29:25.347676+0000","last_became_peered":"2024-03-16T19:29:25.347676+0000","last_unstale":"2024-03-16T19:30:42.626256+0000","last_undegraded":"2024-03-16T19:30:42.626256+0000","last_fullsized":"2024-03-16T19:30:42.626256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:19.024934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624847+0000","last_change":"2024-03-16T19:29:26.345777+0000","last_active":"2024-03-16T19:30:42.624847+0000","last_peered":"2024-03-16T19:30:42.624847+0000","last_clean":"2024-03-16T19:30:42.624847+0000","last_became_active":"2024-03-16T19:29:26.345466+0000","last_became_peered":"2024-03-16T19:29:26.345466+0000","last_unstale":"2024-03-16T19:30:42.624847+0000","last_undegraded":"2024-03-16T19:30:42.624847+0000","last_fullsized":"2024-03-16T19:30:42.624847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:12:19.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"40'4","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625221+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:30:42.625221+0000","last_peered":"2024-03-16T19:30:42.625221+0000","last_clean":"2024-03-16T19:30:42.625221+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:30:42.625221+0000","last_undegraded":"2024-03-16T19:30:42.625221+0000","last_fullsized":"2024-03-16T19:30:42.625221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154796+0000","last_change":"2024-03-16T19:29:26.355476+0000","last_active":"2024-03-16T19:30:43.154796+0000","last_peered":"2024-03-16T19:30:43.154796+0000","last_clean":"2024-03-16T19:30:43.154796+0000","last_became_active":"2024-03-16T19:29:26.355092+0000","last_became_peered":"2024-03-16T19:29:26.355092+0000","last_unstale":"2024-03-16T19:30:43.154796+0000","last_undegraded":"2024-03-16T19:30:43.154796+0000","last_fullsized":"2024-03-16T19:30:43.154796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:29:21.906631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":81,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154039+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:30:43.154039+0000","last_peered":"2024-03-16T19:30:43.154039+0000","last_clean":"2024-03-16T19:30:43.154039+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:30:43.154039+0000","last_undegraded":"2024-03-16T19:30:43.154039+0000","last_fullsized":"2024-03-16T19:30:43.154039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625098+0000","last_change":"2024-03-16T19:29:44.413626+0000","last_active":"2024-03-16T19:30:42.625098+0000","last_peered":"2024-03-16T19:30:42.625098+0000","last_clean":"2024-03-16T19:30:42.625098+0000","last_became_active":"2024-03-16T19:29:44.413174+0000","last_became_peered":"2024-03-16T19:29:44.413174+0000","last_unstale":"2024-03-16T19:30:42.625098+0000","last_undegraded":"2024-03-16T19:30:42.625098+0000","last_fullsized":"2024-03-16T19:30:42.625098+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:10:39.043849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":80,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624916+0000","last_change":"2024-03-16T19:29:25.347503+0000","last_active":"2024-03-16T19:30:42.624916+0000","last_peered":"2024-03-16T19:30:42.624916+0000","last_clean":"2024-03-16T19:30:42.624916+0000","last_became_active":"2024-03-16T19:29:25.347210+0000","last_became_peered":"2024-03-16T19:29:25.347210+0000","last_unstale":"2024-03-16T19:30:42.624916+0000","last_undegraded":"2024-03-16T19:30:42.624916+0000","last_fullsized":"2024-03-16T19:30:42.624916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:30:17.673387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153787+0000","last_change":"2024-03-16T19:29:26.356241+0000","last_active":"2024-03-16T19:30:43.153787+0000","last_peered":"2024-03-16T19:30:43.153787+0000","last_clean":"2024-03-16T19:30:43.153787+0000","last_became_active":"2024-03-16T19:29:26.355785+0000","last_became_peered":"2024-03-16T19:29:26.355785+0000","last_unstale":"2024-03-16T19:30:43.153787+0000","last_undegraded":"2024-03-16T19:30:43.153787+0000","last_fullsized":"2024-03-16T19:30:43.153787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:45.721098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.624975+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:30:42.624975+0000","last_peered":"2024-03-16T19:30:42.624975+0000","last_clean":"2024-03-16T19:30:42.624975+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:30:42.624975+0000","last_undegraded":"2024-03-16T19:30:42.624975+0000","last_fullsized":"2024-03-16T19:30:42.624975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":51,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153750+0000","last_change":"2024-03-16T19:30:07.477317+0000","last_active":"2024-03-16T19:30:43.153750+0000","last_peered":"2024-03-16T19:30:43.153750+0000","last_clean":"2024-03-16T19:30:43.153750+0000","last_became_active":"2024-03-16T19:30:07.477037+0000","last_became_peered":"2024-03-16T19:30:07.477037+0000","last_unstale":"2024-03-16T19:30:43.153750+0000","last_undegraded":"2024-03-16T19:30:43.153750+0000","last_fullsized":"2024-03-16T19:30:43.153750+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:59:02.992861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625053+0000","last_change":"2024-03-16T19:29:25.341486+0000","last_active":"2024-03-16T19:30:42.625053+0000","last_peered":"2024-03-16T19:30:42.625053+0000","last_clean":"2024-03-16T19:30:42.625053+0000","last_became_active":"2024-03-16T19:29:25.341193+0000","last_became_peered":"2024-03-16T19:29:25.341193+0000","last_unstale":"2024-03-16T19:30:42.625053+0000","last_undegraded":"2024-03-16T19:30:42.625053+0000","last_fullsized":"2024-03-16T19:30:42.625053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:05:22.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154478+0000","last_change":"2024-03-16T19:29:26.349322+0000","last_active":"2024-03-16T19:30:43.154478+0000","last_peered":"2024-03-16T19:30:43.154478+0000","last_clean":"2024-03-16T19:30:43.154478+0000","last_became_active":"2024-03-16T19:29:26.349076+0000","last_became_peered":"2024-03-16T19:29:26.349076+0000","last_unstale":"2024-03-16T19:30:43.154478+0000","last_undegraded":"2024-03-16T19:30:43.154478+0000","last_fullsized":"2024-03-16T19:30:43.154478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:01:36.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625716+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:30:42.625716+0000","last_peered":"2024-03-16T19:30:42.625716+0000","last_clean":"2024-03-16T19:30:42.625716+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:30:42.625716+0000","last_undegraded":"2024-03-16T19:30:42.625716+0000","last_fullsized":"2024-03-16T19:30:42.625716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":77,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.153695+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:30:43.153695+0000","last_peered":"2024-03-16T19:30:43.153695+0000","last_clean":"2024-03-16T19:30:43.153695+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:30:43.153695+0000","last_undegraded":"2024-03-16T19:30:43.153695+0000","last_fullsized":"2024-03-16T19:30:43.153695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625836+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:30:42.625836+0000","last_peered":"2024-03-16T19:30:42.625836+0000","last_clean":"2024-03-16T19:30:42.625836+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:30:42.625836+0000","last_undegraded":"2024-03-16T19:30:42.625836+0000","last_fullsized":"2024-03-16T19:30:42.625836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625943+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:30:42.625943+0000","last_peered":"2024-03-16T19:30:42.625943+0000","last_clean":"2024-03-16T19:30:42.625943+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:30:42.625943+0000","last_undegraded":"2024-03-16T19:30:42.625943+0000","last_fullsized":"2024-03-16T19:30:42.625943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625745+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:30:42.625745+0000","last_peered":"2024-03-16T19:30:42.625745+0000","last_clean":"2024-03-16T19:30:42.625745+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:30:42.625745+0000","last_undegraded":"2024-03-16T19:30:42.625745+0000","last_fullsized":"2024-03-16T19:30:42.625745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626158+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:30:42.626158+0000","last_peered":"2024-03-16T19:30:42.626158+0000","last_clean":"2024-03-16T19:30:42.626158+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:30:42.626158+0000","last_undegraded":"2024-03-16T19:30:42.626158+0000","last_fullsized":"2024-03-16T19:30:42.626158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":52,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.625254+0000","last_change":"2024-03-16T19:30:07.684215+0000","last_active":"2024-03-16T19:30:42.625254+0000","last_peered":"2024-03-16T19:30:42.625254+0000","last_clean":"2024-03-16T19:30:42.625254+0000","last_became_active":"2024-03-16T19:30:07.683959+0000","last_became_peered":"2024-03-16T19:30:07.683959+0000","last_unstale":"2024-03-16T19:30:42.625254+0000","last_undegraded":"2024-03-16T19:30:42.625254+0000","last_fullsized":"2024-03-16T19:30:42.625254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:23:17.567774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":78,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154874+0000","last_change":"2024-03-16T19:29:25.341196+0000","last_active":"2024-03-16T19:30:43.154874+0000","last_peered":"2024-03-16T19:30:43.154874+0000","last_clean":"2024-03-16T19:30:43.154874+0000","last_became_active":"2024-03-16T19:29:25.340895+0000","last_became_peered":"2024-03-16T19:29:25.340895+0000","last_unstale":"2024-03-16T19:30:43.154874+0000","last_undegraded":"2024-03-16T19:30:43.154874+0000","last_fullsized":"2024-03-16T19:30:43.154874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:41:32.511869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":75,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:43.154783+0000","last_change":"2024-03-16T19:29:26.363878+0000","last_active":"2024-03-16T19:30:43.154783+0000","last_peered":"2024-03-16T19:30:43.154783+0000","last_clean":"2024-03-16T19:30:43.154783+0000","last_became_active":"2024-03-16T19:29:26.363637+0000","last_became_peered":"2024-03-16T19:29:26.363637+0000","last_unstale":"2024-03-16T19:30:43.154783+0000","last_undegraded":"2024-03-16T19:30:43.154783+0000","last_fullsized":"2024-03-16T19:30:43.154783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:29.735500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":86,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-16T19:30:42.626277+0000","last_change":"2024-03-16T19:29:44.414207+0000","last_active":"2024-03-16T19:30:42.626277+0000","last_peered":"2024-03-16T19:30:42.626277+0000","last_clean":"2024-03-16T19:30:42.626277+0000","last_became_active":"2024-03-16T19:29:44.413852+0000","last_became_peered":"2024-03-16T19:29:44.413852+0000","last_unstale":"2024-03-16T19:30:42.626277+0000","last_undegraded":"2024-03-16T19:30:42.626277+0000","last_fullsized":"2024-03-16T19:30:42.626277+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:25.285467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7460,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":14920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":1208320,"data_stored":859121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":274833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":274297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":634880,"data_stored":274333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":280869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":268171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":630784,"data_stored":274367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1216512,"data_stored":859177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":6662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":6662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:30:45.406 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-16T19:30:45.406 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:30:45.406 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-16T19:30:45.406 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-16T19:30:45.406 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-16T19:30:45.406 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-03-16T19:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:45.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:45.470 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:45.489+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:45.528 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:45.528 INFO:teuthology.orchestra.run.smithi174.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-16T19:30:45.528 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:45.559 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:45.578+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:45.584 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:45.584 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:45.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:45.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:30:45.660 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:45.680+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:45.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.762+0000 7fad82d2c640 1 Processor -- start 2024-03-16T19:30:45.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.762+0000 7fad82d2c640 1 -- start start 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad82d2c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c154e50 0x7fad7c0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad82d2c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fad7c0b5f10 0x7fad7c0cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad82d2c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7c06b110 con 0x7fad7c0caf40 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad82d2c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad7c15b8d0 con 0x7fad7c0b5f10 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad82d2c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c15ad50 con 0x7fad7c154e50 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad80ac8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c154e50 0x7fad7c0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad7bfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fad7c0b5f10 0x7fad7c0cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad812c9640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fad7c0caf40 0x7fad7c06b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:43220/0 (socket says 172.21.15.172:43220) 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad80ac8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c154e50 0x7fad7c0cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48166/0 (socket says 172.21.15.172:48166) 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad7bfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fad7c0b5f10 0x7fad7c0cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40646/0 (socket says 172.21.15.172:40646) 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad80ac8640 1 -- 172.21.15.172:0/1984929989 learned_addr learned my addr 172.21.15.172:0/1984929989 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad80ac8640 1 -- 172.21.15.172:0/1984929989 >> v2:172.21.15.174:3300/0 conn(0x7fad7c0b5f10 msgr2=0x7fad7c0cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad80ac8640 1 --2- 172.21.15.172:0/1984929989 >> v2:172.21.15.174:3300/0 conn(0x7fad7c0b5f10 0x7fad7c0cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad80ac8640 1 -- 172.21.15.172:0/1984929989 >> v1:172.21.15.174:6789/0 conn(0x7fad7c0caf40 legacy=0x7fad7c06b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad80ac8640 1 -- 172.21.15.172:0/1984929989 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c15b030 con 0x7fad7c154e50 2024-03-16T19:30:45.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad7b7fe640 1 -- 172.21.15.172:0/1984929989 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425722391 0 0) 0x7fad7c06b110 con 0x7fad7c0caf40 2024-03-16T19:30:45.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.763+0000 7fad7bfff640 1 --2- 172.21.15.172:0/1984929989 >> v2:172.21.15.174:3300/0 conn(0x7fad7c0b5f10 0x7fad7c0cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.764+0000 7fad80ac8640 1 --2- 172.21.15.172:0/1984929989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c154e50 0x7fad7c0cbc30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fad70009870 tx=0x7fad7002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cabb4a12710e414a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.764+0000 7fad7b7fe640 1 -- 172.21.15.172:0/1984929989 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7002f840 con 0x7fad7c154e50 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.764+0000 7fad7b7fe640 1 -- 172.21.15.172:0/1984929989 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7002f9a0 con 0x7fad7c154e50 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.764+0000 7fad7b7fe640 1 -- 172.21.15.172:0/1984929989 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7002fd30 con 0x7fad7c154e50 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.765+0000 7fad82d2c640 1 -- 172.21.15.172:0/1984929989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c154e50 msgr2=0x7fad7c0cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.765+0000 7fad82d2c640 1 --2- 172.21.15.172:0/1984929989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c154e50 0x7fad7c0cbc30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fad70009870 tx=0x7fad7002ba20 comp rx=0 tx=0).stop 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.765+0000 7fad82d2c640 1 -- 172.21.15.172:0/1984929989 shutdown_connections 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.765+0000 7fad82d2c640 1 --2- 172.21.15.172:0/1984929989 >> v2:172.21.15.174:3300/0 conn(0x7fad7c0b5f10 0x7fad7c0cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.765+0000 7fad82d2c640 1 --2- 172.21.15.172:0/1984929989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c154e50 0x7fad7c0cbc30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.765+0000 7fad82d2c640 1 -- 172.21.15.172:0/1984929989 >> 172.21.15.172:0/1984929989 conn(0x7fad7c066c20 msgr2=0x7fad7c067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.765+0000 7fad82d2c640 1 -- 172.21.15.172:0/1984929989 shutdown_connections 2024-03-16T19:30:45.747 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.765+0000 7fad82d2c640 1 -- 172.21.15.172:0/1984929989 wait complete. 2024-03-16T19:30:45.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.766+0000 7fad82d2c640 1 Processor -- start 2024-03-16T19:30:45.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.766+0000 7fad82d2c640 1 -- start start 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad82d2c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad7c0b5f10 0x7fad7c159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad82d2c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c0caf40 0x7fad7c155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad82d2c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad7c154e50 0x7fad7c155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad82d2c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1e7650 con 0x7fad7c0b5f10 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad82d2c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1571f0 con 0x7fad7c154e50 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad82d2c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1f1710 con 0x7fad7c0caf40 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad80ac8640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad7c0b5f10 0x7fad7c159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad812c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad7c154e50 0x7fad7c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad7bfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c0caf40 0x7fad7c155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad80ac8640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad7c0b5f10 0x7fad7c159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57538/0 (socket says 172.21.15.172:57538) 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad812c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad7c154e50 0x7fad7c155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40660/0 (socket says 172.21.15.172:40660) 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.767+0000 7fad80ac8640 1 -- 172.21.15.172:0/1513093096 learned_addr learned my addr 172.21.15.172:0/1513093096 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad7bfff640 1 -- 172.21.15.172:0/1513093096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad7c154e50 msgr2=0x7fad7c155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad7bfff640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad7c154e50 0x7fad7c155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad7bfff640 1 -- 172.21.15.172:0/1513093096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad7c0b5f10 msgr2=0x7fad7c159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad7bfff640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad7c0b5f10 0x7fad7c159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad7bfff640 1 -- 172.21.15.172:0/1513093096 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad68009400 con 0x7fad7c0caf40 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad812c9640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad7c154e50 0x7fad7c155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad7bfff640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c0caf40 0x7fad7c155310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fad6800e930 tx=0x7fad6800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad797fa640 1 -- 172.21.15.172:0/1513093096 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6800ca00 con 0x7fad7c0caf40 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad797fa640 1 -- 172.21.15.172:0/1513093096 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6800cb60 con 0x7fad7c0caf40 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad700094d0 con 0x7fad7c0caf40 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad797fa640 1 -- 172.21.15.172:0/1513093096 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad68013470 con 0x7fad7c0caf40 2024-03-16T19:30:45.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.768+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7c15ab20 con 0x7fad7c0caf40 2024-03-16T19:30:45.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.769+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7c0c9ab0 con 0x7fad7c0caf40 2024-03-16T19:30:45.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.770+0000 7fad797fa640 1 -- 172.21.15.172:0/1513093096 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fad680135d0 con 0x7fad7c0caf40 2024-03-16T19:30:45.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.770+0000 7fad797fa640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad6006b3e0 0x7fad6006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:45.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.770+0000 7fad797fa640 1 -- 172.21.15.172:0/1513093096 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fad6808a1f0 con 0x7fad7c0caf40 2024-03-16T19:30:45.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.770+0000 7fad80ac8640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad6006b3e0 0x7fad6006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:45.752 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:45.753 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:45.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.771+0000 7fad80ac8640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad6006b3e0 0x7fad6006d8a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fad7002ba60 tx=0x7fad70008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:45.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.775+0000 7fad797fa640 1 -- 172.21.15.172:0/1513093096 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad68054e80 con 0x7fad7c0caf40 2024-03-16T19:30:45.761 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:45.780+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:45.762 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:45.861 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:45.881+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:45.913 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:45.913 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:45.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:45.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:45.976+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fad7c075a70 con 0x7fad7c0caf40 2024-03-16T19:30:45.962 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:45.981+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:46.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:46.063 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:46.083+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:46.164 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:46.184+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:46.190 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:46.190 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:46.199 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:46.265 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:46.284+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:46.308 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:46.308 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:46.317 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:46.366 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:46.385+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:46.467 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:46.487+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:46.473 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:46.473 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:46.484 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:46.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:46.568 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:46.588+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:46.616 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.635+0000 7fad797fa640 1 -- 172.21.15.172:0/1513093096 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v57) v1 ==== 97+0+0 (secure 0 0 0) 0x7fad68058540 con 0x7fad7c0caf40 2024-03-16T19:30:46.616 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:30:46.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad6006b3e0 msgr2=0x7fad6006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad6006b3e0 0x7fad6006d8a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fad7002ba60 tx=0x7fad70008030 comp rx=0 tx=0).stop 2024-03-16T19:30:46.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c0caf40 msgr2=0x7fad7c155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c0caf40 0x7fad7c155310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fad6800e930 tx=0x7fad6800edf0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 shutdown_connections 2024-03-16T19:30:46.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad7c154e50 0x7fad7c155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad6006b3e0 0x7fad6006d8a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad7c0caf40 0x7fad7c155310 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.619 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 --2- 172.21.15.172:0/1513093096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad7c0b5f10 0x7fad7c159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 >> 172.21.15.172:0/1513093096 conn(0x7fad7c066c20 msgr2=0x7fad7c1514f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:46.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.637+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 shutdown_connections 2024-03-16T19:30:46.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.638+0000 7fad82d2c640 1 -- 172.21.15.172:0/1513093096 wait complete. 2024-03-16T19:30:46.634 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:46.634 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:46.643 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:46.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.685+0000 7f6db2aed640 1 Processor -- start 2024-03-16T19:30:46.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.685+0000 7f6db2aed640 1 -- start start 2024-03-16T19:30:46.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.686+0000 7f6db2aed640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6dac0b3630 0x7f6dac0cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.686+0000 7f6db2aed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 0x7f6dac0b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.686+0000 7f6db2aed640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac0b4270 con 0x7f6dac154ed0 2024-03-16T19:30:46.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.686+0000 7f6db2aed640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dac15b250 con 0x7f6dac0b3630 2024-03-16T19:30:46.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.686+0000 7f6db2aed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac1552b0 con 0x7f6dac0cb5e0 2024-03-16T19:30:46.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.686+0000 7f6db0889640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6dac154ed0 0x7f6dac15ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:45128/0 (socket says 172.21.15.172:45128) 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.686+0000 7f6db0889640 1 -- 172.21.15.172:0/3795300486 learned_addr learned my addr 172.21.15.172:0/3795300486 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.686+0000 7f6db108a640 1 --2- 172.21.15.172:0/3795300486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 0x7f6dac0b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.686+0000 7f6dabfff640 1 --2- 172.21.15.172:0/3795300486 >> v2:172.21.15.174:3300/0 conn(0x7f6dac0b3630 0x7f6dac0cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.687+0000 7f6dab7fe640 1 -- 172.21.15.172:0/3795300486 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596737444 0 0) 0x7f6dac0b4270 con 0x7f6dac154ed0 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.687+0000 7f6dab7fe640 1 -- 172.21.15.172:0/3795300486 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d98003520 con 0x7f6dac154ed0 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.687+0000 7f6db108a640 1 -- 172.21.15.172:0/3795300486 >> v2:172.21.15.174:3300/0 conn(0x7f6dac0b3630 msgr2=0x7f6dac0cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.687+0000 7f6db108a640 1 --2- 172.21.15.172:0/3795300486 >> v2:172.21.15.174:3300/0 conn(0x7f6dac0b3630 0x7f6dac0cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.687+0000 7f6db108a640 1 -- 172.21.15.172:0/3795300486 >> v1:172.21.15.172:6789/0 conn(0x7f6dac154ed0 legacy=0x7f6dac15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.687+0000 7f6db108a640 1 -- 172.21.15.172:0/3795300486 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac15b4b0 con 0x7f6dac0cb5e0 2024-03-16T19:30:46.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.687+0000 7f6dabfff640 1 --2- 172.21.15.172:0/3795300486 >> v2:172.21.15.174:3300/0 conn(0x7f6dac0b3630 0x7f6dac0cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6db108a640 1 --2- 172.21.15.172:0/3795300486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 0x7f6dac0b4700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c0097a0 tx=0x7f6d9c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a0f0c3e5b9e3402 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6dab7fe640 1 -- 172.21.15.172:0/3795300486 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c031830 con 0x7f6dac0cb5e0 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6dab7fe640 1 -- 172.21.15.172:0/3795300486 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c031990 con 0x7f6dac0cb5e0 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6dab7fe640 1 -- 172.21.15.172:0/3795300486 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c031d00 con 0x7f6dac0cb5e0 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6db2aed640 1 -- 172.21.15.172:0/3795300486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 msgr2=0x7f6dac0b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6db2aed640 1 --2- 172.21.15.172:0/3795300486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 0x7f6dac0b4700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c0097a0 tx=0x7f6d9c02b8d0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6db2aed640 1 -- 172.21.15.172:0/3795300486 shutdown_connections 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6db2aed640 1 --2- 172.21.15.172:0/3795300486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 0x7f6dac0b4700 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6db2aed640 1 --2- 172.21.15.172:0/3795300486 >> v2:172.21.15.174:3300/0 conn(0x7f6dac0b3630 0x7f6dac0cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6db2aed640 1 -- 172.21.15.172:0/3795300486 >> 172.21.15.172:0/3795300486 conn(0x7f6dac06bf60 msgr2=0x7f6dac06d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:46.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6db2aed640 1 -- 172.21.15.172:0/3795300486 shutdown_connections 2024-03-16T19:30:46.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.688+0000 7f6db2aed640 1 -- 172.21.15.172:0/3795300486 wait complete. 2024-03-16T19:30:46.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.689+0000 7f6db2aed640 1 Processor -- start 2024-03-16T19:30:46.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.689+0000 7f6db2aed640 1 -- start start 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db2aed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dac0b3630 0x7f6dac1488a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db2aed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 0x7f6dac1f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db2aed640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dac154ed0 0x7f6dac1473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db2aed640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac15c6f0 con 0x7f6dac154ed0 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db2aed640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac14a3f0 con 0x7f6dac0b3630 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db2aed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac15d840 con 0x7f6dac0cb5e0 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db108a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dac154ed0 0x7f6dac1473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db108a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dac154ed0 0x7f6dac1473a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57544/0 (socket says 172.21.15.172:57544) 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db108a640 1 -- 172.21.15.172:0/2661457347 learned_addr learned my addr 172.21.15.172:0/2661457347 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.690+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dac0b3630 0x7f6dac1488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6dabfff640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 0x7f6dac1f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6db108a640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 msgr2=0x7f6dac1f0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6db108a640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 0x7f6dac1f0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6db108a640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dac0b3630 msgr2=0x7f6dac1488a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6db108a640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dac0b3630 0x7f6dac1488a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6db108a640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d94009400 con 0x7f6dac154ed0 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dac0b3630 0x7f6dac1488a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6dabfff640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6dac0cb5e0 0x7f6dac1f0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6db108a640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dac154ed0 0x7f6dac1473a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c02b910 tx=0x7f6d9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c03a630 con 0x7f6dac154ed0 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c03a7b0 con 0x7f6dac154ed0 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c009400 con 0x7f6dac154ed0 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c03ab60 con 0x7f6dac154ed0 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.691+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dac149b50 con 0x7f6dac154ed0 2024-03-16T19:30:46.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.692+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6dac068d20 con 0x7f6dac154ed0 2024-03-16T19:30:46.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.693+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6d9c03acc0 con 0x7f6dac154ed0 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.693+0000 7f6da97fa640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6d8c06b3b0 0x7f6d8c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.693+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f6d9c0abff0 con 0x7f6dac154ed0 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.693+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6d8c06b3b0 0x7f6d8c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.693+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v57) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6d9c0ac440 con 0x7f6dac154ed0 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.694+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6dac1f1020 0x7f6dac149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.694+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dac0681c0 con 0x7f6dac1f1020 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.694+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6d8c06b3b0 0x7f6d8c06d870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005c00 tx=0x7f6da4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.694+0000 7f6dabfff640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6dac1f1020 0x7f6dac149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.694+0000 7f6dabfff640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6dac1f1020 0x7f6dac149e30 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.695+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6dac0681c0 con 0x7f6dac1f1020 2024-03-16T19:30:46.685 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:46.705+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:46.691 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.710+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6dac0b28f0 con 0x7f6dac1f1020 2024-03-16T19:30:46.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.712+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6dac0b28f0 con 0x7f6dac1f1020 2024-03-16T19:30:46.693 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:46.694 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:46.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.712+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6dac1478e0 0x7f6dac150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.712+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dac153340 con 0x7f6dac1478e0 2024-03-16T19:30:46.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.712+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6dac1478e0 0x7f6dac150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.712+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6dac1478e0 0x7f6dac150e20 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.713+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6dac153340 con 0x7f6dac1478e0 2024-03-16T19:30:46.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.730+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6dac153060 con 0x7f6dac1478e0 2024-03-16T19:30:46.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.732+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6dac153060 con 0x7f6dac1478e0 2024-03-16T19:30:46.713 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:46.714 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:46.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.732+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6dac1f3e00 0x7f6dac1f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.732+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dac148f50 con 0x7f6dac1f3e00 2024-03-16T19:30:46.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.732+0000 7f6dabfff640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6dac1f3e00 0x7f6dac1f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.732+0000 7f6dabfff640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6dac1f3e00 0x7f6dac1f4220 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.733+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6dac148f50 con 0x7f6dac1f3e00 2024-03-16T19:30:46.729 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.748+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6dac1f4760 con 0x7f6dac1f3e00 2024-03-16T19:30:46.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.749+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6dac1f4760 con 0x7f6dac1f3e00 2024-03-16T19:30:46.731 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:46.731 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:46.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.749+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6dac061fb0 0x7f6dac0623b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.749+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dac0628f0 con 0x7f6dac061fb0 2024-03-16T19:30:46.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.750+0000 7f6db108a640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6dac061fb0 0x7f6dac0623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.750+0000 7f6db108a640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6dac061fb0 0x7f6dac0623b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.751+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6dac0628f0 con 0x7f6dac061fb0 2024-03-16T19:30:46.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.767+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6dac14f960 con 0x7f6dac061fb0 2024-03-16T19:30:46.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.769+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6dac14f960 con 0x7f6dac061fb0 2024-03-16T19:30:46.751 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:46.751 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:46.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.769+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6dac150040 0x7f6dac02bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.769+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dac150480 con 0x7f6dac150040 2024-03-16T19:30:46.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.770+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6dac150040 0x7f6dac02bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.770+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6dac150040 0x7f6dac02bde0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.771+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6dac150480 con 0x7f6dac150040 2024-03-16T19:30:46.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.786+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6dac150480 con 0x7f6dac150040 2024-03-16T19:30:46.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.788+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6dac150480 con 0x7f6dac150040 2024-03-16T19:30:46.770 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:46.770 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:46.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.788+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6dac02c830 0x7f6dac02ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.788+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dac150620 con 0x7f6dac02c830 2024-03-16T19:30:46.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.789+0000 7f6dabfff640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6dac02c830 0x7f6dac02ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.789+0000 7f6dabfff640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6dac02c830 0x7f6dac02ccb0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.790+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6dac150620 con 0x7f6dac02c830 2024-03-16T19:30:46.772 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:46.792+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:46.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.807+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6dac150620 con 0x7f6dac02c830 2024-03-16T19:30:46.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.809+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6dac150620 con 0x7f6dac02c830 2024-03-16T19:30:46.790 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:46.791 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:46.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.809+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6dac2071c0 0x7f6dac2095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.809+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dac02d2a0 con 0x7f6dac2071c0 2024-03-16T19:30:46.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.809+0000 7f6db108a640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6dac2071c0 0x7f6dac2095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.809+0000 7f6db108a640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6dac2071c0 0x7f6dac2095b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.792 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.810+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6dac02d2a0 con 0x7f6dac2071c0 2024-03-16T19:30:46.792 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:46.792 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:46.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:46.792 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:46.792 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.454380447999995, 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:46.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:46.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.453153025999995, 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:46.798 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:46.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:46.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:46.801 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.452750320000007, 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.805 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.452141569000005, 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:46.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:46.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:46.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:46.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.451697699999997, 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:46.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:46.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.451554924000007, 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:46.814 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:46.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:46.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.817 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.451540382999994, 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:46.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:46.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.448662687999999, 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:46.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:46.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:46.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:46.825 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:46.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.824+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6dac02d270 con 0x7f6dac2071c0 2024-03-16T19:30:46.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.826+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6dac02d270 con 0x7f6dac2071c0 2024-03-16T19:30:46.826 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:46.827 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:46.828 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:46.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.826+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6dac20a000 0x7f6dac20c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:46.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.826+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dac20f490 con 0x7f6dac20a000 2024-03-16T19:30:46.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.827+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6dac20a000 0x7f6dac20c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:46.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.827+0000 7f6db0889640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6dac20a000 0x7f6dac20c3f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:46.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.828+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6dac20f490 con 0x7f6dac20a000 2024-03-16T19:30:46.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.842+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6dac20e230 con 0x7f6dac20a000 2024-03-16T19:30:46.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6da97fa640 1 -- 172.21.15.172:0/2661457347 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6dac20e230 con 0x7f6dac20a000 2024-03-16T19:30:46.829 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:46.830 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:46.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6dac1f1020 msgr2=0x7f6dac149e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6dac1f1020 0x7f6dac149e30 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6dac1478e0 msgr2=0x7f6dac150e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f6dac1478e0 0x7f6dac150e20 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6dac1f3e00 msgr2=0x7f6dac1f4220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7f6dac1f3e00 0x7f6dac1f4220 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6dac061fb0 msgr2=0x7f6dac0623b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f6dac061fb0 0x7f6dac0623b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6dac150040 msgr2=0x7f6dac02bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6dac150040 0x7f6dac02bde0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db108a640 1 -- 172.21.15.172:0/2661457347 reap_dead start 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6dac02c830 msgr2=0x7f6dac02ccb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6dac02c830 0x7f6dac02ccb0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db108a640 1 -- 172.21.15.172:0/2661457347 reap_dead start 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6dac2071c0 msgr2=0x7f6dac2095b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f6dac2071c0 0x7f6dac2095b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6dac20a000 msgr2=0x7f6dac20c3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f6dac20a000 0x7f6dac20c3f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db108a640 1 -- 172.21.15.172:0/2661457347 reap_dead start 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6d8c06b3b0 msgr2=0x7f6d8c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6d8c06b3b0 0x7f6d8c06d870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005c00 tx=0x7f6da4009040 comp rx=0 tx=0).stop 2024-03-16T19:30:46.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dac154ed0 msgr2=0x7f6dac1473a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:46.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dac154ed0 0x7f6dac1473a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c02b910 tx=0x7f6d9c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 shutdown_connections 2024-03-16T19:30:46.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6dac154ed0 0x7f6dac1473a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 --2- 172.21.15.172:0/2661457347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6d8c06b3b0 0x7f6d8c06d870 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:46.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 >> 172.21.15.172:0/2661457347 conn(0x7f6dac06bf60 msgr2=0x7f6dac152530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:46.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 shutdown_connections 2024-03-16T19:30:46.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:46.845+0000 7f6db2aed640 1 -- 172.21.15.172:0/2661457347 wait complete. 2024-03-16T19:30:46.911 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:46.931+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:46.973 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:46.993+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:46.978 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:46.978 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:46.978 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:46.978 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:46.987 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:47.075 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:47.095+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:47.143 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:47.143 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:47.143 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:47.143 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:47.143 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:47.152 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:47.175 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:47.195+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:47.258 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:47.258 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:47.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.920212903999996, 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:30:47.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:30:47.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.920133785999994, 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:30:47.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:30:47.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:30:47.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.919519661999999, 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:47.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:30:47.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.919079593000006, 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.918838003000005, 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:30:47.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:30:47.279 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.279 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.279 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.279 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.279 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.279 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:47.280 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:47.280 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:47.300+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:47.381 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:47.401+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:47.387 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:47.387 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:47.387 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:47.387 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:47.395 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:47.483 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:47.503+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:47.502 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:47.502 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:47.502 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:47.503 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:47.503 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:47.511 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:47.583 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:47.603+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:47.622 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:47.622 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:47.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.284019869000005, 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.283995696000005, 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:30:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.283989868999996, 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.633 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:47.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:30:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:47.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:47.638 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.283985956999999, 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:30:47.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:30:47.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:30:47.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.283979974999994, 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:30:47.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:30:47.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.283976308999996, 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:30:47.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:30:47.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:30:47.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:47.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:47.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.283965648999995, 2024-03-16T19:30:47.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:30:47.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.283961759999997, 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:47.660 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:30:47.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.663 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.280445068999995, 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.667 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.280374633999998, 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:30:47.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:30:47.671 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.279992286999999, 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:30:47.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.278226004000004, 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.276413082000005, 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:47.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.275557332999995, 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:47.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.687 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.269696566999997, 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 67.837145359000004, 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:47.696 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 66.269651652999997, 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:30:47.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:47.701 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 64.762039401999999, 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:30:47.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:30:47.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:30:47.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.706 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.265176177000001, 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:30:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.264640261, 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:47.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:47.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:30:47.714 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.715 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.715 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.715 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:47.715 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:47.715 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:47.716 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:47.717 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:47.736+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:47.817 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:47.837+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:47.863 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:47.863 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:47.863 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:47.863 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:47.872 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:47.917 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:47.937+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:47.979 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:47.979 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:47.987 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:48.029 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:48.049+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:48.118 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:48.138+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:48.137 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:48.137 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 91.926214357000006, 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.138 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 91.549518809999995, 2024-03-16T19:30:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.147 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.773542366000001, 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.773457953000005, 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.153 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:48.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:30:48.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:30:48.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.773450596999993, 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.158 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.580911404999995, 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:30:48.163 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:48.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:30:48.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.168 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.580905598000001, 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.580787465, 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:30:48.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:30:48.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:30:48.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.580692268000007, 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.179 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:30:48.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:30:48.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:30:48.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.580688257000006, 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.184 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:30:48.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:30:48.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.161756382999997, 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.189 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:30:48.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.161707014000001, 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:30:48.194 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.161628182000001, 2024-03-16T19:30:48.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:30:48.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:30:48.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.204 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.161624970999995, 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:30:48.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:30:48.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.209 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.161622018000003, 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:30:48.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.161507044000004, 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:30:48.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.161503691999997, 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.220 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:48.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:30:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:30:48.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.161500575999995, 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:48.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:30:48.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:30:48.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:30:48.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 90.161440107000004, 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:30:48.231 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:30:48.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:48.236 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.552760057, 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:48.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:48.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.242 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:48.242 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:48.242 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:48.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:48.244 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:48.263+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:48.344 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:48.364+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:48.351 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:48.352 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:48.361 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:48.446 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:48.466+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:48.509 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:48.509 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:48.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:48.560 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:48.580+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:48.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:30:48.646 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:48.666+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 92.097096489999998, 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:30:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.306870318999998, 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:30:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.306110083999997, 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:30:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.305296128999998, 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:30:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.304656203999997, 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:30:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.303998106999998, 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:30:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.303465532000004, 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:30:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:30:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.245293629000003, 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:30:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:30:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.244691434999993, 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.244048489999997, 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.243544529000005, 2024-03-16T19:30:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:30:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 90.085398263000002, 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:30:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:30:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 90.084868126999993, 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:30:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:30:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:30:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.332898868000001, 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:30:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:30:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:30:48.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:48.742 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.332874332000003, 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:30:48.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:48.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.332869802000005, 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:30:48.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:30:48.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.332859990000003, 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:48.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:30:48.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:30:48.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:30:48.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.332856136999993, 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:30:48.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:30:48.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.332852821000003, 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.767 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:30:48.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.771 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.331657238000005, 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:48.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:30:48.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:48.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:48.776 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:48.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:48.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.757+0000 7f4ef660f640 1 Processor -- start 2024-03-16T19:30:48.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.757+0000 7f4ef660f640 1 -- start start 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4ef660f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef006baf0 0x7f4ef006b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4ef660f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ef01580b0 0x7f4ef0161fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4ef660f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef00c1eb0 con 0x7f4ef006aed0 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4ef660f640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef00c1150 con 0x7f4ef01580b0 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4ef660f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef006b7f0 con 0x7f4ef006baf0 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4ef4bac640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ef01580b0 0x7f4ef0161fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4eef7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef006baf0 0x7f4ef006b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4ef4bac640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ef01580b0 0x7f4ef0161fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59520/0 (socket says 172.21.15.172:59520) 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4ef4bac640 1 -- 172.21.15.172:0/3343279985 learned_addr learned my addr 172.21.15.172:0/3343279985 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4eeeffd640 1 -- 172.21.15.172:0/3343279985 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609254524 0 0) 0x7f4ef00c1eb0 con 0x7f4ef006aed0 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4eef7fe640 1 -- 172.21.15.172:0/3343279985 >> v2:172.21.15.172:3300/0 conn(0x7f4ef01580b0 msgr2=0x7f4ef0161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:48.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4eef7fe640 1 --2- 172.21.15.172:0/3343279985 >> v2:172.21.15.172:3300/0 conn(0x7f4ef01580b0 0x7f4ef0161fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4eef7fe640 1 -- 172.21.15.172:0/3343279985 >> v1:172.21.15.174:6789/0 conn(0x7f4ef006aed0 legacy=0x7f4ef01579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.758+0000 7f4eef7fe640 1 -- 172.21.15.172:0/3343279985 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef00c1350 con 0x7f4ef006baf0 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.759+0000 7f4eef7fe640 1 --2- 172.21.15.172:0/3343279985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef006baf0 0x7f4ef006b2b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40097a0 tx=0x7f4ee402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=990bfa64de9a78ba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4eeeffd640 1 -- 172.21.15.172:0/3343279985 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4031830 con 0x7f4ef006baf0 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4eeeffd640 1 -- 172.21.15.172:0/3343279985 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee4031990 con 0x7f4ef006baf0 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4eeeffd640 1 -- 172.21.15.172:0/3343279985 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4031d20 con 0x7f4ef006baf0 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4ef660f640 1 -- 172.21.15.172:0/3343279985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef006baf0 msgr2=0x7f4ef006b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4ef660f640 1 --2- 172.21.15.172:0/3343279985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef006baf0 0x7f4ef006b2b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40097a0 tx=0x7f4ee402b8d0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4ef660f640 1 -- 172.21.15.172:0/3343279985 shutdown_connections 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4ef660f640 1 --2- 172.21.15.172:0/3343279985 >> v2:172.21.15.172:3300/0 conn(0x7f4ef01580b0 0x7f4ef0161fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4ef660f640 1 --2- 172.21.15.172:0/3343279985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef006baf0 0x7f4ef006b2b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4ef660f640 1 -- 172.21.15.172:0/3343279985 >> 172.21.15.172:0/3343279985 conn(0x7f4ef00cc030 msgr2=0x7f4ef00cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:48.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4ef660f640 1 -- 172.21.15.172:0/3343279985 shutdown_connections 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.760+0000 7f4ef660f640 1 -- 172.21.15.172:0/3343279985 wait complete. 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.762+0000 7f4ef660f640 1 Processor -- start 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.762+0000 7f4ef660f640 1 -- start start 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.763+0000 7f4ef660f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef006aed0 0x7f4ef0143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef660f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef006baf0 0x7f4ef01e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef660f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef01580b0 0x7f4ef01e3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef660f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef006dc30 con 0x7f4ef006aed0 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef660f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef006d740 con 0x7f4ef006baf0 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef660f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00ca290 con 0x7f4ef01580b0 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4eeffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef006aed0 0x7f4ef0143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4eef7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef006baf0 0x7f4ef01e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4eeffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef006aed0 0x7f4ef0143f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59534/0 (socket says 172.21.15.172:59534) 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4eef7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef006baf0 0x7f4ef01e0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:37194/0 (socket says 172.21.15.172:37194) 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4eeffff640 1 -- 172.21.15.172:0/2548777452 learned_addr learned my addr 172.21.15.172:0/2548777452 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef4bac640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef01580b0 0x7f4ef01e3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef4bac640 1 -- 172.21.15.172:0/2548777452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef006baf0 msgr2=0x7f4ef01e0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef4bac640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef006baf0 0x7f4ef01e0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef4bac640 1 -- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef006aed0 msgr2=0x7f4ef0143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef4bac640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef006aed0 0x7f4ef0143f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef4bac640 1 -- 172.21.15.172:0/2548777452 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee4009400 con 0x7f4ef01580b0 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4eeffff640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef006aed0 0x7f4ef0143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:48.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef4bac640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef01580b0 0x7f4ef01e3ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800eb50 tx=0x7f4ee800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4eecff9640 1 -- 172.21.15.172:0/2548777452 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee800cb20 con 0x7f4ef01580b0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4eecff9640 1 -- 172.21.15.172:0/2548777452 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee800f040 con 0x7f4ef01580b0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4eecff9640 1 -- 172.21.15.172:0/2548777452 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee800cea0 con 0x7f4ef01580b0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.764+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef01e38d0 con 0x7f4ef01580b0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.765+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef01e3310 con 0x7f4ef01580b0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.765+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef00ca440 con 0x7f4ef01580b0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.766+0000 7f4eecff9640 1 -- 172.21.15.172:0/2548777452 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4ee8003990 con 0x7f4ef01580b0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.766+0000 7f4eecff9640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4edc06b3b0 0x7f4edc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.766+0000 7f4eecff9640 1 -- 172.21.15.172:0/2548777452 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f4ee8089e50 con 0x7f4ef01580b0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.766+0000 7f4eeffff640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4edc06b3b0 0x7f4edc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.767+0000 7f4eeffff640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4edc06b3b0 0x7f4edc06d870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0000f60 tx=0x7f4ee0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:48.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.770+0000 7f4eecff9640 1 -- 172.21.15.172:0/2548777452 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ef00ca440 con 0x7f4ef01580b0 2024-03-16T19:30:48.788 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:48.797+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:48.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:48.878 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:48.898+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:48.932 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:48.932 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:48.932 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:48.932 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:48.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:48.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.973+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f4ef00ca440 con 0x7f4ef01580b0 2024-03-16T19:30:48.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.993+0000 7fea23e64640 1 Processor -- start 2024-03-16T19:30:48.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.993+0000 7fea23e64640 1 -- start start 2024-03-16T19:30:48.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.994+0000 7fea23e64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c0b3630 0x7fea1c0cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.994+0000 7fea23e64640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c0b4270 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.994+0000 7fea213ff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c0b3630 0x7fea1c0cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.994+0000 7fea21c00640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fea1c154ed0 0x7fea1c15ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:53844/0 (socket says 172.21.15.172:53844) 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.994+0000 7fea21c00640 1 -- 172.21.15.172:0/2365030306 learned_addr learned my addr 172.21.15.172:0/2365030306 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.994+0000 7fea23e64640 1 -- 172.21.15.172:0/2365030306 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c15b250 con 0x7fea1c154ed0 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.994+0000 7fea23e64640 1 -- 172.21.15.172:0/2365030306 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1552b0 con 0x7fea1c0b3630 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.995+0000 7fea20bfe640 1 -- 172.21.15.172:0/2365030306 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659209097 0 0) 0x7fea1c15b250 con 0x7fea1c154ed0 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.995+0000 7fea20bfe640 1 -- 172.21.15.172:0/2365030306 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea08003520 con 0x7fea1c154ed0 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.995+0000 7fea20bfe640 1 -- 172.21.15.172:0/2365030306 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504481694 0 0) 0x7fea1c0b4270 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.995+0000 7fea20bfe640 1 -- 172.21.15.172:0/2365030306 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea1c15b250 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.995+0000 7fea20bfe640 1 -- 172.21.15.172:0/2365030306 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1899139792 0 0) 0x7fea08003520 con 0x7fea1c154ed0 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.995+0000 7fea213ff640 1 -- 172.21.15.172:0/2365030306 >> v1:172.21.15.174:6789/0 conn(0x7fea1c154ed0 legacy=0x7fea1c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.995+0000 7fea213ff640 1 -- 172.21.15.172:0/2365030306 >> v1:172.21.15.172:6789/0 conn(0x7fea1c0cb5e0 legacy=0x7fea1c0b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.995+0000 7fea213ff640 1 -- 172.21.15.172:0/2365030306 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c15b4b0 con 0x7fea1c0b3630 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.996+0000 7fea213ff640 1 --2- 172.21.15.172:0/2365030306 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c0b3630 0x7fea1c0cb0a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009750 tx=0x7fea0c02b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e92bf918e748e39 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.996+0000 7fea20bfe640 1 -- 172.21.15.172:0/2365030306 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea18001670 con 0x7fea1c0b3630 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.996+0000 7fea20bfe640 1 -- 172.21.15.172:0/2365030306 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c02f050 con 0x7fea1c0b3630 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.996+0000 7fea20bfe640 1 -- 172.21.15.172:0/2365030306 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c031730 con 0x7fea1c0b3630 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.997+0000 7fea23e64640 1 -- 172.21.15.172:0/2365030306 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c0b3630 msgr2=0x7fea1c0cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.997+0000 7fea23e64640 1 --2- 172.21.15.172:0/2365030306 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c0b3630 0x7fea1c0cb0a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009750 tx=0x7fea0c02b7b0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.997+0000 7fea23e64640 1 -- 172.21.15.172:0/2365030306 shutdown_connections 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.997+0000 7fea23e64640 1 --2- 172.21.15.172:0/2365030306 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c0b3630 0x7fea1c0cb0a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.997+0000 7fea23e64640 1 -- 172.21.15.172:0/2365030306 >> 172.21.15.172:0/2365030306 conn(0x7fea1c06bf60 msgr2=0x7fea1c06d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.997+0000 7fea23e64640 1 -- 172.21.15.172:0/2365030306 shutdown_connections 2024-03-16T19:30:48.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.997+0000 7fea23e64640 1 -- 172.21.15.172:0/2365030306 wait complete. 2024-03-16T19:30:48.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.999+0000 7fea23e64640 1 Processor -- start 2024-03-16T19:30:48.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:48.999+0000 7fea23e64640 1 -- start start 2024-03-16T19:30:48.980 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:48.999+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:48.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea23e64640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea1c0b3630 0x7fea1c1f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea23e64640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1c0cb5e0 0x7fea1c157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea23e64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c154ed0 0x7fea1c157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea23e64640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1f3f50 con 0x7fea1c0b3630 2024-03-16T19:30:48.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea21c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea1c0b3630 0x7fea1c1f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea21c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea1c0b3630 0x7fea1c1f4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59544/0 (socket says 172.21.15.172:59544) 2024-03-16T19:30:48.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea21c00640 1 -- 172.21.15.172:0/3935298347 learned_addr learned my addr 172.21.15.172:0/3935298347 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:48.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1564b0 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1f3920 con 0x7fea1c154ed0 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea213ff640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1c0cb5e0 0x7fea1c157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.000+0000 7fea22401640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c154ed0 0x7fea1c157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea213ff640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c154ed0 msgr2=0x7fea1c157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea213ff640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea1c154ed0 0x7fea1c157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea213ff640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea1c0b3630 msgr2=0x7fea1c1f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea213ff640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea1c0b3630 0x7fea1c1f4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea213ff640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0c009400 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea21c00640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fea1c0b3630 0x7fea1c1f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea213ff640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1c0cb5e0 0x7fea1c157880 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea0c033000 tx=0x7fea0c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c03c810 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c03c990 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c03cd40 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1c156700 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.001+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1c156cc0 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.002+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fea1c068d20 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.003+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fea0c03a040 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.003+0000 7fea127fc640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fea0006b3e0 0x7fea0006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.004+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fea0c0aaee0 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.004+0000 7fea21c00640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fea0006b3e0 0x7fea0006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.004+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v57) v1 ==== 54+0+16 (secure 0 0 0) 0x7fea0c075c00 con 0x7fea1c0cb5e0 2024-03-16T19:30:48.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.004+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fea1c157350 0x7fea1c1553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:48.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.004+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea1c0b3c20 con 0x7fea1c157350 2024-03-16T19:30:48.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.004+0000 7fea21c00640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fea0006b3e0 0x7fea0006d8a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fea18000f60 tx=0x7fea18001a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:48.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.004+0000 7fea22401640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fea1c157350 0x7fea1c1553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:48.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.005+0000 7fea22401640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fea1c157350 0x7fea1c1553f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:48.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.005+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fea14001b60 con 0x7fea1c157350 2024-03-16T19:30:49.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.036+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fea1c15ba70 con 0x7fea1c157350 2024-03-16T19:30:49.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.038+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fea14001ff0 con 0x7fea1c157350 2024-03-16T19:30:49.018 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:49.019 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:49.019 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.038+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fea1c155e40 0x7fea1c1f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:49.019 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.038+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea1c156220 con 0x7fea1c155e40 2024-03-16T19:30:49.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.038+0000 7fea21c00640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fea1c155e40 0x7fea1c1f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:49.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.039+0000 7fea21c00640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fea1c155e40 0x7fea1c1f03b0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:49.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.039+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fea1c156220 con 0x7fea1c155e40 2024-03-16T19:30:49.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.057+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fea1c06d6d0 con 0x7fea1c155e40 2024-03-16T19:30:49.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.059+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fea1c06d6d0 con 0x7fea1c155e40 2024-03-16T19:30:49.039 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:49.040 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:49.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.059+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fea1c1f0e00 0x7fea1c061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:49.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.059+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea1c0b2ec0 con 0x7fea1c1f0e00 2024-03-16T19:30:49.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.059+0000 7fea213ff640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fea1c1f0e00 0x7fea1c061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:49.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.060+0000 7fea213ff640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fea1c1f0e00 0x7fea1c061930 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:49.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.060+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.2 v2:172.21.15.172:6823/1599824697 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fea1c0b2ec0 con 0x7fea1c1f0e00 2024-03-16T19:30:49.057 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.076+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fea1c0b2be0 con 0x7fea1c1f0e00 2024-03-16T19:30:49.058 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.078+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.2 v2:172.21.15.172:6823/1599824697 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fea1c0b2be0 con 0x7fea1c1f0e00 2024-03-16T19:30:49.058 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2: {} 2024-03-16T19:30:49.059 INFO:teuthology.orchestra.run.smithi172.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-16T19:30:49.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.078+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fea1c062380 0x7fea1c0c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:49.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.078+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea1c1f3c00 con 0x7fea1c062380 2024-03-16T19:30:49.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.078+0000 7fea22401640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fea1c062380 0x7fea1c0c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:49.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.079+0000 7fea22401640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fea1c062380 0x7fea1c0c7840 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:49.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.079+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fea1c1f3c00 con 0x7fea1c062380 2024-03-16T19:30:49.068 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:49.068 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.095+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fea1c0c7d80 con 0x7fea1c062380 2024-03-16T19:30:49.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:49.077 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.097+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fea1c0c7d80 con 0x7fea1c062380 2024-03-16T19:30:49.078 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.097+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fea1c0c8400 0x7fea1c02bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.097+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea1c2063b0 con 0x7fea1c0c8400 2024-03-16T19:30:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.097+0000 7fea21c00640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fea1c0c8400 0x7fea1c02bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.098+0000 7fea21c00640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fea1c0c8400 0x7fea1c02bde0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.098+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fea1c2063b0 con 0x7fea1c0c8400 2024-03-16T19:30:49.081 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:49.100+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:49.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.114+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fea1c02c320 con 0x7fea1c0c8400 2024-03-16T19:30:49.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.115+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fea1c02c320 con 0x7fea1c0c8400 2024-03-16T19:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:49.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.116+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fea1c02c9a0 0x7fea1c02ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:49.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.116+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea1c02d340 con 0x7fea1c02c9a0 2024-03-16T19:30:49.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.116+0000 7fea213ff640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fea1c02c9a0 0x7fea1c02ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:49.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.116+0000 7fea213ff640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fea1c02c9a0 0x7fea1c02ce00 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:49.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.117+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fea0c082040 con 0x7fea1c02c9a0 2024-03-16T19:30:49.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.132+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fea1c206cb0 con 0x7fea1c02c9a0 2024-03-16T19:30:49.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.133+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fea1c206cb0 con 0x7fea1c02c9a0 2024-03-16T19:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:49.114 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:49.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.133+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fea1c207330 0x7fea1c209720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:49.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.134+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea1c20c4f0 con 0x7fea1c207330 2024-03-16T19:30:49.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.134+0000 7fea22401640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fea1c207330 0x7fea1c209720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:49.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.134+0000 7fea22401640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fea1c207330 0x7fea1c209720 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:49.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.135+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fea1c20c4f0 con 0x7fea1c207330 2024-03-16T19:30:49.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.165+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fea1c209c60 con 0x7fea1c207330 2024-03-16T19:30:49.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.168+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fea1c209c60 con 0x7fea1c207330 2024-03-16T19:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:49.149 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:49.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.168+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fea1c20a2e0 0x7fea1c20c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:49.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.168+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea1c20f670 con 0x7fea1c20a2e0 2024-03-16T19:30:49.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.168+0000 7fea21c00640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fea1c20a2e0 0x7fea1c20c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:49.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.169+0000 7fea21c00640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fea1c20a2e0 0x7fea1c20c6d0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:49.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.169+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fea1c20f670 con 0x7fea1c20a2e0 2024-03-16T19:30:49.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.184+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fea1c062860 con 0x7fea1c20a2e0 2024-03-16T19:30:49.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.186+0000 7fea127fc640 1 -- 172.21.15.172:0/3935298347 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fea1c062860 con 0x7fea1c20a2e0 2024-03-16T19:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fea1c157350 msgr2=0x7fea1c1553f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fea1c157350 0x7fea1c1553f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fea1c155e40 msgr2=0x7fea1c1f03b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fea1c155e40 0x7fea1c1f03b0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fea1c1f0e00 msgr2=0x7fea1c061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6823/1599824697,v1:172.21.15.172:6825/1599824697] conn(0x7fea1c1f0e00 0x7fea1c061930 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fea1c062380 msgr2=0x7fea1c0c7840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fea1c062380 0x7fea1c0c7840 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fea1c0c8400 msgr2=0x7fea1c02bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fea1c0c8400 0x7fea1c02bde0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fea1c02c9a0 msgr2=0x7fea1c02ce00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fea1c02c9a0 0x7fea1c02ce00 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fea1c207330 msgr2=0x7fea1c209720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fea1c207330 0x7fea1c209720 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fea1c20a2e0 msgr2=0x7fea1c20c6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fea1c20a2e0 0x7fea1c20c6d0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea22401640 1 -- 172.21.15.172:0/3935298347 reap_dead start 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea22401640 1 -- 172.21.15.172:0/3935298347 reap_dead start 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea22401640 1 -- 172.21.15.172:0/3935298347 reap_dead start 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea22401640 1 -- 172.21.15.172:0/3935298347 reap_dead start 2024-03-16T19:30:49.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea22401640 1 -- 172.21.15.172:0/3935298347 reap_dead start 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fea0006b3e0 msgr2=0x7fea0006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fea0006b3e0 0x7fea0006d8a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fea18000f60 tx=0x7fea18001a00 comp rx=0 tx=0).stop 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1c0cb5e0 msgr2=0x7fea1c157880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1c0cb5e0 0x7fea1c157880 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea0c033000 tx=0x7fea0c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 shutdown_connections 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fea0006b3e0 0x7fea0006d8a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 --2- 172.21.15.172:0/3935298347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1c0cb5e0 0x7fea1c157880 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 >> 172.21.15.172:0/3935298347 conn(0x7fea1c06bf60 msgr2=0x7fea1c1518b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 shutdown_connections 2024-03-16T19:30:49.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.187+0000 7fea23e64640 1 -- 172.21.15.172:0/3935298347 wait complete. 2024-03-16T19:30:49.182 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:49.201+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:49.207 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:49.207 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:49.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:30:49.282 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:49.302+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:49.367 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:49.367 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:49.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-16T19:30:49.420 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:49.440+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:49.484 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:49.504+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:49.485 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:49.485 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:49.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-16T19:30:49.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:49.605+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:49.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.642+0000 7f4eecff9640 1 -- 172.21.15.172:0/2548777452 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v58) v1 ==== 91+0+0 (secure 0 0 0) 0x7f4ef00ca440 con 0x7f4ef01580b0 2024-03-16T19:30:49.623 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:30:49.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4edc06b3b0 msgr2=0x7f4edc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.625 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4edc06b3b0 0x7f4edc06d870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0000f60 tx=0x7f4ee0009040 comp rx=0 tx=0).stop 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef01580b0 msgr2=0x7f4ef01e3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef01580b0 0x7f4ef01e3ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800eb50 tx=0x7f4ee800c490 comp rx=0 tx=0).stop 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 shutdown_connections 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4edc06b3b0 0x7f4edc06d870 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef01580b0 0x7f4ef01e3ce0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef006baf0 0x7f4ef01e0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 --2- 172.21.15.172:0/2548777452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef006aed0 0x7f4ef0143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 >> 172.21.15.172:0/2548777452 conn(0x7f4ef00cc030 msgr2=0x7f4ef00c9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 shutdown_connections 2024-03-16T19:30:49.626 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:49.645+0000 7f4ef660f640 1 -- 172.21.15.172:0/2548777452 wait complete. 2024-03-16T19:30:49.646 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:49.646 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:49.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:49.686 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:49.706+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:49.787 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:49.807+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:49.810 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:49.811 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:49.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:49.888 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:49.908+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:49.930 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:49.930 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:49.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:49.989 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:50.009+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:50.047 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:50.048 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:50.057 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:50.101 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:50.121+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:50.164 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:50.165 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:50.173 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:50.215 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:50.235+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:50.292 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:30:50.312+0000 7f198b49c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37736) UID: 0 2024-03-16T19:30:50.324 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:50.324 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:50.333 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:50.393 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:50.413+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:50.442 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:50.442 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:50.451 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:50.493 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:50.513+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:50.540 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:30:50.540 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:30:50.540 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:30:50.540 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-16T19:30:50.540 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T19:30:50.569 INFO:tasks.ceph.osd.2:Stopped 2024-03-16T19:30:50.569 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-03-16T19:30:50.569 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi172.front.sepia.ceph.com 2024-03-16T19:30:50.569 DEBUG:teuthology.orchestra.run.smithi172:> type ceph-objectstore-tool 2024-03-16T19:30:50.593 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:50.613+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:50.599 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:50.599 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:50.608 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:50.623 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-16T19:30:50.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-03-16T19:30:50.694 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:50.714+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:50.755 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:50.755 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:50.764 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:50.805 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:50.825+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:50.895 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:50.915+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:50.911 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:50.911 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:50.919 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:51.002 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:51.022+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:51.066 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:51.066 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:51.075 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:51.097 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:51.117+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:51.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:51.198 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:51.218+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:51.221 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:51.221 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:51.230 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:51.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.307+0000 7f04b90ed640 1 Processor -- start 2024-03-16T19:30:51.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.307+0000 7f04b90ed640 1 -- start start 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b90ed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b4154ed0 0x7f04b415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b90ed640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f04b40b4700 0x7f04b40b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b90ed640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40b4270 con 0x7f04b40b3630 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b90ed640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b41552b0 con 0x7f04b40b4700 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b90ed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b415b4b0 con 0x7f04b4154ed0 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b259c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f04b40b3630 0x7f04b40cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:46438/0 (socket says 172.21.15.172:46438) 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b259c640 1 -- 172.21.15.172:0/1027552633 learned_addr learned my addr 172.21.15.172:0/1027552633 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b359e640 1 --2- 172.21.15.172:0/1027552633 >> v2:172.21.15.174:3300/0 conn(0x7f04b40b4700 0x7f04b40b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/1027552633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b4154ed0 0x7f04b415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b1d9b640 1 -- 172.21.15.172:0/1027552633 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789469836 0 0) 0x7f04b40b4270 con 0x7f04b40b3630 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b1d9b640 1 -- 172.21.15.172:0/1027552633 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c003520 con 0x7f04b40b3630 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b2d9d640 1 -- 172.21.15.172:0/1027552633 >> v2:172.21.15.174:3300/0 conn(0x7f04b40b4700 msgr2=0x7f04b40b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:51.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/1027552633 >> v2:172.21.15.174:3300/0 conn(0x7f04b40b4700 0x7f04b40b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b2d9d640 1 -- 172.21.15.172:0/1027552633 >> v1:172.21.15.172:6789/0 conn(0x7f04b40b3630 legacy=0x7f04b40cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:51.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b2d9d640 1 -- 172.21.15.172:0/1027552633 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b415b790 con 0x7f04b4154ed0 2024-03-16T19:30:51.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.308+0000 7f04b359e640 1 --2- 172.21.15.172:0/1027552633 >> v2:172.21.15.174:3300/0 conn(0x7f04b40b4700 0x7f04b40b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:51.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.309+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/1027552633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b4154ed0 0x7f04b415ab40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f04a40097a0 tx=0x7f04a402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ebb2d942a6e8b9a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.309+0000 7f04b1d9b640 1 -- 172.21.15.172:0/1027552633 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4031830 con 0x7f04b4154ed0 2024-03-16T19:30:51.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.309+0000 7f04b1d9b640 1 -- 172.21.15.172:0/1027552633 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a4031990 con 0x7f04b4154ed0 2024-03-16T19:30:51.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.309+0000 7f04b1d9b640 1 -- 172.21.15.172:0/1027552633 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4031d20 con 0x7f04b4154ed0 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.310+0000 7f04b90ed640 1 -- 172.21.15.172:0/1027552633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b4154ed0 msgr2=0x7f04b415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.310+0000 7f04b90ed640 1 --2- 172.21.15.172:0/1027552633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b4154ed0 0x7f04b415ab40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f04a40097a0 tx=0x7f04a402b8d0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.310+0000 7f04b90ed640 1 -- 172.21.15.172:0/1027552633 shutdown_connections 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.310+0000 7f04b90ed640 1 --2- 172.21.15.172:0/1027552633 >> v2:172.21.15.174:3300/0 conn(0x7f04b40b4700 0x7f04b40b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.310+0000 7f04b90ed640 1 --2- 172.21.15.172:0/1027552633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b4154ed0 0x7f04b415ab40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.310+0000 7f04b90ed640 1 -- 172.21.15.172:0/1027552633 >> 172.21.15.172:0/1027552633 conn(0x7f04b406bf60 msgr2=0x7f04b406d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.310+0000 7f04b90ed640 1 -- 172.21.15.172:0/1027552633 shutdown_connections 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.310+0000 7f04b90ed640 1 -- 172.21.15.172:0/1027552633 wait complete. 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.311+0000 7f04b90ed640 1 Processor -- start 2024-03-16T19:30:51.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.311+0000 7f04b90ed640 1 -- start start 2024-03-16T19:30:51.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b90ed640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40b3630 0x7f04b40c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b90ed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b40b4700 0x7f04b4147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b90ed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04b4154ed0 0x7f04b4147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b90ed640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40c8230 con 0x7f04b40b3630 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b90ed640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40c8f50 con 0x7f04b4154ed0 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b90ed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40c8410 con 0x7f04b40b4700 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b2d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40b3630 0x7f04b40c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b2d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40b3630 0x7f04b40c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59546/0 (socket says 172.21.15.172:59546) 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b2d9d640 1 -- 172.21.15.172:0/2187558250 learned_addr learned my addr 172.21.15.172:0/2187558250 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b359e640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04b4154ed0 0x7f04b4147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b259c640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b40b4700 0x7f04b4147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b2d9d640 1 -- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b40b4700 msgr2=0x7f04b4147560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04b40b4700 0x7f04b4147560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b2d9d640 1 -- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04b4154ed0 msgr2=0x7f04b4147aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04b4154ed0 0x7f04b4147aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b2d9d640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac009a70 con 0x7f04b40b3630 2024-03-16T19:30:51.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.312+0000 7f04b359e640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04b4154ed0 0x7f04b4147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:51.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.313+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f04b40b3630 0x7f04b40c6220 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f04a4000c00 tx=0x7f04a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.313+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a403a630 con 0x7f04b40b3630 2024-03-16T19:30:51.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.313+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a403a7d0 con 0x7f04b40b3630 2024-03-16T19:30:51.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.313+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a4009400 con 0x7f04b40b3630 2024-03-16T19:30:51.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.313+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a403ab80 con 0x7f04b40b3630 2024-03-16T19:30:51.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.313+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b40c8990 con 0x7f04b40b3630 2024-03-16T19:30:51.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.314+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f04b4068d20 con 0x7f04b40b3630 2024-03-16T19:30:51.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.315+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f04a403ace0 con 0x7f04b40b3630 2024-03-16T19:30:51.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.315+0000 7f049b7fe640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f04a006b3b0 0x7f04a006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.315+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f04a40ab890 con 0x7f04b40b3630 2024-03-16T19:30:51.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.315+0000 7f04b259c640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f04a006b3b0 0x7f04a006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.315+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v59) v1 ==== 54+0+16 (secure 0 0 0) 0x7f04a40b0240 con 0x7f04b40b3630 2024-03-16T19:30:51.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.316+0000 7f04b90ed640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f04b40ca460 0x7f04b4159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.316+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b40b2e20 con 0x7f04b40ca460 2024-03-16T19:30:51.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.316+0000 7f04b359e640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f04b40ca460 0x7f04b4159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.316+0000 7f04b259c640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f04a006b3b0 0x7f04a006d870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f04b40c3860 tx=0x7f04a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.316+0000 7f04b359e640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f04b40ca460 0x7f04b4159290 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.317+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f04ac00f040 con 0x7f04b40ca460 2024-03-16T19:30:51.314 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:51.334+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:51.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.343+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f04b40681c0 con 0x7f04b40ca460 2024-03-16T19:30:51.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.344+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b40681c0 con 0x7f04b40ca460 2024-03-16T19:30:51.325 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:51.325 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:51.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.345+0000 7f04b90ed640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f04b41f14a0 0x7f04b41f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.345+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b41534c0 con 0x7f04b41f14a0 2024-03-16T19:30:51.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.345+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f04b41f14a0 0x7f04b41f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.345+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f04b41f14a0 0x7f04b41f2070 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.346+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f04b41534c0 con 0x7f04b41f14a0 2024-03-16T19:30:51.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.362+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f04b406d670 con 0x7f04b41f14a0 2024-03-16T19:30:51.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.364+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b406d670 con 0x7f04b41f14a0 2024-03-16T19:30:51.344 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:51.345 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:51.345 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.364+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=60}) v3 -- 0x7f04b40c8b80 con 0x7f04b40b3630 2024-03-16T19:30:51.345 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.364+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04b41f35a0 con 0x7f04b40b3630 2024-03-16T19:30:51.345 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.364+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04a4079d00 con 0x7f04b40b3630 2024-03-16T19:30:51.345 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:30:51.345 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.364+0000 7f04b90ed640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f04b41f40b0 0x7f04b4061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.345 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.364+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b4061e70 con 0x7f04b41f40b0 2024-03-16T19:30:51.346 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.364+0000 7f04b259c640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f04b41f40b0 0x7f04b4061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.346 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.365+0000 7f04b259c640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f04b41f40b0 0x7f04b4061930 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.346 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.366+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f04b4061e70 con 0x7f04b41f40b0 2024-03-16T19:30:51.361 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.381+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f04b40622c0 con 0x7f04b41f40b0 2024-03-16T19:30:51.363 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.383+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b40622c0 con 0x7f04b41f40b0 2024-03-16T19:30:51.364 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:51.364 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:51.364 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.383+0000 7f04b90ed640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f04b40c42f0 0x7f04b40c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.364 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.383+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b4062820 con 0x7f04b40c42f0 2024-03-16T19:30:51.364 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.384+0000 7f04b359e640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f04b40c42f0 0x7f04b40c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.364 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.384+0000 7f04b359e640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f04b40c42f0 0x7f04b40c4730 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.385+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f04b4062820 con 0x7f04b40c42f0 2024-03-16T19:30:51.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.400+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f04b402bde0 con 0x7f04b40c42f0 2024-03-16T19:30:51.381 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:51.381 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:51.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.401+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b402bde0 con 0x7f04b40c42f0 2024-03-16T19:30:51.382 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:51.383 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:51.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.402+0000 7f04b90ed640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f04b402c4c0 0x7f04b402c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.402+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b402ce40 con 0x7f04b402c4c0 2024-03-16T19:30:51.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.402+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f04b402c4c0 0x7f04b402c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.402+0000 7f04b2d9d640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f04b402c4c0 0x7f04b402c900 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.403+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f04b402ce40 con 0x7f04b402c4c0 2024-03-16T19:30:51.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:51.399 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.419+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f04b402ce40 con 0x7f04b402c4c0 2024-03-16T19:30:51.399 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:51.419+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:51.400 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.420+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b402ce40 con 0x7f04b402c4c0 2024-03-16T19:30:51.401 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:51.402 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:51.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.421+0000 7f04b90ed640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f04b4204920 0x7f04b4206d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.421+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b4209ae0 con 0x7f04b4204920 2024-03-16T19:30:51.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.421+0000 7f04b259c640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f04b4204920 0x7f04b4206d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.421+0000 7f04b259c640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f04b4204920 0x7f04b4206d10 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.422+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f04b4209ae0 con 0x7f04b4204920 2024-03-16T19:30:51.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.437+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f04b4207250 con 0x7f04b4204920 2024-03-16T19:30:51.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.438+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b4207250 con 0x7f04b4204920 2024-03-16T19:30:51.419 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:51.419 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:51.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.439+0000 7f04b90ed640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f04b4207930 0x7f04b4209d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.439+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b420caf0 con 0x7f04b4207930 2024-03-16T19:30:51.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.439+0000 7f04b359e640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f04b4207930 0x7f04b4209d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.439+0000 7f04b359e640 1 --2- 172.21.15.172:0/2187558250 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f04b4207930 0x7f04b4209d20 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.440+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f04b420caf0 con 0x7f04b4207930 2024-03-16T19:30:51.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.455+0000 7f04b90ed640 1 -- 172.21.15.172:0/2187558250 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f04b420a260 con 0x7f04b4207930 2024-03-16T19:30:51.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.457+0000 7f049b7fe640 1 -- 172.21.15.172:0/2187558250 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b420a260 con 0x7f04b4207930 2024-03-16T19:30:51.437 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:51.437 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:30:51.441 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:30:51.500 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:51.520+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:51.505 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:51.505 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:51.505 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:51.505 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:51.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:51.601 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:51.621+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:51.625 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:51.625 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:51.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:51.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:30:51.702 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:51.722+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:51.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.766+0000 7f7b648a4640 1 Processor -- start 2024-03-16T19:30:51.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.766+0000 7f7b648a4640 1 -- start start 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b648a4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7b60154e50 0x7f7b600cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b648a4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b600b5f10 0x7f7b600cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b648a4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6006b110 con 0x7f7b600caf40 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b648a4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6015b8d0 con 0x7f7b60154e50 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b648a4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6015ad50 con 0x7f7b600b5f10 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b5dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b600b5f10 0x7f7b600cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b5e59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7b60154e50 0x7f7b600cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b5dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b600b5f10 0x7f7b600cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:57868/0 (socket says 172.21.15.172:57868) 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b5e59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7b60154e50 0x7f7b600cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:37230/0 (socket says 172.21.15.172:37230) 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.767+0000 7f7b5dd9b640 1 -- 172.21.15.172:0/291857895 learned_addr learned my addr 172.21.15.172:0/291857895 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:51.748 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.768+0000 7f7b5e59c640 1 -- 172.21.15.172:0/291857895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b600b5f10 msgr2=0x7f7b600cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:51.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.768+0000 7f7b5e59c640 1 --2- 172.21.15.172:0/291857895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b600b5f10 0x7f7b600cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.768+0000 7f7b5e59c640 1 -- 172.21.15.172:0/291857895 >> v1:172.21.15.174:6789/0 conn(0x7f7b600caf40 legacy=0x7f7b6006b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:51.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.768+0000 7f7b5e59c640 1 -- 172.21.15.172:0/291857895 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6015b030 con 0x7f7b60154e50 2024-03-16T19:30:51.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.769+0000 7f7b5e59c640 1 --2- 172.21.15.172:0/291857895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b60154e50 0x7f7b600cbc30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7b500097a0 tx=0x7f7b5002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb9b098a90b56f29 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.769+0000 7f7b5d59a640 1 -- 172.21.15.172:0/291857895 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50030610 con 0x7f7b60154e50 2024-03-16T19:30:51.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.769+0000 7f7b5d59a640 1 -- 172.21.15.172:0/291857895 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b50030770 con 0x7f7b60154e50 2024-03-16T19:30:51.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.769+0000 7f7b5d59a640 1 -- 172.21.15.172:0/291857895 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50030b00 con 0x7f7b60154e50 2024-03-16T19:30:51.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.770+0000 7f7b648a4640 1 -- 172.21.15.172:0/291857895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b60154e50 msgr2=0x7f7b600cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:51.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.770+0000 7f7b648a4640 1 --2- 172.21.15.172:0/291857895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b60154e50 0x7f7b600cbc30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7b500097a0 tx=0x7f7b5002b960 comp rx=0 tx=0).stop 2024-03-16T19:30:51.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.770+0000 7f7b648a4640 1 -- 172.21.15.172:0/291857895 shutdown_connections 2024-03-16T19:30:51.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.770+0000 7f7b648a4640 1 --2- 172.21.15.172:0/291857895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b600b5f10 0x7f7b600cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.770+0000 7f7b648a4640 1 --2- 172.21.15.172:0/291857895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b60154e50 0x7f7b600cbc30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.750 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.770+0000 7f7b648a4640 1 -- 172.21.15.172:0/291857895 >> 172.21.15.172:0/291857895 conn(0x7f7b60066c20 msgr2=0x7f7b60067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:51.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.770+0000 7f7b648a4640 1 -- 172.21.15.172:0/291857895 shutdown_connections 2024-03-16T19:30:51.751 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.770+0000 7f7b648a4640 1 -- 172.21.15.172:0/291857895 wait complete. 2024-03-16T19:30:51.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.772+0000 7f7b648a4640 1 Processor -- start 2024-03-16T19:30:51.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.772+0000 7f7b648a4640 1 -- start start 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b648a4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b600b5f10 0x7f7b60143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b648a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b600caf40 0x7f7b601f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b648a4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b60154e50 0x7f7b60148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b648a4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b60145380 con 0x7f7b600b5f10 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b648a4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b60144440 con 0x7f7b600caf40 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b648a4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b60158b40 con 0x7f7b60154e50 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b5e59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b600b5f10 0x7f7b60143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b5e59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b600b5f10 0x7f7b60143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59558/0 (socket says 172.21.15.172:59558) 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b5e59c640 1 -- 172.21.15.172:0/543328928 learned_addr learned my addr 172.21.15.172:0/543328928 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b5ed9d640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b60154e50 0x7f7b60148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.773+0000 7f7b5dd9b640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b600caf40 0x7f7b601f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b5e59c640 1 -- 172.21.15.172:0/543328928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b60154e50 msgr2=0x7f7b60148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b5e59c640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b60154e50 0x7f7b60148110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b5e59c640 1 -- 172.21.15.172:0/543328928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b600caf40 msgr2=0x7f7b601f48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:51.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b5e59c640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b600caf40 0x7f7b601f48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:51.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b5e59c640 1 -- 172.21.15.172:0/543328928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b50009400 con 0x7f7b600b5f10 2024-03-16T19:30:51.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b5ed9d640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b60154e50 0x7f7b60148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:51.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b5e59c640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b600b5f10 0x7f7b60143f00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7b5002bf70 tx=0x7f7b500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b4effd640 1 -- 172.21.15.172:0/543328928 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5003e070 con 0x7f7b600b5f10 2024-03-16T19:30:51.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b4effd640 1 -- 172.21.15.172:0/543328928 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b50030e10 con 0x7f7b600b5f10 2024-03-16T19:30:51.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b4effd640 1 -- 172.21.15.172:0/543328928 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50039c50 con 0x7f7b600b5f10 2024-03-16T19:30:51.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6015ab20 con 0x7f7b600b5f10 2024-03-16T19:30:51.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.774+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b601f4e00 con 0x7f7b600b5f10 2024-03-16T19:30:51.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.775+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b600c9ab0 con 0x7f7b600b5f10 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.181913238999996, 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:30:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.391687067999996, 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:30:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:30:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.390926832999995, 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:30:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.390112877999996, 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:30:51.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:30:51.772 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.389472952999995, 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.388814855999996, 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:30:51.776 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.388282281000002, 2024-03-16T19:30:51.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.776+0000 7f7b4effd640 1 -- 172.21.15.172:0/543328928 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7b5004b400 con 0x7f7b600b5f10 2024-03-16T19:30:51.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:30:51.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:30:51.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.330110378000001, 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.785 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:30:51.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:30:51.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:30:51.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.329508184000005, 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.789 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:30:51.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.328865238999995, 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:30:51.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.328361278000003, 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:51.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:30:51.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:30:51.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:30:51.800 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 93.170215012, 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:30:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:30:51.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 93.169684876000005, 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.417715616999999, 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.417691081000001, 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:30:51.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:30:51.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:51.818 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.417686551000003, 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:30:51.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:30:51.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.417676739000001, 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.823 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:51.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:30:51.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.417672886000005, 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:51.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:30:51.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:30:51.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:30:51.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.832 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.417669570000001, 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.416473987000003, 2024-03-16T19:30:51.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:30:51.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:30:51.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:51.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:30:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:30:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:51.840 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:51.840 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:51.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.776+0000 7f7b4effd640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b4006b3b0 0x7f7b4006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:51.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.776+0000 7f7b4effd640 1 -- 172.21.15.172:0/543328928 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f7b500abcc0 con 0x7f7b600b5f10 2024-03-16T19:30:51.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.776+0000 7f7b5dd9b640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b4006b3b0 0x7f7b4006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:51.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.777+0000 7f7b5dd9b640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b4006b3b0 0x7f7b4006d870 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7b6014f4a0 tx=0x7f7b54009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:51.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:51.785+0000 7f7b4effd640 1 -- 172.21.15.172:0/543328928 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b500769e0 con 0x7f7b600b5f10 2024-03-16T19:30:51.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:51.842 INFO:teuthology.orchestra.run.smithi172.stdout:3.9 2024-03-16T19:30:51.842 INFO:teuthology.orchestra.run.smithi172.stdout:3.8 2024-03-16T19:30:51.842 INFO:teuthology.orchestra.run.smithi172.stdout:3.3d 2024-03-16T19:30:51.842 INFO:teuthology.orchestra.run.smithi172.stdout:3.3b 2024-03-16T19:30:51.842 INFO:teuthology.orchestra.run.smithi172.stdout:3.2e 2024-03-16T19:30:51.842 INFO:teuthology.orchestra.run.smithi172.stdout:3.2d 2024-03-16T19:30:51.842 INFO:teuthology.orchestra.run.smithi172.stdout:3.2a 2024-03-16T19:30:51.843 INFO:teuthology.orchestra.run.smithi172.stdout:3.21 2024-03-16T19:30:51.843 INFO:teuthology.orchestra.run.smithi172.stdout:2.e 2024-03-16T19:30:51.843 INFO:teuthology.orchestra.run.smithi172.stdout:3.4 2024-03-16T19:30:51.843 INFO:teuthology.orchestra.run.smithi172.stdout:2.c 2024-03-16T19:30:51.843 INFO:teuthology.orchestra.run.smithi172.stdout:2.3 2024-03-16T19:30:51.843 INFO:teuthology.orchestra.run.smithi172.stdout:3.0 2024-03-16T19:30:51.843 INFO:teuthology.orchestra.run.smithi172.stdout:2.1 2024-03-16T19:30:51.843 INFO:teuthology.orchestra.run.smithi172.stdout:2.10 2024-03-16T19:30:51.883 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:51.903+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:51.944 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:51.964+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:51.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.012+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f7b601f4e00 con 0x7f7b600b5f10 2024-03-16T19:30:52.000 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:52.000 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:52.000 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:52.000 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:52.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:52.045 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:52.065+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:52.146 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:52.166+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:52.172 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:52.172 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:52.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:52.247 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:52.267+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.993886189999998, 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:30:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:30:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:30:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.993869075999996, 2024-03-16T19:30:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:52.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:30:52.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.993861796000004, 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:30:52.342 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:30:52.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:30:52.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:30:52.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:30:52.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.993847928999998, 2024-03-16T19:30:52.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:52.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:30:52.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.993175093999994, 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:30:52.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:30:52.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.990876843999999, 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.990831743000001, 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:30:52.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:30:52.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:52.366 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.990803688, 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.990782273999997, 2024-03-16T19:30:52.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:30:52.374 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.990759922999999, 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:30:52.378 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.990739524000006, 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:30:52.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.990633532000004, 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:30:52.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:52.386 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.988841820999994, 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.988794319999997, 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:52.390 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:30:52.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:30:52.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.988555069, 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:30:52.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 75.980056923999996, 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:30:52.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.402 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 75.978717047999993, 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:30:52.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:30:52.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.406 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.975726066, 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:30:52.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:30:52.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.411 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.975095264, 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:30:52.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:30:52.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:47 2.6 e40/18 ondisk, result = 0)", 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.973612854999999, 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017677, 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377611+0000", 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9599999999999998e-07 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377618+0000", 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9990000000000002e-06 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377626+0000", 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4249999999999998e-06 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377632+0000", 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377707+0000", 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4506999999999997e-05 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377714+0000", 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5819999999999998e-06 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377787+0000", 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3232000000000006e-05 2024-03-16T19:30:52.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.421 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:52.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 3.4 --file /var/log/ceph/exp.3.4.2 2024-03-16T19:30:52.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:52.424 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:52.442+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:52.523 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:52.543+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:52.583 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:52.611 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:52.612 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:52.612 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:52.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:52.623 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:52.644+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:52.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.658+0000 7f7b4effd640 1 -- 172.21.15.172:0/543328928 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v61) v1 ==== 101+0+0 (secure 0 0 0) 0x7f7b601f4e00 con 0x7f7b600b5f10 2024-03-16T19:30:52.638 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b4006b3b0 msgr2=0x7f7b4006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b4006b3b0 0x7f7b4006d870 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7b6014f4a0 tx=0x7f7b54009240 comp rx=0 tx=0).stop 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b600b5f10 msgr2=0x7f7b60143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b600b5f10 0x7f7b60143f00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7b5002bf70 tx=0x7f7b500046c0 comp rx=0 tx=0).stop 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 shutdown_connections 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b4006b3b0 0x7f7b4006d870 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b60154e50 0x7f7b60148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b600caf40 0x7f7b601f48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 --2- 172.21.15.172:0/543328928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b600b5f10 0x7f7b60143f00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.661+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 >> 172.21.15.172:0/543328928 conn(0x7f7b60066c20 msgr2=0x7f7b600cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:52.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.662+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 shutdown_connections 2024-03-16T19:30:52.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:52.662+0000 7f7b648a4640 1 -- 172.21.15.172:0/543328928 wait complete. 2024-03-16T19:30:52.724 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:52.744+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:52.769 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:52.769 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:52.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:52.825 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:52.845+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.57908956, 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.917 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:30:52.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.579075087000007, 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:52.924 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.578570597999999, 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.578555202000004, 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:30:52.928 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:30:52.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:30:52.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.578096518999999, 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.577550349000006, 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.576993258000002, 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:30:52.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.943 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.575791510000002, 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.575212801000006, 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.575163274000005, 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.574202631000006, 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.570488432999994, 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:30:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:30:52.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:30:52.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 73.130087520999993, 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:30:52.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.964 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.564534518000002, 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:30:52.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.968 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.056935792999994, 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:30:52.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:52.973 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:52.973 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:52.974 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:52.974 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:52.993+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:53.074 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:53.094+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:53.083 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:53.084 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:53.084 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:53.084 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:53.092 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:53.177 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:53.197+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:53.241 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:53.241 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:53.241 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:53.241 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:53.242 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:53.250 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:53.276 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:53.296+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:53.377 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:53.397+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:53.401 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:53.401 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:53.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:53.401 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:53.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.401 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.063500907000005, 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:53.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:53.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:53.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.407 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.062273485000006, 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:53.408 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:53.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:53.411 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.061870779000003, 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:53.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:53.413 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:53.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:53.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.416 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.061262028000002, 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:53.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:53.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:53.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:53.421 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.060818158999993, 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:53.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:53.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:53.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.060675383000003, 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:53.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:53.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:53.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.060660842000004, 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:53.431 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:53.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:53.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.057783146999995, 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:53.436 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:53.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.440 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:53.441 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.441 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:53.441 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:53.441 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:53.442 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:53.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:53.483 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:53.503+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:53.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.564+0000 7fa35773a640 1 Processor -- start 2024-03-16T19:30:53.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.564+0000 7fa35773a640 1 -- start start 2024-03-16T19:30:53.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa35773a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa3500b3630 0x7fa3500cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa35773a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3500cb5e0 0x7fa3500b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa35773a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3500b4270 con 0x7fa350154ed0 2024-03-16T19:30:53.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa35773a640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35015b250 con 0x7fa3500b3630 2024-03-16T19:30:53.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa35773a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3501552b0 con 0x7fa3500cb5e0 2024-03-16T19:30:53.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa354cd5640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa3500b3630 0x7fa3500cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa354cd5640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa3500b3630 0x7fa3500cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59572/0 (socket says 172.21.15.172:59572) 2024-03-16T19:30:53.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa3554d6640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa350154ed0 0x7fa35015ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:46452/0 (socket says 172.21.15.172:46452) 2024-03-16T19:30:53.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa354cd5640 1 -- 172.21.15.172:0/1200558255 learned_addr learned my addr 172.21.15.172:0/1200558255 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:53.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa355cd7640 1 --2- 172.21.15.172:0/1200558255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3500cb5e0 0x7fa3500b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa354cd5640 1 -- 172.21.15.172:0/1200558255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3500cb5e0 msgr2=0x7fa3500b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:53.548 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa354cd5640 1 --2- 172.21.15.172:0/1200558255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3500cb5e0 0x7fa3500b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:53.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa354cd5640 1 -- 172.21.15.172:0/1200558255 >> v1:172.21.15.172:6789/0 conn(0x7fa350154ed0 legacy=0x7fa35015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:53.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.565+0000 7fa354cd5640 1 -- 172.21.15.172:0/1200558255 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35015b4b0 con 0x7fa3500b3630 2024-03-16T19:30:53.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.566+0000 7fa354cd5640 1 --2- 172.21.15.172:0/1200558255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3500b3630 0x7fa3500cb0a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa340009870 tx=0x7fa34002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa916765b60d768 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.566+0000 7fa347fff640 1 -- 172.21.15.172:0/1200558255 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340030840 con 0x7fa3500b3630 2024-03-16T19:30:53.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.566+0000 7fa347fff640 1 -- 172.21.15.172:0/1200558255 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3400309a0 con 0x7fa3500b3630 2024-03-16T19:30:53.549 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.566+0000 7fa347fff640 1 -- 172.21.15.172:0/1200558255 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340030d30 con 0x7fa3500b3630 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.567+0000 7fa35773a640 1 -- 172.21.15.172:0/1200558255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3500b3630 msgr2=0x7fa3500cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.567+0000 7fa35773a640 1 --2- 172.21.15.172:0/1200558255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3500b3630 0x7fa3500cb0a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa340009870 tx=0x7fa34002b980 comp rx=0 tx=0).stop 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.567+0000 7fa35773a640 1 -- 172.21.15.172:0/1200558255 shutdown_connections 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.567+0000 7fa35773a640 1 --2- 172.21.15.172:0/1200558255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3500cb5e0 0x7fa3500b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.567+0000 7fa35773a640 1 --2- 172.21.15.172:0/1200558255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3500b3630 0x7fa3500cb0a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.567+0000 7fa35773a640 1 -- 172.21.15.172:0/1200558255 >> 172.21.15.172:0/1200558255 conn(0x7fa35006bf60 msgr2=0x7fa35006d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.567+0000 7fa35773a640 1 -- 172.21.15.172:0/1200558255 shutdown_connections 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.567+0000 7fa35773a640 1 -- 172.21.15.172:0/1200558255 wait complete. 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.569+0000 7fa35773a640 1 Processor -- start 2024-03-16T19:30:53.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.569+0000 7fa35773a640 1 -- start start 2024-03-16T19:30:53.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa35773a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3500b3630 0x7fa3500c4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa35773a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3500cb5e0 0x7fa3500d0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa35773a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa350154ed0 0x7fa3500d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa35773a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35015bde0 con 0x7fa350154ed0 2024-03-16T19:30:53.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa35773a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500d10d0 con 0x7fa3500b3630 2024-03-16T19:30:53.557 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa35773a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500cfa20 con 0x7fa3500cb5e0 2024-03-16T19:30:53.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa355cd7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa350154ed0 0x7fa3500d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa355cd7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa350154ed0 0x7fa3500d0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59574/0 (socket says 172.21.15.172:59574) 2024-03-16T19:30:53.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa355cd7640 1 -- 172.21.15.172:0/3375782496 learned_addr learned my addr 172.21.15.172:0/3375782496 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:53.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa354cd5640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3500cb5e0 0x7fa3500d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa3554d6640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3500b3630 0x7fa3500c4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa354cd5640 1 -- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3500b3630 msgr2=0x7fa3500c4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:53.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa354cd5640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3500b3630 0x7fa3500c4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:53.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa354cd5640 1 -- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa350154ed0 msgr2=0x7fa3500d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:53.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa354cd5640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa350154ed0 0x7fa3500d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:53.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.570+0000 7fa354cd5640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400094d0 con 0x7fa3500cb5e0 2024-03-16T19:30:53.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.571+0000 7fa354cd5640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3500cb5e0 0x7fa3500d0650 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa35008de10 tx=0x7fa340004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.571+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340039a10 con 0x7fa3500cb5e0 2024-03-16T19:30:53.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.571+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa340039b70 con 0x7fa3500cb5e0 2024-03-16T19:30:53.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.571+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35015c480 con 0x7fa3500cb5e0 2024-03-16T19:30:53.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.571+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340043410 con 0x7fa3500cb5e0 2024-03-16T19:30:53.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.571+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3500cec60 con 0x7fa3500cb5e0 2024-03-16T19:30:53.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.572+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa3500c3b90 con 0x7fa3500cb5e0 2024-03-16T19:30:53.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.573+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa34004c3b0 con 0x7fa3500cb5e0 2024-03-16T19:30:53.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.573+0000 7fa345ffb640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa33406b3b0 0x7fa33406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.573+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fa3400abf00 con 0x7fa3500cb5e0 2024-03-16T19:30:53.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.573+0000 7fa3554d6640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa33406b3b0 0x7fa33406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.573+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v61) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa3400ae540 con 0x7fa3500cb5e0 2024-03-16T19:30:53.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.574+0000 7fa35773a640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fa3500c4f80 0x7fa3500cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.574+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa350153a20 con 0x7fa3500c4f80 2024-03-16T19:30:53.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.574+0000 7fa355cd7640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fa3500c4f80 0x7fa3500cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.574+0000 7fa3554d6640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa33406b3b0 0x7fa33406d870 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa34c007fd0 tx=0x7fa34c007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.574+0000 7fa355cd7640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fa3500c4f80 0x7fa3500cedf0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.575+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa350153a20 con 0x7fa3500c4f80 2024-03-16T19:30:53.563 INFO:teuthology.orchestra.run.smithi172.stderr:Exporting 3.4 info 3.4( empty local-lis/les=19/20 n=0 ec=19/19 lis/c=19/19 les/c/f=20/20/0 sis=19) 2024-03-16T19:30:53.564 INFO:teuthology.orchestra.run.smithi172.stderr:Export successful 2024-03-16T19:30:53.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.594+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3500b3f00 con 0x7fa3500c4f80 2024-03-16T19:30:53.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.596+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa34800f040 con 0x7fa3500c4f80 2024-03-16T19:30:53.577 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:53.577 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:53.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.596+0000 7fa35773a640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fa350147330 0x7fa350147710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.596+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa35015ba70 con 0x7fa350147330 2024-03-16T19:30:53.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.596+0000 7fa3554d6640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fa350147330 0x7fa350147710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.596+0000 7fa3554d6640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fa350147330 0x7fa350147710 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.597+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa34c004ac0 con 0x7fa350147330 2024-03-16T19:30:53.579 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:53.599+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:53.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.613+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa350068d20 con 0x7fa350147330 2024-03-16T19:30:53.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.614+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa350068d20 con 0x7fa350147330 2024-03-16T19:30:53.595 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:53.597 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:53.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.614+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=62}) v3 -- 0x7fa3500b3310 con 0x7fa3500cb5e0 2024-03-16T19:30:53.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.614+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa3500681c0 con 0x7fa3500cb5e0 2024-03-16T19:30:53.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.615+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa340076d40 con 0x7fa3500cb5e0 2024-03-16T19:30:53.597 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:30:53.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.615+0000 7fa35773a640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fa350061e40 0x7fa350062250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.615+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3500cbbe0 con 0x7fa350061e40 2024-03-16T19:30:53.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.615+0000 7fa354cd5640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fa350061e40 0x7fa350062250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.615+0000 7fa354cd5640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fa350061e40 0x7fa350062250 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.616+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3500cbbe0 con 0x7fa350061e40 2024-03-16T19:30:53.602 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:53.602 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:53.602 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:53.602 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:53.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.631+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3500b2ec0 con 0x7fa350061e40 2024-03-16T19:30:53.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.632+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3500b2ec0 con 0x7fa350061e40 2024-03-16T19:30:53.613 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:53.614 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:53.614 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.633+0000 7fa35773a640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fa3500c8110 0x7fa35015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.633+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3500b2be0 con 0x7fa3500c8110 2024-03-16T19:30:53.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.633+0000 7fa355cd7640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fa3500c8110 0x7fa35015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.633+0000 7fa355cd7640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fa3500c8110 0x7fa35015d6f0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.615 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.634+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3500b2be0 con 0x7fa3500c8110 2024-03-16T19:30:53.616 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:53.630 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.649+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3500c8860 con 0x7fa3500c8110 2024-03-16T19:30:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.652+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3480117c0 con 0x7fa3500c8110 2024-03-16T19:30:53.632 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:53.633 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:53.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.652+0000 7fa35773a640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fa35015e2b0 0x7fa35002bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.652+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa35015e710 con 0x7fa35015e2b0 2024-03-16T19:30:53.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.652+0000 7fa3554d6640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fa35015e2b0 0x7fa35002bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.653+0000 7fa3554d6640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fa35015e2b0 0x7fa35002bde0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.653+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa35015e710 con 0x7fa35015e2b0 2024-03-16T19:30:53.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.668+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa35015e710 con 0x7fa35015e2b0 2024-03-16T19:30:53.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.670+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa35015e710 con 0x7fa35015e2b0 2024-03-16T19:30:53.651 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:53.652 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:53.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.670+0000 7fa35773a640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fa35002c830 0x7fa35002cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.670+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa35002d250 con 0x7fa35002c830 2024-03-16T19:30:53.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.670+0000 7fa354cd5640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fa35002c830 0x7fa35002cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.671+0000 7fa354cd5640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fa35002c830 0x7fa35002cc90 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.671+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa35002d250 con 0x7fa35002c830 2024-03-16T19:30:53.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.689+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3502075b0 con 0x7fa35002c830 2024-03-16T19:30:53.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.691+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3502075b0 con 0x7fa35002c830 2024-03-16T19:30:53.672 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:53.672 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:53.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.692+0000 7fa35773a640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fa350207a40 0x7fa350209e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:53.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.692+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa35020ce10 con 0x7fa350207a40 2024-03-16T19:30:53.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.692+0000 7fa355cd7640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fa350207a40 0x7fa350209e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:53.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.692+0000 7fa355cd7640 1 --2- 172.21.15.172:0/3375782496 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fa350207a40 0x7fa350209e30 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:53.680 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:53.700+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:53.725 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:53.725 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:53.733 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:53.781 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:53.801+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:53.882 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:53.902+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:53.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.902+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa35020ce10 con 0x7fa350207a40 2024-03-16T19:30:53.891 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:53.891 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:53.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.917+0000 7fa35773a640 1 -- 172.21.15.172:0/3375782496 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa35020cde0 con 0x7fa350207a40 2024-03-16T19:30:53.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:53.918+0000 7fa345ffb640 1 -- 172.21.15.172:0/3375782496 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa35020cde0 con 0x7fa350207a40 2024-03-16T19:30:53.899 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:53.899 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:53.902 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:30:53.902 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:30:53.903 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:53.982 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:54.002+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:54.010 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:54.010 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:54.021 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:54.083 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:54.103+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:54.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 3.4 2024-03-16T19:30:54.132 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:54.132 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:54.142 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:54.184 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:54.204+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:54.285 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:54.305+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:54.289 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:54.289 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:54.297 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:54.385 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:54.405+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:54.407 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:54.407 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:54.416 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:54.500 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:54.520+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:54.563 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:54.563 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:54.572 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:54.613 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:54.633+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:54.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:30:54.687 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:54.707+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:54.722 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:54.722 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:54.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:54.788 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:54.808+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:54.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.810+0000 7f7048efb640 1 Processor -- start 2024-03-16T19:30:54.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.810+0000 7f7048efb640 1 -- start start 2024-03-16T19:30:54.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.811+0000 7f7048efb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7044157e50 0x7f70440ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:54.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.811+0000 7f7048efb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7044158230 0x7f70440cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:54.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.811+0000 7f7048efb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f70440c9530 0x7f704406b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.811+0000 7f7048efb640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704406aed0 con 0x7f70440c9530 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.811+0000 7f7048efb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70440719e0 con 0x7f7044157e50 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.811+0000 7f7048efb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70440c7b50 con 0x7f7044158230 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.811+0000 7f7042d9d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f70440c9530 0x7f704406b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.811+0000 7f7041d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7044158230 0x7f70440cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.812+0000 7f7041d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7044158230 0x7f70440cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:57900/0 (socket says 172.21.15.172:57900) 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.812+0000 7f7041d9b640 1 -- 172.21.15.172:0/2933916952 learned_addr learned my addr 172.21.15.172:0/2933916952 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.812+0000 7f704259c640 1 --2- 172.21.15.172:0/2933916952 >> v2:172.21.15.174:3300/0 conn(0x7f7044157e50 0x7f70440ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.812+0000 7f7041d9b640 1 -- 172.21.15.172:0/2933916952 >> v2:172.21.15.174:3300/0 conn(0x7f7044157e50 msgr2=0x7f70440ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.812+0000 7f7041d9b640 1 --2- 172.21.15.172:0/2933916952 >> v2:172.21.15.174:3300/0 conn(0x7f7044157e50 0x7f70440ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.812+0000 7f7041d9b640 1 -- 172.21.15.172:0/2933916952 >> v2:172.21.15.172:3300/0 conn(0x7f70440c9530 msgr2=0x7f704406b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:54.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.812+0000 7f7041d9b640 1 --2- 172.21.15.172:0/2933916952 >> v2:172.21.15.172:3300/0 conn(0x7f70440c9530 0x7f704406b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:54.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.812+0000 7f7041d9b640 1 -- 172.21.15.172:0/2933916952 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70440c7e10 con 0x7f7044158230 2024-03-16T19:30:54.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.812+0000 7f7042d9d640 1 --2- 172.21.15.172:0/2933916952 >> v2:172.21.15.172:3300/0 conn(0x7f70440c9530 0x7f704406b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:54.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.813+0000 7f7041d9b640 1 --2- 172.21.15.172:0/2933916952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7044158230 0x7f70440cd4a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f70440cda80 tx=0x7f703802b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e95f6984c8970f51 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:54.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.813+0000 7f704159a640 1 -- 172.21.15.172:0/2933916952 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038031860 con 0x7f7044158230 2024-03-16T19:30:54.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.813+0000 7f704159a640 1 -- 172.21.15.172:0/2933916952 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70380319c0 con 0x7f7044158230 2024-03-16T19:30:54.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.813+0000 7f704159a640 1 -- 172.21.15.172:0/2933916952 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038031d50 con 0x7f7044158230 2024-03-16T19:30:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.814+0000 7f7048efb640 1 -- 172.21.15.172:0/2933916952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7044158230 msgr2=0x7f70440cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.814+0000 7f7048efb640 1 --2- 172.21.15.172:0/2933916952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7044158230 0x7f70440cd4a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f70440cda80 tx=0x7f703802b900 comp rx=0 tx=0).stop 2024-03-16T19:30:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.814+0000 7f7048efb640 1 -- 172.21.15.172:0/2933916952 shutdown_connections 2024-03-16T19:30:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.814+0000 7f7048efb640 1 --2- 172.21.15.172:0/2933916952 >> v2:172.21.15.172:3300/0 conn(0x7f70440c9530 0x7f704406b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.814+0000 7f7048efb640 1 --2- 172.21.15.172:0/2933916952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7044158230 0x7f70440cd4a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:54.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.814+0000 7f7048efb640 1 --2- 172.21.15.172:0/2933916952 >> v2:172.21.15.174:3300/0 conn(0x7f7044157e50 0x7f70440ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:54.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.814+0000 7f7048efb640 1 -- 172.21.15.172:0/2933916952 >> 172.21.15.172:0/2933916952 conn(0x7f704406db70 msgr2=0x7f704406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:54.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.814+0000 7f7048efb640 1 -- 172.21.15.172:0/2933916952 shutdown_connections 2024-03-16T19:30:54.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.814+0000 7f7048efb640 1 -- 172.21.15.172:0/2933916952 wait complete. 2024-03-16T19:30:54.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.817+0000 7f7048efb640 1 Processor -- start 2024-03-16T19:30:54.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.817+0000 7f7048efb640 1 -- start start 2024-03-16T19:30:54.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.818+0000 7f7048efb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70440c9530 0x7f70440c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:54.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.818+0000 7f7048efb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7044157e50 0x7f704414c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:54.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.818+0000 7f7048efb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7044158230 0x7f70440c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:54.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.818+0000 7f704259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70440c9530 0x7f70440c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:54.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.818+0000 7f704259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70440c9530 0x7f70440c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:57910/0 (socket says 172.21.15.172:57910) 2024-03-16T19:30:54.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.818+0000 7f7041d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7044157e50 0x7f704414c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:54.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.818+0000 7f7041d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7044157e50 0x7f704414c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:37252/0 (socket says 172.21.15.172:37252) 2024-03-16T19:30:54.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7048efb640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70441ed120 con 0x7f7044158230 2024-03-16T19:30:54.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70441ec2c0 con 0x7f7044157e50 2024-03-16T19:30:54.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70441ec5a0 con 0x7f70440c9530 2024-03-16T19:30:54.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7042d9d640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7044158230 0x7f70440c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:54.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f704259c640 1 -- 172.21.15.172:0/1068955714 learned_addr learned my addr 172.21.15.172:0/1068955714 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:54.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7042d9d640 1 -- 172.21.15.172:0/1068955714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70440c9530 msgr2=0x7f70440c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7042d9d640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70440c9530 0x7f70440c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7042d9d640 1 -- 172.21.15.172:0/1068955714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7044157e50 msgr2=0x7f704414c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7042d9d640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7044157e50 0x7f704414c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7042d9d640 1 -- 172.21.15.172:0/1068955714 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c009a70 con 0x7f7044158230 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.819+0000 7f7041d9b640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7044157e50 0x7f704414c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.820+0000 7f7042d9d640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7044158230 0x7f70440c4630 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f703c00eea0 tx=0x7f703c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.820+0000 7f7032ffd640 1 -- 172.21.15.172:0/1068955714 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c010040 con 0x7f7044158230 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.820+0000 7f7032ffd640 1 -- 172.21.15.172:0/1068955714 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c014400 con 0x7f7044158230 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.821+0000 7f7032ffd640 1 -- 172.21.15.172:0/1068955714 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c0147b0 con 0x7f7044158230 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.821+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70380094d0 con 0x7f7044158230 2024-03-16T19:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.821+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70441ecb60 con 0x7f7044158230 2024-03-16T19:30:54.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.822+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7044075a70 con 0x7f7044158230 2024-03-16T19:30:54.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.823+0000 7f7032ffd640 1 -- 172.21.15.172:0/1068955714 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f703c014910 con 0x7f7044158230 2024-03-16T19:30:54.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.823+0000 7f7032ffd640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f702406b3b0 0x7f702406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:54.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.823+0000 7f7032ffd640 1 -- 172.21.15.172:0/1068955714 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f703c08a860 con 0x7f7044158230 2024-03-16T19:30:54.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.824+0000 7f704259c640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f702406b3b0 0x7f702406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:54.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.824+0000 7f704259c640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f702406b3b0 0x7f702406d870 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f703400fcd0 tx=0x7f7034013040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:54.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:54.831+0000 7f7032ffd640 1 -- 172.21.15.172:0/1068955714 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f703c0553c0 con 0x7f7044158230 2024-03-16T19:30:54.852 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:54.852 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:54.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:54.889 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:54.909+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:54.990 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:55.010+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:55.052 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:55.052 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:55.062 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:55.107 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:55.127+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:55.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.138+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f70440c4390 con 0x7f7044158230 2024-03-16T19:30:55.191 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:55.211+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:55.204 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:55.204 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:55.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:55.236 INFO:teuthology.orchestra.run.smithi172.stdout: marking collection for removal 2024-03-16T19:30:55.236 INFO:teuthology.orchestra.run.smithi172.stdout:setting '_remove' omap key 2024-03-16T19:30:55.236 INFO:teuthology.orchestra.run.smithi172.stdout:finish_remove_pgs 3.4_head removing 3.4 2024-03-16T19:30:55.236 INFO:teuthology.orchestra.run.smithi172.stdout:Remove successful 2024-03-16T19:30:55.292 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:55.312+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:55.389 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:55.389 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:55.393 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:55.413+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:55.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:55.494 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:55.514+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:55.551 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:55.551 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:55.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:55.601 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:55.621+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:55.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.712+0000 7f7032ffd640 1 -- 172.21.15.172:0/1068955714 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v62) v1 ==== 87+0+0 (secure 0 0 0) 0x7f703c058a80 con 0x7f7044158230 2024-03-16T19:30:55.692 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:30:55.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f702406b3b0 msgr2=0x7f702406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:55.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f702406b3b0 0x7f702406d870 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f703400fcd0 tx=0x7f7034013040 comp rx=0 tx=0).stop 2024-03-16T19:30:55.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7044158230 msgr2=0x7f70440c4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:55.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7044158230 0x7f70440c4630 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f703c00eea0 tx=0x7f703c00ef90 comp rx=0 tx=0).stop 2024-03-16T19:30:55.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 shutdown_connections 2024-03-16T19:30:55.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7044158230 0x7f70440c4630 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:55.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7044157e50 0x7f704414c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:55.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f702406b3b0 0x7f702406d870 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:55.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 --2- 172.21.15.172:0/1068955714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70440c9530 0x7f70440c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:55.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.715+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 >> 172.21.15.172:0/1068955714 conn(0x7f704406db70 msgr2=0x7f70440c9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:55.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.716+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 shutdown_connections 2024-03-16T19:30:55.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:55.716+0000 7f7048efb640 1 -- 172.21.15.172:0/1068955714 wait complete. 2024-03-16T19:30:55.697 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:55.717+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:55.735 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:55.735 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:55.747 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:55.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.4.2 2024-03-16T19:30:55.798 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:55.818+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:55.898 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:55.918+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:55.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:30:55.967 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:55.967 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:55.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:55.999 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:56.019+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:56.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.041+0000 7f96575f4640 1 Processor -- start 2024-03-16T19:30:56.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.041+0000 7f96575f4640 1 -- start start 2024-03-16T19:30:56.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f96575f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9650151ae0 0x7f9650071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f96575f4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f96500c44a0 0x7f96500720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f96575f4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965006ca50 con 0x7f96500c7290 2024-03-16T19:30:56.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f96575f4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965006ccf0 con 0x7f96500c44a0 2024-03-16T19:30:56.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f96575f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96500c8c50 con 0x7f9650151ae0 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f9655390640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9650151ae0 0x7f9650071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f9655390640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9650151ae0 0x7f9650071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:57918/0 (socket says 172.21.15.172:57918) 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f9655390640 1 -- 172.21.15.172:0/240191579 learned_addr learned my addr 172.21.15.172:0/240191579 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f9654b8f640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f96500c7290 0x7f9650157720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:53870/0 (socket says 172.21.15.172:53870) 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.042+0000 7f9655b91640 1 --2- 172.21.15.172:0/240191579 >> v2:172.21.15.174:3300/0 conn(0x7f96500c44a0 0x7f96500720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.043+0000 7f9655390640 1 -- 172.21.15.172:0/240191579 >> v2:172.21.15.174:3300/0 conn(0x7f96500c44a0 msgr2=0x7f96500720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.043+0000 7f9647fff640 1 -- 172.21.15.172:0/240191579 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819564123 0 0) 0x7f965006ca50 con 0x7f96500c7290 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.043+0000 7f9655390640 1 --2- 172.21.15.172:0/240191579 >> v2:172.21.15.174:3300/0 conn(0x7f96500c44a0 0x7f96500720d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.043+0000 7f9655390640 1 -- 172.21.15.172:0/240191579 >> v1:172.21.15.174:6789/0 conn(0x7f96500c7290 legacy=0x7f9650157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.043+0000 7f9655390640 1 -- 172.21.15.172:0/240191579 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96500c8f30 con 0x7f9650151ae0 2024-03-16T19:30:56.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.043+0000 7f9655b91640 1 --2- 172.21.15.172:0/240191579 >> v2:172.21.15.174:3300/0 conn(0x7f96500c44a0 0x7f96500720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:56.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.044+0000 7f9655390640 1 --2- 172.21.15.172:0/240191579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9650151ae0 0x7f9650071b90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f964c009870 tx=0x7f964c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3c5b0b520957f81 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.044+0000 7f9647fff640 1 -- 172.21.15.172:0/240191579 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c030840 con 0x7f9650151ae0 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.044+0000 7f9647fff640 1 -- 172.21.15.172:0/240191579 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964c0309a0 con 0x7f9650151ae0 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.044+0000 7f9647fff640 1 -- 172.21.15.172:0/240191579 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c030d30 con 0x7f9650151ae0 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.045+0000 7f96575f4640 1 -- 172.21.15.172:0/240191579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9650151ae0 msgr2=0x7f9650071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.045+0000 7f96575f4640 1 --2- 172.21.15.172:0/240191579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9650151ae0 0x7f9650071b90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f964c009870 tx=0x7f964c02b980 comp rx=0 tx=0).stop 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.045+0000 7f96575f4640 1 -- 172.21.15.172:0/240191579 shutdown_connections 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.045+0000 7f96575f4640 1 --2- 172.21.15.172:0/240191579 >> v2:172.21.15.174:3300/0 conn(0x7f96500c44a0 0x7f96500720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.045+0000 7f96575f4640 1 --2- 172.21.15.172:0/240191579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9650151ae0 0x7f9650071b90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.045+0000 7f96575f4640 1 -- 172.21.15.172:0/240191579 >> 172.21.15.172:0/240191579 conn(0x7f9650066c20 msgr2=0x7f9650067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.045+0000 7f96575f4640 1 -- 172.21.15.172:0/240191579 shutdown_connections 2024-03-16T19:30:56.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.045+0000 7f96575f4640 1 -- 172.21.15.172:0/240191579 wait complete. 2024-03-16T19:30:56.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.047+0000 7f96575f4640 1 Processor -- start 2024-03-16T19:30:56.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.047+0000 7f96575f4640 1 -- start start 2024-03-16T19:30:56.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.047+0000 7f96575f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96500c44a0 0x7f96500cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.047+0000 7f96575f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96500c7290 0x7f96501578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.047+0000 7f96575f4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9650151ae0 0x7f9650154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.047+0000 7f96575f4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9650155db0 con 0x7f9650151ae0 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.047+0000 7f96575f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9650157440 con 0x7f96500c44a0 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.047+0000 7f96575f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9650156830 con 0x7f96500c7290 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655b91640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9650151ae0 0x7f9650154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655b91640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9650151ae0 0x7f9650154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59604/0 (socket says 172.21.15.172:59604) 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655b91640 1 -- 172.21.15.172:0/1847596719 learned_addr learned my addr 172.21.15.172:0/1847596719 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96500c7290 0x7f96501578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655390640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96500c44a0 0x7f96500cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655b91640 1 -- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96500c7290 msgr2=0x7f96501578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655b91640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96500c7290 0x7f96501578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655b91640 1 -- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96500c44a0 msgr2=0x7f96500cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655b91640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96500c44a0 0x7f96500cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:56.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655b91640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9648009a70 con 0x7f9650151ae0 2024-03-16T19:30:56.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655390640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96500c44a0 0x7f96500cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:30:56.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9655b91640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9650151ae0 0x7f9650154100 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f964800efb0 tx=0x7f964800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.048+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9648014480 con 0x7f9650151ae0 2024-03-16T19:30:56.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.049+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964800f040 con 0x7f9650151ae0 2024-03-16T19:30:56.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.049+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964c0094d0 con 0x7f9650151ae0 2024-03-16T19:30:56.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.049+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96480147e0 con 0x7f9650151ae0 2024-03-16T19:30:56.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.049+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9650156df0 con 0x7f9650151ae0 2024-03-16T19:30:56.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.050+0000 7f963b7fe640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f96501575f0 con 0x7f9650151ae0 2024-03-16T19:30:56.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.051+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9648014940 con 0x7f9650151ae0 2024-03-16T19:30:56.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.051+0000 7f9645ffb640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f963406b3b0 0x7f963406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.052+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f964808ac00 con 0x7f9650151ae0 2024-03-16T19:30:56.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.052+0000 7f9655390640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f963406b3b0 0x7f963406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.052+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v62) v1 ==== 54+0+16 (secure 0 0 0) 0x7f96501575f0 con 0x7f9650151ae0 2024-03-16T19:30:56.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.052+0000 7f96575f4640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9628001640 0x7f9628003b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.052+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9628006ba0 con 0x7f9628001640 2024-03-16T19:30:56.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.052+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9628001640 0x7f9628003b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.052+0000 7f9655390640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f963406b3b0 0x7f963406d870 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f964c036000 tx=0x7f964c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9628001640 0x7f9628003b00 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9654b8f640 1 -- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9628001640 msgr2=0x7f9628003b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-03-16T19:30:56.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9654b8f640 1 -- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9628001640 msgr2=0x7f9628003b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:30:56.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9628001640 0x7f9628003b00 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:30:56.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9628001640 0x7f9628003b00 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:56.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9628001640 msgr2=0x7f9628003b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:30:56.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9645ffb640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9628001640 0x7f9628003b00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:56.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9645ffb640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9634074f10 0x7f9634077300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9634077860 con 0x7f9634074f10 2024-03-16T19:30:56.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=63}) v3 -- 0x7f96340779d0 con 0x7f9650151ae0 2024-03-16T19:30:56.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9634074f10 0x7f9634077300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.053+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f9634074f10 0x7f9634077300 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.054+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9634077860 con 0x7f9634074f10 2024-03-16T19:30:56.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.079+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9628005c20 con 0x7f9634074f10 2024-03-16T19:30:56.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.081+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9628005c20 con 0x7f9634074f10 2024-03-16T19:30:56.062 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:56.062 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:56.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.082+0000 7f96575f4640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9628004180 0x7f9628003a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.082+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9628006880 con 0x7f9628004180 2024-03-16T19:30:56.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.082+0000 7f9655390640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9628004180 0x7f9628003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.083+0000 7f9655390640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f9628004180 0x7f9628003a60 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.083+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9628006880 con 0x7f9628004180 2024-03-16T19:30:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.112+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9628004850 con 0x7f9628004180 2024-03-16T19:30:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.115+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9628004850 con 0x7f9628004180 2024-03-16T19:30:56.095 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:56.096 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:56.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.115+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9628004cc0 con 0x7f9650151ae0 2024-03-16T19:30:56.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.115+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9648055750 con 0x7f9650151ae0 2024-03-16T19:30:56.096 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:30:56.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.116+0000 7f96575f4640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9628005280 0x7f96280092e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.116+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f962800c380 con 0x7f9628005280 2024-03-16T19:30:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.116+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9628005280 0x7f96280092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.116+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f9628005280 0x7f96280092e0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.117+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f962800c380 con 0x7f9628005280 2024-03-16T19:30:56.100 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:56.120+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:56.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.132+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f962800b370 con 0x7f9628005280 2024-03-16T19:30:56.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.134+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f962800b370 con 0x7f9628005280 2024-03-16T19:30:56.114 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:56.115 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:56.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.134+0000 7f96575f4640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9628009960 0x7f9628009dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.134+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f962800a310 con 0x7f9628009960 2024-03-16T19:30:56.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.134+0000 7f9655b91640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9628009960 0x7f9628009dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.135+0000 7f9655b91640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f9628009960 0x7f9628009dd0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.135+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f962800a310 con 0x7f9628009960 2024-03-16T19:30:56.118 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:30:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:30:56.119 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:56.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.150+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f962800a810 con 0x7f9628009960 2024-03-16T19:30:56.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.152+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f962800a810 con 0x7f9628009960 2024-03-16T19:30:56.132 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:56.133 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:56.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.152+0000 7f96575f4640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f962800e590 0x7f9628010980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.152+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9628013750 con 0x7f962800e590 2024-03-16T19:30:56.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.153+0000 7f9655390640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f962800e590 0x7f9628010980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.153+0000 7f9655390640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f962800e590 0x7f9628010980 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.154+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9628013750 con 0x7f962800e590 2024-03-16T19:30:56.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.182+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9628010ec0 con 0x7f962800e590 2024-03-16T19:30:56.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.184+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9628010ec0 con 0x7f962800e590 2024-03-16T19:30:56.164 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:56.165 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:56.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.184+0000 7f96575f4640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9628011570 0x7f96280119d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.184+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9628016ab0 con 0x7f9628011570 2024-03-16T19:30:56.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.184+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9628011570 0x7f96280119d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.185+0000 7f9654b8f640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f9628011570 0x7f96280119d0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.186+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9628016ab0 con 0x7f9628011570 2024-03-16T19:30:56.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:30:56.187 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:30:56.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.213+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9628011f60 con 0x7f9628011570 2024-03-16T19:30:56.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.215+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9628011f60 con 0x7f9628011570 2024-03-16T19:30:56.195 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:56.196 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:56.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.215+0000 7f96575f4640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f96280125e0 0x7f9628012a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:56.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.215+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96280130f0 con 0x7f96280125e0 2024-03-16T19:30:56.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.216+0000 7f9655b91640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f96280125e0 0x7f9628012a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:56.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.217+0000 7f9655b91640 1 --2- 172.21.15.172:0/1847596719 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f96280125e0 0x7f9628012a20 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:56.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.218+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f96280130f0 con 0x7f96280125e0 2024-03-16T19:30:56.201 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:56.221+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:56.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.246+0000 7f96575f4640 1 -- 172.21.15.172:0/1847596719 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9628013370 con 0x7f96280125e0 2024-03-16T19:30:56.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:56.249+0000 7f9645ffb640 1 -- 172.21.15.172:0/1847596719 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f9628013370 con 0x7f96280125e0 2024-03-16T19:30:56.229 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:56.230 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:30:56.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:30:56.289 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:30:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:30:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:30:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.951327771999999, 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.951313299000006, 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:30:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:30:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.950808809999998, 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:30:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.950793414000003, 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:30:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:30:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.950334730999998, 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:30:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:30:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.949788561000005, 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:30:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:30:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.949231470000001, 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.948029722000001, 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:30:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.947451013000006, 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:30:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:30:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.947401486000004, 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:30:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.946440843000005, 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.942726644999993, 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:30:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.502325733000006, 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:30:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:30:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:30:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 74.936772730000001, 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:30:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:30:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 73.429174004999993, 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:30:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:30:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:30:56.351 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:56.353 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:56.353 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:56.372+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:56.453 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:56.473+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:56.460 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:56.460 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:56.460 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:56.460 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:56.469 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:56.554 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:56.574+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:56.602 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:56.602 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:56.602 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:56.602 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:56.602 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:56.614 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:30:56.655 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:56.675+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:56.725 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.387450704000003, 2024-03-16T19:30:56.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:30:56.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:30:56.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:56.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.730 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.386223282000003, 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:56.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:30:56.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:30:56.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.385820576, 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:30:56.734 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:56.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.738 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.385211824999999, 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:56.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:56.742 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.384767956000005, 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:30:56.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:56.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:56.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.38462518, 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:56.747 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:56.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:30:56.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.384610639000002, 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:30:56.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.381732944000007, 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:56.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:30:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:56.759 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:56.760 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:30:56.760 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:56.780+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:56.861 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:56.881+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:56.867 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:56.867 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:56.867 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:56.867 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:56.876 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:30:56.919 INFO:teuthology.orchestra.run.smithi172.stdout:get_pg_num_history pg_num_history pg_num_history(e58 pg_nums {1={15=1},2={18=32},3={19=64}} deleted_pools ) 2024-03-16T19:30:56.919 INFO:teuthology.orchestra.run.smithi172.stdout:Importing pgid 3.4 2024-03-16T19:30:56.920 INFO:teuthology.orchestra.run.smithi172.stdout:write_pg epoch 20 info 3.4( empty local-lis/les=19/20 n=0 ec=19/19 lis/c=19/19 les/c/f=20/20/0 sis=19) 2024-03-16T19:30:56.920 INFO:teuthology.orchestra.run.smithi172.stdout:Import successful 2024-03-16T19:30:56.961 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:56.981+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:57.036 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:57.037 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:57.037 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:57.037 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:57.037 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:57.048 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:30:57.062 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:57.082+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:57.154 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.816097940000006, 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:30:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.816018822000004, 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:30:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.815404697999995, 2024-03-16T19:30:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:30:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.814964629000002, 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:30:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 85.814723039, 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:30:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.176 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:57.177 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:30:57.178 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:57.197+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:57.277 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:57.297+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:57.323 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:57.323 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:57.323 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:57.332 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:30:57.378 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:57.398+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:57.439 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:57.439 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:57.439 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:57.439 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:57.439 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:57.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /var/log/ceph/exp.3.4.2 2024-03-16T19:30:57.448 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:30:57.482 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:57.502+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:57.579 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:57.599+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:57.595 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.257213719999996, 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.257189546999996, 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.601 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:30:57.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:30:57.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:30:57.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.25718372, 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:30:57.606 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:30:57.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:30:57.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.257179808000004, 2024-03-16T19:30:57.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:30:57.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.257173825999999, 2024-03-16T19:30:57.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:57.621 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.257170160000001, 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:57.626 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.2571595, 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:30:57.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:30:57.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.257155611000002, 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.630 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:30:57.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:30:57.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:30:57.633 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.25363892, 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:30:57.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.253568485000002, 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.253186138000004, 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:30:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:30:57.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:30:57.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.251419854999995, 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.649 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.249606932999995, 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:30:57.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 86.248751184, 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.653 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:30:57.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:30:57.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:30:57.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 81.242890418000002, 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.658 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:30:57.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:30:57.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:30:57.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 77.810339209999995, 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.662 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:30:57.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:30:57.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:30:57.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:30:57.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 76.242845504000002, 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:30:57.667 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:30:57.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:30:57.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:30:57.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:30:57.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 74.735233253000004, 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.672 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:30:57.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:30:57.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:30:57.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:30:57.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.238370027999999, 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.677 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:30:57.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:30:57.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 32.237834112000002, 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:57.681 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:30:57.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:30:57.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:30:57.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:57.685 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:57.687 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:30:57.687 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:57.706+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:57.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:30:57.787 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:57.807+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:57.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.835+0000 7fcc305ac640 1 Processor -- start 2024-03-16T19:30:57.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.835+0000 7fcc305ac640 1 -- start start 2024-03-16T19:30:57.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.836+0000 7fcc305ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815c340 0x7fcc280c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:57.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.836+0000 7fcc305ac640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc28152e90 con 0x7fcc28152ab0 2024-03-16T19:30:57.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.836+0000 7fcc305ac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280c30e0 con 0x7fcc280c9af0 2024-03-16T19:30:57.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.836+0000 7fcc305ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc280c85e0 con 0x7fcc2815c340 2024-03-16T19:30:57.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.836+0000 7fcc2e348640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fcc28152ab0 0x7fcc2815bc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:46454/0 (socket says 172.21.15.172:46454) 2024-03-16T19:30:57.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.836+0000 7fcc2e348640 1 -- 172.21.15.172:0/4168891077 learned_addr learned my addr 172.21.15.172:0/4168891077 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:30:57.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.836+0000 7fcc2db47640 1 --2- 172.21.15.172:0/4168891077 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815c340 0x7fcc280c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:57.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.836+0000 7fcc2d346640 1 -- 172.21.15.172:0/4168891077 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935649105 0 0) 0x7fcc24001110 con 0x7fcc28152ab0 2024-03-16T19:30:57.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.836+0000 7fcc2d346640 1 -- 172.21.15.172:0/4168891077 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc14003520 con 0x7fcc28152ab0 2024-03-16T19:30:57.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.837+0000 7fcc2d346640 1 -- 172.21.15.172:0/4168891077 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831824884 0 0) 0x7fcc280c30e0 con 0x7fcc280c9af0 2024-03-16T19:30:57.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.837+0000 7fcc2d346640 1 -- 172.21.15.172:0/4168891077 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc24001110 con 0x7fcc280c9af0 2024-03-16T19:30:57.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.837+0000 7fcc2db47640 1 -- 172.21.15.172:0/4168891077 >> v1:172.21.15.174:6789/0 conn(0x7fcc280c9af0 legacy=0x7fcc2806e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:57.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.837+0000 7fcc2db47640 1 -- 172.21.15.172:0/4168891077 >> v1:172.21.15.172:6789/0 conn(0x7fcc28152ab0 legacy=0x7fcc2815bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:57.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.837+0000 7fcc2db47640 1 -- 172.21.15.172:0/4168891077 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc280c33c0 con 0x7fcc2815c340 2024-03-16T19:30:57.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.837+0000 7fcc2db47640 1 --2- 172.21.15.172:0/4168891077 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815c340 0x7fcc280c8ac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcc180097d0 tx=0x7fcc1802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d63c072ea52d6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:57.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc2d346640 1 -- 172.21.15.172:0/4168891077 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18030800 con 0x7fcc2815c340 2024-03-16T19:30:57.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc2d346640 1 -- 172.21.15.172:0/4168891077 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18030960 con 0x7fcc2815c340 2024-03-16T19:30:57.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc2d346640 1 -- 172.21.15.172:0/4168891077 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18030cd0 con 0x7fcc2815c340 2024-03-16T19:30:57.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc305ac640 1 -- 172.21.15.172:0/4168891077 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815c340 msgr2=0x7fcc280c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:57.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc305ac640 1 --2- 172.21.15.172:0/4168891077 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815c340 0x7fcc280c8ac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcc180097d0 tx=0x7fcc1802b940 comp rx=0 tx=0).stop 2024-03-16T19:30:57.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc305ac640 1 -- 172.21.15.172:0/4168891077 shutdown_connections 2024-03-16T19:30:57.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc305ac640 1 --2- 172.21.15.172:0/4168891077 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815c340 0x7fcc280c8ac0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:57.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc305ac640 1 -- 172.21.15.172:0/4168891077 >> 172.21.15.172:0/4168891077 conn(0x7fcc28070810 msgr2=0x7fcc28070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:57.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc305ac640 1 -- 172.21.15.172:0/4168891077 shutdown_connections 2024-03-16T19:30:57.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.838+0000 7fcc305ac640 1 -- 172.21.15.172:0/4168891077 wait complete. 2024-03-16T19:30:57.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.840+0000 7fcc305ac640 1 Processor -- start 2024-03-16T19:30:57.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.840+0000 7fcc305ac640 1 -- start start 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc305ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc280c9af0 0x7fcc281f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc305ac640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc28152ab0 0x7fcc28159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc305ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815add0 0x7fcc281567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc305ac640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc28158be0 con 0x7fcc28152ab0 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc305ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc281595b0 con 0x7fcc280c9af0 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc305ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc28158600 con 0x7fcc2815add0 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc2db47640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc28152ab0 0x7fcc28159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc2e348640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc280c9af0 0x7fcc281f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc2db47640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc28152ab0 0x7fcc28159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59616/0 (socket says 172.21.15.172:59616) 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc2db47640 1 -- 172.21.15.172:0/74010632 learned_addr learned my addr 172.21.15.172:0/74010632 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.841+0000 7fcc2eb49640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815add0 0x7fcc281567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:57.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc2e348640 1 -- 172.21.15.172:0/74010632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815add0 msgr2=0x7fcc281567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc2e348640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815add0 0x7fcc281567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc2e348640 1 -- 172.21.15.172:0/74010632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc28152ab0 msgr2=0x7fcc28159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc2e348640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc28152ab0 0x7fcc28159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc2e348640 1 -- 172.21.15.172:0/74010632 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc18009400 con 0x7fcc280c9af0 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc2e348640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc280c9af0 0x7fcc281f0ee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400c730 tx=0x7fcc2400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc1effd640 1 -- 172.21.15.172:0/74010632 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24005ab0 con 0x7fcc280c9af0 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc1effd640 1 -- 172.21.15.172:0/74010632 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc24005c10 con 0x7fcc280c9af0 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc281570a0 con 0x7fcc280c9af0 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc1effd640 1 -- 172.21.15.172:0/74010632 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24007070 con 0x7fcc280c9af0 2024-03-16T19:30:57.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.842+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2815a310 con 0x7fcc280c9af0 2024-03-16T19:30:57.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.843+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc280c42f0 con 0x7fcc280c9af0 2024-03-16T19:30:57.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.844+0000 7fcc1effd640 1 -- 172.21.15.172:0/74010632 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fcc240071d0 con 0x7fcc280c9af0 2024-03-16T19:30:57.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.844+0000 7fcc1effd640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcc0c06b3b0 0x7fcc0c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:57.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.845+0000 7fcc1effd640 1 -- 172.21.15.172:0/74010632 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fcc240890a0 con 0x7fcc280c9af0 2024-03-16T19:30:57.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.845+0000 7fcc2db47640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcc0c06b3b0 0x7fcc0c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:57.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.845+0000 7fcc2db47640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcc0c06b3b0 0x7fcc0c06d870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009720 tx=0x7fcc18008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:57.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:57.854+0000 7fcc1effd640 1 -- 172.21.15.172:0/74010632 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc24053d30 con 0x7fcc280c9af0 2024-03-16T19:30:57.834 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:57.835 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:57.835 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:30:57.835 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:57.844 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:30:57.888 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:57.908+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:57.950 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:57.950 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:30:57.950 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:30:57.950 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:30:57.951 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:57.959 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:30:57.989 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:58.009+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:58.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.057+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fcc28152e90 con 0x7fcc280c9af0 2024-03-16T19:30:58.089 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:58.109+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:58.131 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:30:58.131 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:30:58.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:30:58.131 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:30:58.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.131 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 101.920256689, 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.136 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 101.543561142, 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.76758469799999, 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:58.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:30:58.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:30:58.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:30:58.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.145 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.767500285, 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:30:58.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:30:58.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:30:58.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.767492929, 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:30:58.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:30:58.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:30:58.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:30:58.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.574953737, 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.156 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:30:58.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.57494792999999, 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:30:58.161 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:58.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:30:58.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:58.166 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.57482979700001, 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:30:58.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.5747346, 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:58.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:30:58.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.176 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.574730589, 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.177 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:30:58.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:30:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:30:58.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.155798715, 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:30:58.182 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:30:58.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:30:58.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.187 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.15574934599999, 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:30:58.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:58.193 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.15567051399999, 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.155667303, 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:30:58.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:30:58.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.15566435, 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:30:58.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:58.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:30:58.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:30:58.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.155549376, 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.209 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:30:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.155546024, 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.214 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:58.219 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.155542908, 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.224 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.155482439, 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:30:58.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.546802389000007, 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:30:58.230 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:30:58.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:30:58.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:30:58.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:30:58.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:30:58.235 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.235 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.235 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.235 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:30:58.235 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:30:58.235 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:30:58.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:30:58.236 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:58.256+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:58.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:30:58.337 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:58.357+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:58.353 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:30:58.353 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:30:58.353 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:30:58.354 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:30:58.365 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:30:58.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.394+0000 7faa235e2640 1 Processor -- start 2024-03-16T19:30:58.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.394+0000 7faa235e2640 1 -- start start 2024-03-16T19:30:58.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa235e2640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faa1c070ed0 0x7faa1c159430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa235e2640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c159970 con 0x7faa1c0b0e60 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa235e2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c15ce70 con 0x7faa1c070af0 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa235e2640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1c15bfc0 con 0x7faa1c070ed0 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa20b7d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faa1c070ed0 0x7faa1c159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa20b7d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faa1c070ed0 0x7faa1c159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59632/0 (socket says 172.21.15.172:59632) 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa20b7d640 1 -- 172.21.15.172:0/1545068151 learned_addr learned my addr 172.21.15.172:0/1545068151 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa20b7d640 1 -- 172.21.15.172:0/1545068151 >> v1:172.21.15.174:6789/0 conn(0x7faa1c070af0 legacy=0x7faa1c0c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa13fff640 1 -- 172.21.15.172:0/1545068151 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750091958 0 0) 0x7faa1c15ce70 con 0x7faa1c070af0 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa20b7d640 1 -- 172.21.15.172:0/1545068151 >> v1:172.21.15.172:6789/0 conn(0x7faa1c0b0e60 legacy=0x7faa1c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.395+0000 7faa20b7d640 1 -- 172.21.15.172:0/1545068151 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c15c260 con 0x7faa1c070ed0 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.396+0000 7faa20b7d640 1 --2- 172.21.15.172:0/1545068151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa1c070ed0 0x7faa1c159430 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009870 tx=0x7faa0c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f94ccda25fc041 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.396+0000 7faa13fff640 1 -- 172.21.15.172:0/1545068151 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c02f840 con 0x7faa1c070ed0 2024-03-16T19:30:58.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.397+0000 7faa13fff640 1 -- 172.21.15.172:0/1545068151 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c02f9a0 con 0x7faa1c070ed0 2024-03-16T19:30:58.378 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.397+0000 7faa13fff640 1 -- 172.21.15.172:0/1545068151 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c02fd30 con 0x7faa1c070ed0 2024-03-16T19:30:58.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.397+0000 7faa235e2640 1 -- 172.21.15.172:0/1545068151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa1c070ed0 msgr2=0x7faa1c159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:58.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.397+0000 7faa235e2640 1 --2- 172.21.15.172:0/1545068151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa1c070ed0 0x7faa1c159430 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009870 tx=0x7faa0c02ba20 comp rx=0 tx=0).stop 2024-03-16T19:30:58.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.397+0000 7faa235e2640 1 -- 172.21.15.172:0/1545068151 shutdown_connections 2024-03-16T19:30:58.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.397+0000 7faa235e2640 1 --2- 172.21.15.172:0/1545068151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa1c070ed0 0x7faa1c159430 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:58.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.398+0000 7faa235e2640 1 -- 172.21.15.172:0/1545068151 >> 172.21.15.172:0/1545068151 conn(0x7faa1c06e880 msgr2=0x7faa1c06ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:58.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.398+0000 7faa235e2640 1 -- 172.21.15.172:0/1545068151 shutdown_connections 2024-03-16T19:30:58.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.398+0000 7faa235e2640 1 -- 172.21.15.172:0/1545068151 wait complete. 2024-03-16T19:30:58.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.399+0000 7faa235e2640 1 Processor -- start 2024-03-16T19:30:58.380 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.399+0000 7faa235e2640 1 -- start start 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa235e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c070af0 0x7faa1c157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa235e2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa1c070ed0 0x7faa1c14e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa235e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1c0b0e60 0x7faa1c14e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa235e2640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c150aa0 con 0x7faa1c070ed0 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa235e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c150200 con 0x7faa1c0b0e60 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa235e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c1504e0 con 0x7faa1c070af0 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa2137e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c070af0 0x7faa1c157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa2137e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c070af0 0x7faa1c157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:57956/0 (socket says 172.21.15.172:57956) 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa2137e640 1 -- 172.21.15.172:0/2634468652 learned_addr learned my addr 172.21.15.172:0/2634468652 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:30:58.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa21b7f640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1c0b0e60 0x7faa1c14e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.400+0000 7faa20b7d640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa1c070ed0 0x7faa1c14e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa2137e640 1 -- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1c0b0e60 msgr2=0x7faa1c14e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa2137e640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1c0b0e60 0x7faa1c14e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa2137e640 1 -- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa1c070ed0 msgr2=0x7faa1c14e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa2137e640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa1c070ed0 0x7faa1c14e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa2137e640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c0094d0 con 0x7faa1c070af0 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa20b7d640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa1c070ed0 0x7faa1c14e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa2137e640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa1c070af0 0x7faa1c157ae0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faa1800c460 tx=0x7faa1800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18005480 con 0x7faa1c070af0 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa180035f0 con 0x7faa1c070af0 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.401+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa180039a0 con 0x7faa1c070af0 2024-03-16T19:30:58.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.402+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c1507c0 con 0x7faa1c070af0 2024-03-16T19:30:58.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.402+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c0b1450 con 0x7faa1c070af0 2024-03-16T19:30:58.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.403+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7faa1c068d20 con 0x7faa1c070af0 2024-03-16T19:30:58.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.404+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7faa18003b00 con 0x7faa1c070af0 2024-03-16T19:30:58.384 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.404+0000 7faa11ffb640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa0006b3b0 0x7faa0006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.405+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5828+0+0 (secure 0 0 0) 0x7faa180897a0 con 0x7faa1c070af0 2024-03-16T19:30:58.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.405+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v62) v1 ==== 54+0+16 (secure 0 0 0) 0x7faa18090070 con 0x7faa1c070af0 2024-03-16T19:30:58.385 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.405+0000 7faa20b7d640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa0006b3b0 0x7faa0006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.405+0000 7faa235e2640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7faa1c158b00 0x7faa1c15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.405+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa1c15c820 con 0x7faa1c158b00 2024-03-16T19:30:58.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.405+0000 7faa21b7f640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7faa1c158b00 0x7faa1c15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.405+0000 7faa20b7d640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa0006b3b0 0x7faa0006d870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faa0c036000 tx=0x7faa0c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.406+0000 7faa21b7f640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7faa1c158b00 0x7faa1c15b6d0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.386 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.406+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faa14001a80 con 0x7faa1c158b00 2024-03-16T19:30:58.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.432+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa1c0c96c0 con 0x7faa1c158b00 2024-03-16T19:30:58.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.434+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faa14001f10 con 0x7faa1c158b00 2024-03-16T19:30:58.414 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:30:58.415 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:30:58.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.434+0000 7faa235e2640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7faa1c14cf70 0x7faa1c1eee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.434+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa1c14dca0 con 0x7faa1c14cf70 2024-03-16T19:30:58.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.434+0000 7faa2137e640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7faa1c14cf70 0x7faa1c1eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.435+0000 7faa2137e640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7faa1c14cf70 0x7faa1c1eee40 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.436+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faa1c14dca0 con 0x7faa1c14cf70 2024-03-16T19:30:58.438 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:58.458+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:58.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.464+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa1c06eeb0 con 0x7faa1c14cf70 2024-03-16T19:30:58.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.466+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faa1c06eeb0 con 0x7faa1c14cf70 2024-03-16T19:30:58.446 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:30:58.448 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:30:58.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.466+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=63}) v3 -- 0x7faa1c0cafb0 con 0x7faa1c070af0 2024-03-16T19:30:58.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.466+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faa1c1ef5d0 con 0x7faa1c070af0 2024-03-16T19:30:58.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.467+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa18057c30 con 0x7faa1c070af0 2024-03-16T19:30:58.448 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:30:58.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.467+0000 7faa235e2640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7faa1c061e40 0x7faa1c062270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.467+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa1c1efba0 con 0x7faa1c061e40 2024-03-16T19:30:58.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.467+0000 7faa20b7d640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7faa1c061e40 0x7faa1c062270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.448 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.468+0000 7faa20b7d640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7faa1c061e40 0x7faa1c062270 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.449 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.469+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faa1c1efba0 con 0x7faa1c061e40 2024-03-16T19:30:58.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.491+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa1c159d20 con 0x7faa1c061e40 2024-03-16T19:30:58.474 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.494+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faa1c159d20 con 0x7faa1c061e40 2024-03-16T19:30:58.474 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:30:58.478 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:30:58.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.494+0000 7faa235e2640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7faa1c15a760 0x7faa1c02bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.494+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa1c15abc0 con 0x7faa1c15a760 2024-03-16T19:30:58.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.494+0000 7faa21b7f640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7faa1c15a760 0x7faa1c02bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.494+0000 7faa21b7f640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7faa1c15a760 0x7faa1c02bde0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.495+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faa1c15abc0 con 0x7faa1c15a760 2024-03-16T19:30:58.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.523+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa1c02c570 con 0x7faa1c15a760 2024-03-16T19:30:58.505 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.525+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faa1c02c570 con 0x7faa1c15a760 2024-03-16T19:30:58.506 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:30:58.506 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:30:58.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.526+0000 7faa235e2640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7faa1c02cbf0 0x7faa1c2001c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.526+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa1c150620 con 0x7faa1c02cbf0 2024-03-16T19:30:58.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.526+0000 7faa2137e640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7faa1c02cbf0 0x7faa1c2001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.526+0000 7faa2137e640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7faa1c02cbf0 0x7faa1c2001c0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.527+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faa1c150620 con 0x7faa1c02cbf0 2024-03-16T19:30:58.533 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:58.533 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:58.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.554+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa1c150620 con 0x7faa1c02cbf0 2024-03-16T19:30:58.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.556+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faa1c150620 con 0x7faa1c02cbf0 2024-03-16T19:30:58.536 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:30:58.536 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:30:58.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.556+0000 7faa235e2640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7faa1c02d030 0x7faa1c202c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.556+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa1c2059f0 con 0x7faa1c02d030 2024-03-16T19:30:58.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.556+0000 7faa20b7d640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7faa1c02d030 0x7faa1c202c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.557+0000 7faa20b7d640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7faa1c02d030 0x7faa1c202c20 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.557+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faa1c2059f0 con 0x7faa1c02d030 2024-03-16T19:30:58.538 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:58.558+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:58.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:30:58.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.572+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa1c203160 con 0x7faa1c02d030 2024-03-16T19:30:58.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.574+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faa1c203160 con 0x7faa1c02d030 2024-03-16T19:30:58.555 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:30:58.555 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:30:58.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.574+0000 7faa235e2640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7faa1c203840 0x7faa1c205c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:30:58.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.574+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa1c208a00 con 0x7faa1c203840 2024-03-16T19:30:58.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.574+0000 7faa21b7f640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7faa1c203840 0x7faa1c205c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:30:58.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.575+0000 7faa21b7f640 1 --2- 172.21.15.172:0/2634468652 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7faa1c203840 0x7faa1c205c30 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:30:58.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.575+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faa1c208a00 con 0x7faa1c203840 2024-03-16T19:30:58.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.593+0000 7faa235e2640 1 -- 172.21.15.172:0/2634468652 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faa1c206170 con 0x7faa1c203840 2024-03-16T19:30:58.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.595+0000 7faa11ffb640 1 -- 172.21.15.172:0/2634468652 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7faa1c206170 con 0x7faa1c203840 2024-03-16T19:30:58.575 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:30:58.575 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:30:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:30:58.579 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:30:58.639 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:58.659+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:58.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:30:58.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.742+0000 7fcc1effd640 1 -- 172.21.15.172:0/74010632 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v63) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcc28152e90 con 0x7fcc280c9af0 2024-03-16T19:30:58.724 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcc0c06b3b0 msgr2=0x7fcc0c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcc0c06b3b0 0x7fcc0c06d870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009720 tx=0x7fcc18008030 comp rx=0 tx=0).stop 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc280c9af0 msgr2=0x7fcc281f0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc280c9af0 0x7fcc281f0ee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400c730 tx=0x7fcc2400cbf0 comp rx=0 tx=0).stop 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 shutdown_connections 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcc0c06b3b0 0x7fcc0c06d870 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc2815add0 0x7fcc281567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc28152ab0 0x7fcc28159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 --2- 172.21.15.172:0/74010632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc280c9af0 0x7fcc281f0ee0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.746+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 >> 172.21.15.172:0/74010632 conn(0x7fcc28070810 msgr2=0x7fcc280c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.747+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 shutdown_connections 2024-03-16T19:30:58.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:30:58.747+0000 7fcc305ac640 1 -- 172.21.15.172:0/74010632 wait complete. 2024-03-16T19:30:58.740 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:58.760+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:58.840 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:30:58.860+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:30:58.874 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:58.875 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:58.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:30:58.941 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:58.961+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:59.015 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:59.015 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:59.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:30:59.042 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:59.062+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:59.143 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:30:59.162+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:30:59.200 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:59.200 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:59.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:30:59.254 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:59.274+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:59.345 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:59.365+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:59.363 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:59.363 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:59.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:30:59.458 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:59.478+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:59.523 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:59.523 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:59.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:30:59.545 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:30:59.565+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:30:59.646 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:30:59.666+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:30:59.686 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:59.686 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:59.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:30:59.747 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:30:59.767+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:30:59.827 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:59.828 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:59.836 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:30:59.879 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:30:59.899+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:30:59.946 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:30:59.946 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:30:59.949 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:30:59.969+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:30:59.955 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:00.050 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:00.070+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:00.102 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:00.102 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:00.111 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:00.152 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:00.172+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:00.252 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:00.272+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:00.258 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:00.258 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:00.267 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:00.353 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:00.373+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:00.415 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:00.415 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:00.423 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:00.464 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:00.484+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:00.555 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:00.575+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:00.570 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:00.570 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:00.579 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:00.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:31:00.656 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:00.676+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:00.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.707+0000 7f21b8fd7640 1 Processor -- start 2024-03-16T19:31:00.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.707+0000 7f21b8fd7640 1 -- start start 2024-03-16T19:31:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.708+0000 7f21b8fd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b4154ed0 0x7f21b415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.708+0000 7f21b8fd7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f21b40b3630 0x7f21b40cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.708+0000 7f21b8fd7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b40b4270 con 0x7f21b40cb5e0 2024-03-16T19:31:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.708+0000 7f21b8fd7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b41552b0 con 0x7f21b40b3630 2024-03-16T19:31:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.708+0000 7f21b8fd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b415b4b0 con 0x7f21b4154ed0 2024-03-16T19:31:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.709+0000 7f21b259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b4154ed0 0x7f21b415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.709+0000 7f21b1d9b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f21b40b3630 0x7f21b40cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.709+0000 7f21b2d9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f21b40cb5e0 0x7f21b40b4700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:41494/0 (socket says 172.21.15.172:41494) 2024-03-16T19:31:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.709+0000 7f21b2d9d640 1 -- 172.21.15.172:0/1113426999 learned_addr learned my addr 172.21.15.172:0/1113426999 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:00.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.709+0000 7f21b259c640 1 -- 172.21.15.172:0/1113426999 >> v2:172.21.15.174:3300/0 conn(0x7f21b40b3630 msgr2=0x7f21b40cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:00.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.709+0000 7f21b259c640 1 --2- 172.21.15.172:0/1113426999 >> v2:172.21.15.174:3300/0 conn(0x7f21b40b3630 0x7f21b40cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.709+0000 7f21b259c640 1 -- 172.21.15.172:0/1113426999 >> v1:172.21.15.172:6789/0 conn(0x7f21b40cb5e0 legacy=0x7f21b40b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:00.691 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.709+0000 7f21b259c640 1 -- 172.21.15.172:0/1113426999 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b415b790 con 0x7f21b4154ed0 2024-03-16T19:31:00.691 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.709+0000 7f21b1d9b640 1 --2- 172.21.15.172:0/1113426999 >> v2:172.21.15.174:3300/0 conn(0x7f21b40b3630 0x7f21b40cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:00.691 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.710+0000 7f21b259c640 1 --2- 172.21.15.172:0/1113426999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b4154ed0 0x7f21b415ab40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f21a40097a0 tx=0x7f21a402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a5adb5a9c8355fe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.691 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.710+0000 7f21b159a640 1 -- 172.21.15.172:0/1113426999 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a402d040 con 0x7f21b4154ed0 2024-03-16T19:31:00.691 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.710+0000 7f21b159a640 1 -- 172.21.15.172:0/1113426999 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a4031810 con 0x7f21b4154ed0 2024-03-16T19:31:00.691 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.710+0000 7f21b159a640 1 -- 172.21.15.172:0/1113426999 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4031ba0 con 0x7f21b4154ed0 2024-03-16T19:31:00.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.711+0000 7f21b8fd7640 1 -- 172.21.15.172:0/1113426999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b4154ed0 msgr2=0x7f21b415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:00.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.711+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/1113426999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b4154ed0 0x7f21b415ab40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f21a40097a0 tx=0x7f21a402b8b0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.711+0000 7f21b8fd7640 1 -- 172.21.15.172:0/1113426999 shutdown_connections 2024-03-16T19:31:00.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.711+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/1113426999 >> v2:172.21.15.174:3300/0 conn(0x7f21b40b3630 0x7f21b40cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.711+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/1113426999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b4154ed0 0x7f21b415ab40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.711+0000 7f21b8fd7640 1 -- 172.21.15.172:0/1113426999 >> 172.21.15.172:0/1113426999 conn(0x7f21b406bf60 msgr2=0x7f21b406d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:00.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.711+0000 7f21b8fd7640 1 -- 172.21.15.172:0/1113426999 shutdown_connections 2024-03-16T19:31:00.692 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.711+0000 7f21b8fd7640 1 -- 172.21.15.172:0/1113426999 wait complete. 2024-03-16T19:31:00.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.713+0000 7f21b8fd7640 1 Processor -- start 2024-03-16T19:31:00.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.713+0000 7f21b8fd7640 1 -- start start 2024-03-16T19:31:00.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.714+0000 7f21b8fd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b40b3630 0x7f21b4157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.714+0000 7f21b8fd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b40cb5e0 0x7f21b40c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.714+0000 7f21b8fd7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b4154ed0 0x7f21b40c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.714+0000 7f21b8fd7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40c69a0 con 0x7f21b4154ed0 2024-03-16T19:31:00.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.714+0000 7f21b8fd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40ca2f0 con 0x7f21b40b3630 2024-03-16T19:31:00.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.714+0000 7f21b8fd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b41f13b0 con 0x7f21b40cb5e0 2024-03-16T19:31:00.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b1d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b40cb5e0 0x7f21b40c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b2d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b4154ed0 0x7f21b40c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b1d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b40cb5e0 0x7f21b40c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:49724/0 (socket says 172.21.15.172:49724) 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b1d9b640 1 -- 172.21.15.172:0/3884439936 learned_addr learned my addr 172.21.15.172:0/3884439936 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b2d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b4154ed0 0x7f21b40c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57082/0 (socket says 172.21.15.172:57082) 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b259c640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b40b3630 0x7f21b4157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b1d9b640 1 -- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b40b3630 msgr2=0x7f21b4157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b1d9b640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b40b3630 0x7f21b4157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b1d9b640 1 -- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b4154ed0 msgr2=0x7f21b40c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b1d9b640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b4154ed0 0x7f21b40c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b1d9b640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a8009400 con 0x7f21b40cb5e0 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b2d9d640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f21b4154ed0 0x7f21b40c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b259c640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21b40b3630 0x7f21b4157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21b1d9b640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21b40cb5e0 0x7f21b40c7980 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f21a800e960 tx=0x7f21a800ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.715+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a800ca00 con 0x7f21b40cb5e0 2024-03-16T19:31:00.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.716+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a800cb60 con 0x7f21b40cb5e0 2024-03-16T19:31:00.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.716+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8013470 con 0x7f21b40cb5e0 2024-03-16T19:31:00.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.717+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a4009400 con 0x7f21b40cb5e0 2024-03-16T19:31:00.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.717+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b40ca4e0 con 0x7f21b40cb5e0 2024-03-16T19:31:00.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.718+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f21b4068d20 con 0x7f21b40cb5e0 2024-03-16T19:31:00.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.718+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f21a80135d0 con 0x7f21b40cb5e0 2024-03-16T19:31:00.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.718+0000 7f21a2ffd640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f219406b3b0 0x7f219406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.719+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f21a8089dc0 con 0x7f21b40cb5e0 2024-03-16T19:31:00.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.719+0000 7f21b259c640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f219406b3b0 0x7f219406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.719+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v63) v1 ==== 54+0+16 (secure 0 0 0) 0x7f21a8054ae0 con 0x7f21b40cb5e0 2024-03-16T19:31:00.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.719+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f21b40c64a0 0x7f21b40c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.719+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b40b2e20 con 0x7f21b40c64a0 2024-03-16T19:31:00.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.719+0000 7f21b259c640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f219406b3b0 0x7f219406d870 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f21a4036000 tx=0x7f21a403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.719+0000 7f21b2d9d640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f21b40c64a0 0x7f21b40c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.720+0000 7f21b2d9d640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f21b40c64a0 0x7f21b40c4df0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.720+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21ac0020b0 con 0x7f21b40c64a0 2024-03-16T19:31:00.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.746+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b40681c0 con 0x7f21b40c64a0 2024-03-16T19:31:00.729 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.749+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21b40681c0 con 0x7f21b40c64a0 2024-03-16T19:31:00.729 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:00.729 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:00.729 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:00.730 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:00.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.749+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f21b414eb00 0x7f21b41f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.749+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b40cbbe0 con 0x7f21b414eb00 2024-03-16T19:31:00.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.749+0000 7f21b259c640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f21b414eb00 0x7f21b41f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.749+0000 7f21b259c640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f21b414eb00 0x7f21b41f3920 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.750+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21b40cbbe0 con 0x7f21b414eb00 2024-03-16T19:31:00.739 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:00.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:31:00.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.781+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b4158ce0 con 0x7f21b414eb00 2024-03-16T19:31:00.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.783+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21b4158ce0 con 0x7f21b414eb00 2024-03-16T19:31:00.763 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:00.764 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:00.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.783+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=64}) v3 -- 0x7f21b4158a00 con 0x7f21b40cb5e0 2024-03-16T19:31:00.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.783+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21b4153830 con 0x7f21b40cb5e0 2024-03-16T19:31:00.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.784+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21a8054ae0 con 0x7f21b40cb5e0 2024-03-16T19:31:00.765 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:00.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.784+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f21b40c5840 0x7f21b41f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.784+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b41553f0 con 0x7f21b40c5840 2024-03-16T19:31:00.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.784+0000 7f21b1d9b640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f21b40c5840 0x7f21b41f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.785+0000 7f21b1d9b640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f21b40c5840 0x7f21b41f4370 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.786+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21b41553f0 con 0x7f21b40c5840 2024-03-16T19:31:00.782 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:00.802+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:00.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.803+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b4155810 con 0x7f21b40c5840 2024-03-16T19:31:00.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.806+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21b4155810 con 0x7f21b40c5840 2024-03-16T19:31:00.786 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:00.787 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:00.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.806+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f21b4061930 0x7f21b4061d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.806+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b4156250 con 0x7f21b4061930 2024-03-16T19:31:00.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.806+0000 7f21b2d9d640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f21b4061930 0x7f21b4061d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.807+0000 7f21b2d9d640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f21b4061930 0x7f21b4061d70 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.807+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21b4156250 con 0x7f21b4061930 2024-03-16T19:31:00.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.822+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b4062580 con 0x7f21b4061930 2024-03-16T19:31:00.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.823+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21b4062580 con 0x7f21b4061930 2024-03-16T19:31:00.804 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:00.804 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:00.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.824+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f21b402c2f0 0x7f21b402c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.824+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b40626f0 con 0x7f21b402c2f0 2024-03-16T19:31:00.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.824+0000 7f21b259c640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f21b402c2f0 0x7f21b402c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.824+0000 7f21b259c640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f21b402c2f0 0x7f21b402c750 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.825+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21b40626f0 con 0x7f21b402c2f0 2024-03-16T19:31:00.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.839+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b40628f0 con 0x7f21b402c2f0 2024-03-16T19:31:00.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.841+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21b40628f0 con 0x7f21b402c2f0 2024-03-16T19:31:00.822 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:00.822 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:00.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.842+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f21b4204ca0 0x7f21b4207090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.842+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b4062860 con 0x7f21b4204ca0 2024-03-16T19:31:00.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.842+0000 7f21b1d9b640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f21b4204ca0 0x7f21b4207090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.843+0000 7f21b1d9b640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f21b4204ca0 0x7f21b4207090 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.843+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21b4062860 con 0x7f21b4204ca0 2024-03-16T19:31:00.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.858+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b41539f0 con 0x7f21b4204ca0 2024-03-16T19:31:00.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.859+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21b41539f0 con 0x7f21b4204ca0 2024-03-16T19:31:00.840 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:00.840 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:00.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.860+0000 7f21b8fd7640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f21b4207ae0 0x7f21b4209ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.860+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b402d2a0 con 0x7f21b4207ae0 2024-03-16T19:31:00.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.860+0000 7f21b2d9d640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f21b4207ae0 0x7f21b4209ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.860+0000 7f21b2d9d640 1 --2- 172.21.15.172:0/3884439936 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f21b4207ae0 0x7f21b4209ed0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.861+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21b402d2a0 con 0x7f21b4207ae0 2024-03-16T19:31:00.845 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:00.845 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:00.846 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:00.846 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:00.854 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:00.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.876+0000 7f21b8fd7640 1 -- 172.21.15.172:0/3884439936 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b402d270 con 0x7f21b4207ae0 2024-03-16T19:31:00.857 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:00.877+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:00.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.878+0000 7f21a2ffd640 1 -- 172.21.15.172:0/3884439936 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f21b402d270 con 0x7f21b4207ae0 2024-03-16T19:31:00.859 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:00.859 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:00.863 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:00.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.911+0000 7fb4ac809640 1 Processor -- start 2024-03-16T19:31:00.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.911+0000 7fb4ac809640 1 -- start start 2024-03-16T19:31:00.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4ac809640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb4a80b5f10 0x7fb4a806b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4ac809640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb4a80caf40 0x7fb4a806bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4ac809640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a806b110 con 0x7fb4a8154e50 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4ac809640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a815b8d0 con 0x7fb4a80b5f10 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4ac809640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a815ad50 con 0x7fb4a80caf40 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4a659c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fb4a8154e50 0x7fb4a80cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:41500/0 (socket says 172.21.15.172:41500) 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4a659c640 1 -- 172.21.15.172:0/921683087 learned_addr learned my addr 172.21.15.172:0/921683087 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4a6d9d640 1 --2- 172.21.15.172:0/921683087 >> v2:172.21.15.174:3300/0 conn(0x7fb4a80caf40 0x7fb4a806bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4a5d9b640 1 --2- 172.21.15.172:0/921683087 >> v2:172.21.15.172:3300/0 conn(0x7fb4a80b5f10 0x7fb4a806b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4a6d9d640 1 -- 172.21.15.172:0/921683087 >> v2:172.21.15.172:3300/0 conn(0x7fb4a80b5f10 msgr2=0x7fb4a806b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4a6d9d640 1 --2- 172.21.15.172:0/921683087 >> v2:172.21.15.172:3300/0 conn(0x7fb4a80b5f10 0x7fb4a806b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4a6d9d640 1 -- 172.21.15.172:0/921683087 >> v1:172.21.15.172:6789/0 conn(0x7fb4a8154e50 legacy=0x7fb4a80cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:00.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.912+0000 7fb4a6d9d640 1 -- 172.21.15.172:0/921683087 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a815b030 con 0x7fb4a80caf40 2024-03-16T19:31:00.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.914+0000 7fb4a6d9d640 1 --2- 172.21.15.172:0/921683087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80caf40 0x7fb4a806bae0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8075200 tx=0x7fb4a002bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34a5d70214f6e5e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.914+0000 7fb4a559a640 1 -- 172.21.15.172:0/921683087 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a002cd20 con 0x7fb4a80caf40 2024-03-16T19:31:00.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.914+0000 7fb4a559a640 1 -- 172.21.15.172:0/921683087 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a002ce80 con 0x7fb4a80caf40 2024-03-16T19:31:00.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.914+0000 7fb4a559a640 1 -- 172.21.15.172:0/921683087 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0034660 con 0x7fb4a80caf40 2024-03-16T19:31:00.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.915+0000 7fb4ac809640 1 -- 172.21.15.172:0/921683087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80caf40 msgr2=0x7fb4a806bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:00.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.915+0000 7fb4ac809640 1 --2- 172.21.15.172:0/921683087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80caf40 0x7fb4a806bae0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8075200 tx=0x7fb4a002bf70 comp rx=0 tx=0).stop 2024-03-16T19:31:00.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.915+0000 7fb4ac809640 1 -- 172.21.15.172:0/921683087 shutdown_connections 2024-03-16T19:31:00.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.915+0000 7fb4ac809640 1 --2- 172.21.15.172:0/921683087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80caf40 0x7fb4a806bae0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.915+0000 7fb4ac809640 1 --2- 172.21.15.172:0/921683087 >> v2:172.21.15.172:3300/0 conn(0x7fb4a80b5f10 0x7fb4a806b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.915+0000 7fb4ac809640 1 -- 172.21.15.172:0/921683087 >> 172.21.15.172:0/921683087 conn(0x7fb4a8066c20 msgr2=0x7fb4a8067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:00.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.915+0000 7fb4ac809640 1 -- 172.21.15.172:0/921683087 shutdown_connections 2024-03-16T19:31:00.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.915+0000 7fb4ac809640 1 -- 172.21.15.172:0/921683087 wait complete. 2024-03-16T19:31:00.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.916+0000 7fb4ac809640 1 Processor -- start 2024-03-16T19:31:00.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.916+0000 7fb4ac809640 1 -- start start 2024-03-16T19:31:00.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4ac809640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80b5f10 0x7fb4a81f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4ac809640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a80caf40 0x7fb4a8157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4a5d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a80caf40 0x7fb4a8157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4a5d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a80caf40 0x7fb4a8157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57100/0 (socket says 172.21.15.172:57100) 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4ac809640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4a8154e50 0x7fb4a814d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4a659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80b5f10 0x7fb4a81f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4a5d9b640 1 -- 172.21.15.172:0/3441749059 learned_addr learned my addr 172.21.15.172:0/3441749059 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81f0650 con 0x7fb4a80caf40 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a80c6e20 con 0x7fb4a80b5f10 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81f0bd0 con 0x7fb4a8154e50 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.917+0000 7fb4a6d9d640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4a8154e50 0x7fb4a814d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb4a659c640 1 -- 172.21.15.172:0/3441749059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4a8154e50 msgr2=0x7fb4a814d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb4a659c640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4a8154e50 0x7fb4a814d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb4a659c640 1 -- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a80caf40 msgr2=0x7fb4a8157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb4a659c640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a80caf40 0x7fb4a8157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb4a659c640 1 -- 172.21.15.172:0/3441749059 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a0009a70 con 0x7fb4a80b5f10 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb4a5d9b640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a80caf40 0x7fb4a8157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb4a659c640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80b5f10 0x7fb4a81f1060 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb49800c440 tx=0x7fb49800c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb496ffd640 1 -- 172.21.15.172:0/3441749059 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498005ce0 con 0x7fb4a80b5f10 2024-03-16T19:31:00.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb496ffd640 1 -- 172.21.15.172:0/3441749059 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb498005e40 con 0x7fb4a80b5f10 2024-03-16T19:31:00.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb496ffd640 1 -- 172.21.15.172:0/3441749059 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4980072d0 con 0x7fb4a80b5f10 2024-03-16T19:31:00.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a815ab20 con 0x7fb4a80b5f10 2024-03-16T19:31:00.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.918+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a81f4800 con 0x7fb4a80b5f10 2024-03-16T19:31:00.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.919+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a80c9ab0 con 0x7fb4a80b5f10 2024-03-16T19:31:00.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.920+0000 7fb496ffd640 1 -- 172.21.15.172:0/3441749059 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb498007430 con 0x7fb4a80b5f10 2024-03-16T19:31:00.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.920+0000 7fb496ffd640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb48806b3b0 0x7fb48806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:00.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.921+0000 7fb496ffd640 1 -- 172.21.15.172:0/3441749059 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fb498089020 con 0x7fb4a80b5f10 2024-03-16T19:31:00.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.921+0000 7fb4a5d9b640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb48806b3b0 0x7fb48806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:00.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.921+0000 7fb4a5d9b640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb48806b3b0 0x7fb48806d870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80b5ad0 tx=0x7fb49c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:00.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:00.925+0000 7fb496ffd640 1 -- 172.21.15.172:0/3441749059 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb498053cb0 con 0x7fb4a80b5f10 2024-03-16T19:31:00.958 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:00.978+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:01.004 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:01.004 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:01.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:01.004 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:01.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.004 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.665768565999997, 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:01.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:31:01.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.009 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.665744392999997, 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:01.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:31:01.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:31:01.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:01.015 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.665738566000002, 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.665734654000005, 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:31:01.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.665728672, 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:31:01.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.665725006000002, 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.031 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:31:01.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:31:01.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:01.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.665714346000001, 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:31:01.036 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:31:01.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:31:01.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.665710457000003, 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:01.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.043 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.662193766000001, 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.662123331000004, 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.047 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:31:01.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:31:01.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:31:01.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.661740984000005, 2024-03-16T19:31:01.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:31:01.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.659974700999996, 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:31:01.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.059 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.658161778999997, 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.657306030000001, 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:31:01.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:31:01.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 84.651445264000003, 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:31:01.068 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 81.218894055999996, 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.077 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 79.651400350000003, 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:31:01.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 78.143788099000005, 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:31:01.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 41.646924874, 2024-03-16T19:31:01.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:01.092 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.646388958000003, 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:01.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:31:01.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:31:01.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:31:01.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.097 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.097 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.097 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:01.098 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:01.099 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:01.118+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:01.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.213+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fb4a81f4aa0 con 0x7fb4a80b5f10 2024-03-16T19:31:01.198 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:01.218+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:01.242 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:01.242 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:01.242 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:01.242 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:01.254 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:01.299 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:01.319+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:01.387 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:01.387 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:01.387 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:01.387 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:01.387 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:01.399 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:01.400 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:01.420+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:01.501 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:01.521+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:01.507 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:01.507 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:01.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:01.507 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:01.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.507 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:01.507 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:01.507 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.296250711, 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:31:01.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 104.919555164, 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.513 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:31:01.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:31:01.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 104.14357871999999, 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.517 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:31:01.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 104.143494307, 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:31:01.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.528 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 104.143486951, 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:31:01.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:31:01.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:31:01.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.950947759, 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.536 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:01.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.95094195199999, 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.95082381899999, 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:31:01.547 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.552 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.950728622, 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:31:01.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:31:01.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:31:01.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:31:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.950724611, 2024-03-16T19:31:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:31:01.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:31:01.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.531792737, 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.564 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.569 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.53174336799999, 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:01.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:31:01.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.53166453599999, 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.575 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:31:01.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.580 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.531661325, 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:31:01.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:31:01.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:31:01.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:31:01.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.531658372, 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:31:01.586 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:31:01.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:31:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.531543398, 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:31:01.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:01.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:01.598 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.531540046, 2024-03-16T19:31:01.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:31:01.598 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.598 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.598 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.598 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:31:01.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:31:01.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:31:01.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.53153693, 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:31:01.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:01.609 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.531476461, 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:31:01.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:31:01.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:31:01.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:31:01.614 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 102.92279641099999, 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:31:01.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:01.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:31:01.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:31:01.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:31:01.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:01.620 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:01.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:01.622 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:01.641+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:01.722 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:01.742+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:01.738 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:01.738 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:01.738 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:01.738 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:01.738 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.756+0000 7fb496ffd640 1 -- 172.21.15.172:0/3441749059 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v64) v1 ==== 91+0+0 (secure 0 0 0) 0x7fb498057370 con 0x7fb4a80b5f10 2024-03-16T19:31:01.738 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb48806b3b0 msgr2=0x7fb48806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb48806b3b0 0x7fb48806d870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80b5ad0 tx=0x7fb49c009190 comp rx=0 tx=0).stop 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80b5f10 msgr2=0x7fb4a81f1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80b5f10 0x7fb4a81f1060 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb49800c440 tx=0x7fb49800c900 comp rx=0 tx=0).stop 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 shutdown_connections 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb48806b3b0 0x7fb48806d870 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4a8154e50 0x7fb4a814d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4a80caf40 0x7fb4a8157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 --2- 172.21.15.172:0/3441749059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4a80b5f10 0x7fb4a81f1060 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 >> 172.21.15.172:0/3441749059 conn(0x7fb4a8066c20 msgr2=0x7fb4a806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:01.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 shutdown_connections 2024-03-16T19:31:01.742 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:01.760+0000 7fb4ac809640 1 -- 172.21.15.172:0/3441749059 wait complete. 2024-03-16T19:31:01.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:01.823 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:01.843+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:01.860 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:01.860 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:01.860 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:01.861 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:01.861 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:01.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:01.923 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:01.943+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:02.024 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:02.044+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.508451638, 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:31:02.084 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:31:02.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.088 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.718225467, 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:31:02.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:31:02.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.092 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.717465232, 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:02.096 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.716651277, 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.716011352, 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.715353255, 2024-03-16T19:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.71482068, 2024-03-16T19:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.656648777, 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:31:02.112 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.65604658300001, 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:31:02.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.655403638, 2024-03-16T19:31:02.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:31:02.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.654899677, 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 103.496753411, 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 103.49622327500001, 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.133 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:02.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.137 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 90.744254015999999, 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:31:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.143 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 90.744229480000001, 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:31:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:31:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:31:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 90.744224950000003, 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:31:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:31:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:31:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.155 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 90.744215138000001, 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:31:02.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 90.744211285000006, 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 90.744207969000001, 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:31:02.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:31:02.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 90.743012386000004, 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:31:02.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.173 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:02.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:02.176 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:02.194+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:02.276 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:02.296+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:02.327 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:02.327 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:02.327 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:02.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:02.378 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:02.398+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:02.478 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:02.498+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:02.489 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:02.489 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:02.490 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:02.490 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:02.490 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:02.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:02.510 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-16T19:31:02.510 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:31:02.510 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-16T19:31:02.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-03-16T19:31:02.586 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:02.606+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:02.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.666+0000 7fc982671640 1 Processor -- start 2024-03-16T19:31:02.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.666+0000 7fc982671640 1 -- start start 2024-03-16T19:31:02.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.667+0000 7fc982671640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc97c0b5f10 0x7fc97c06b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.667+0000 7fc982671640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc97c0caf40 0x7fc97c06bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.667+0000 7fc982671640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c06b110 con 0x7fc97c154e50 2024-03-16T19:31:02.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.667+0000 7fc982671640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc97c15b8d0 con 0x7fc97c0caf40 2024-03-16T19:31:02.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.667+0000 7fc982671640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc97c15ad50 con 0x7fc97c0b5f10 2024-03-16T19:31:02.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.667+0000 7fc97bfff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc97c154e50 0x7fc97c0cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:41514/0 (socket says 172.21.15.172:41514) 2024-03-16T19:31:02.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.667+0000 7fc97bfff640 1 -- 172.21.15.172:0/1443680150 learned_addr learned my addr 172.21.15.172:0/1443680150 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:02.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.667+0000 7fc980c0e640 1 --2- 172.21.15.172:0/1443680150 >> v2:172.21.15.172:3300/0 conn(0x7fc97c0caf40 0x7fc97c06bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.667+0000 7fc97b7fe640 1 --2- 172.21.15.172:0/1443680150 >> v2:172.21.15.174:3300/0 conn(0x7fc97c0b5f10 0x7fc97c06b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.668+0000 7fc980c0e640 1 -- 172.21.15.172:0/1443680150 >> v2:172.21.15.174:3300/0 conn(0x7fc97c0b5f10 msgr2=0x7fc97c06b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:02.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.668+0000 7fc980c0e640 1 --2- 172.21.15.172:0/1443680150 >> v2:172.21.15.174:3300/0 conn(0x7fc97c0b5f10 0x7fc97c06b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.668+0000 7fc980c0e640 1 -- 172.21.15.172:0/1443680150 >> v1:172.21.15.172:6789/0 conn(0x7fc97c154e50 legacy=0x7fc97c0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:02.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.668+0000 7fc980c0e640 1 -- 172.21.15.172:0/1443680150 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c15b030 con 0x7fc97c0caf40 2024-03-16T19:31:02.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.668+0000 7fc97affd640 1 -- 172.21.15.172:0/1443680150 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225933532 0 0) 0x7fc97c06b110 con 0x7fc97c154e50 2024-03-16T19:31:02.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.668+0000 7fc980c0e640 1 --2- 172.21.15.172:0/1443680150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c0caf40 0x7fc97c06bae0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc97c074e40 tx=0x7fc97402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238d4e6e15fb9d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:02.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.669+0000 7fc97affd640 1 -- 172.21.15.172:0/1443680150 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97402cd00 con 0x7fc97c0caf40 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.669+0000 7fc97affd640 1 -- 172.21.15.172:0/1443680150 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97402ce60 con 0x7fc97c0caf40 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.669+0000 7fc97affd640 1 -- 172.21.15.172:0/1443680150 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974034660 con 0x7fc97c0caf40 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.669+0000 7fc982671640 1 -- 172.21.15.172:0/1443680150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c0caf40 msgr2=0x7fc97c06bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.669+0000 7fc982671640 1 --2- 172.21.15.172:0/1443680150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c0caf40 0x7fc97c06bae0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc97c074e40 tx=0x7fc97402bf50 comp rx=0 tx=0).stop 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.669+0000 7fc982671640 1 -- 172.21.15.172:0/1443680150 shutdown_connections 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.669+0000 7fc982671640 1 --2- 172.21.15.172:0/1443680150 >> v2:172.21.15.174:3300/0 conn(0x7fc97c0b5f10 0x7fc97c06b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.669+0000 7fc982671640 1 --2- 172.21.15.172:0/1443680150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c0caf40 0x7fc97c06bae0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.669+0000 7fc982671640 1 -- 172.21.15.172:0/1443680150 >> 172.21.15.172:0/1443680150 conn(0x7fc97c066c20 msgr2=0x7fc97c067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.670+0000 7fc982671640 1 -- 172.21.15.172:0/1443680150 shutdown_connections 2024-03-16T19:31:02.651 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.670+0000 7fc982671640 1 -- 172.21.15.172:0/1443680150 wait complete. 2024-03-16T19:31:02.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.671+0000 7fc982671640 1 Processor -- start 2024-03-16T19:31:02.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.671+0000 7fc982671640 1 -- start start 2024-03-16T19:31:02.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc982671640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c0b5f10 0x7fc97c1599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc982671640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c0caf40 0x7fc97c147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc982671640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c154e50 0x7fc97c152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc982671640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1512f0 con 0x7fc97c154e50 2024-03-16T19:31:02.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc982671640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1536c0 con 0x7fc97c0b5f10 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc982671640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c153980 con 0x7fc97c0caf40 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc980c0e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c154e50 0x7fc97c152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc980c0e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c154e50 0x7fc97c152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57112/0 (socket says 172.21.15.172:57112) 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc980c0e640 1 -- 172.21.15.172:0/1236231942 learned_addr learned my addr 172.21.15.172:0/1236231942 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc97b7fe640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c0caf40 0x7fc97c147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc97bfff640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c0b5f10 0x7fc97c1599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc97b7fe640 1 -- 172.21.15.172:0/1236231942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c0b5f10 msgr2=0x7fc97c1599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc97b7fe640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c0b5f10 0x7fc97c1599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc97b7fe640 1 -- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c154e50 msgr2=0x7fc97c152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc97b7fe640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c154e50 0x7fc97c152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.672+0000 7fc97b7fe640 1 -- 172.21.15.172:0/1236231942 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc974009a70 con 0x7fc97c0caf40 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.673+0000 7fc980c0e640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c154e50 0x7fc97c152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:02.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.673+0000 7fc97b7fe640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c0caf40 0x7fc97c147870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc97000d5f0 tx=0x7fc97000dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:02.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.673+0000 7fc978ff9640 1 -- 172.21.15.172:0/1236231942 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9700125f0 con 0x7fc97c0caf40 2024-03-16T19:31:02.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.673+0000 7fc978ff9640 1 -- 172.21.15.172:0/1236231942 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc970012750 con 0x7fc97c0caf40 2024-03-16T19:31:02.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.673+0000 7fc978ff9640 1 -- 172.21.15.172:0/1236231942 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970012b00 con 0x7fc97c0caf40 2024-03-16T19:31:02.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.673+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c15ab20 con 0x7fc97c0caf40 2024-03-16T19:31:02.656 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.673+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c1f0660 con 0x7fc97c0caf40 2024-03-16T19:31:02.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.674+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97c0c9ab0 con 0x7fc97c0caf40 2024-03-16T19:31:02.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.675+0000 7fc978ff9640 1 -- 172.21.15.172:0/1236231942 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc970012c60 con 0x7fc97c0caf40 2024-03-16T19:31:02.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.675+0000 7fc978ff9640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc96806b3e0 0x7fc96806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.675+0000 7fc978ff9640 1 -- 172.21.15.172:0/1236231942 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5828+0+0 (secure 0 0 0) 0x7fc970089fd0 con 0x7fc97c0caf40 2024-03-16T19:31:02.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.676+0000 7fc97bfff640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc96806b3e0 0x7fc96806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.676+0000 7fc97bfff640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc96806b3e0 0x7fc96806d8a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc96c007fd0 tx=0x7fc96c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:02.659 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:02.659 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:02.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:02.659 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.320267829000002, 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:31:02.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:31:02.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:31:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.320250715, 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:31:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:31:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.320243434999995, 2024-03-16T19:31:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:31:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:31:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.320229568000002, 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:31:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:31:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:31:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:31:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.319556732999999, 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:31:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:31:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.317258483000003, 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.685 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.317213382000006, 2024-03-16T19:31:02.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:02.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:31:02.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.317185327000004, 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.317163913000002, 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.317141562000003, 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:31:02.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:31:02.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.317121162999996, 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.706 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.710 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.317015170999994, 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:31:02.714 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.315223459999999, 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:02.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:31:02.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.718 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.315175959000001, 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:02.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.722 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.314936708000005, 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:02.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:31:02.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 86.306438563, 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:31:02.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:31:02.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:02.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:31:02.732 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 86.305098686999997, 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:02.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.302107704999997, 2024-03-16T19:31:02.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:31:02.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.301476903000001, 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:02.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:31:02.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:31:02.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:31:02.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:31:02.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:47 2.6 e40/18 ondisk, result = 0)", 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.299994494000003, 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017677, 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:02.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377611+0000", 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9599999999999998e-07 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377618+0000", 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9990000000000002e-06 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377626+0000", 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4249999999999998e-06 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377632+0000", 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377707+0000", 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4506999999999997e-05 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377714+0000", 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5819999999999998e-06 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377787+0000", 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3232000000000006e-05 2024-03-16T19:31:02.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.751 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:02.751 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:02.751 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:02.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.680+0000 7fc978ff9640 1 -- 172.21.15.172:0/1236231942 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc970054c70 con 0x7fc97c0caf40 2024-03-16T19:31:02.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:02.753 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:02.772+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:02.853 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:02.873+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:02.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:31:02.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.884+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd out", "ids": ["3"]} v 0) v1 -- 0x7fc97c0cea70 con 0x7fc97c0caf40 2024-03-16T19:31:02.865 INFO:tasks.ceph.mon.a.smithi172.stderr:2024-03-16T19:31:02.885+0000 7f2459211640 -1 mon.a@0(leader).osd e64 definitely_dead 0 2024-03-16T19:31:02.914 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:02.914 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:02.914 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:02.914 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:02.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:02.954 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:02.974+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:02.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.986+0000 7f68f0779640 1 Processor -- start 2024-03-16T19:31:02.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.986+0000 7f68f0779640 1 -- start start 2024-03-16T19:31:02.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68f0779640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e8154ed0 0x7f68e815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68f0779640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e80b4270 con 0x7f68e80b3630 2024-03-16T19:31:02.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68ee515640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e8154ed0 0x7f68e815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68edd14640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f68e80b3630 0x7f68e80cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:41530/0 (socket says 172.21.15.172:41530) 2024-03-16T19:31:02.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68edd14640 1 -- 172.21.15.172:0/3546466671 learned_addr learned my addr 172.21.15.172:0/3546466671 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:02.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68f0779640 1 -- 172.21.15.172:0/3546466671 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e81552b0 con 0x7f68e80b4700 2024-03-16T19:31:02.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68f0779640 1 -- 172.21.15.172:0/3546466671 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815b4b0 con 0x7f68e8154ed0 2024-03-16T19:31:02.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316088358 0 0) 0x7f68e80b4270 con 0x7f68e80b3630 2024-03-16T19:31:02.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d4003520 con 0x7f68e80b3630 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836827873 0 0) 0x7f68e81552b0 con 0x7f68e80b4700 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.987+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68e80b4270 con 0x7f68e80b4700 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 260103879 0 0) 0x7f68d4003520 con 0x7f68e80b3630 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e8154ed0 msgr2=0x7f68e815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e8154ed0 0x7f68e815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 >> v1:172.21.15.174:6789/0 conn(0x7f68e80b4700 legacy=0x7f68e81654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e815b790 con 0x7f68e80b3630 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f68d8001670 con 0x7f68e80b3630 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 >> v1:172.21.15.172:6789/0 conn(0x7f68e80b3630 legacy=0x7f68e80cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4004bc0 0x7f68d4004fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e8154ed0 0x7f68d4003760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68d4003ca0 0x7f68d4004120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e00015b0 con 0x7f68d4004bc0 2024-03-16T19:31:02.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815b4b0 con 0x7f68d4003ca0 2024-03-16T19:31:02.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d400bab0 con 0x7f68e8154ed0 2024-03-16T19:31:02.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68eed16640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68d4003ca0 0x7f68d4004120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68edd14640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e8154ed0 0x7f68d4003760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68ee515640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4004bc0 0x7f68d4004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68eed16640 1 -- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e8154ed0 msgr2=0x7f68d4003760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:02.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68eed16640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e8154ed0 0x7f68d4003760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68eed16640 1 -- 172.21.15.172:0/3546466671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4004bc0 msgr2=0x7f68d4004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:02.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68eed16640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4004bc0 0x7f68d4004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.988+0000 7f68eed16640 1 -- 172.21.15.172:0/3546466671 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d4003520 con 0x7f68d4003ca0 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.989+0000 7f68eed16640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68d4003ca0 0x7f68d4004120 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f68e000c320 tx=0x7f68d40057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27e00ac3ac45d336 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.989+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e002cc20 con 0x7f68d4003ca0 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.989+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e002cda0 con 0x7f68d4003ca0 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.989+0000 7f68ed513640 1 -- 172.21.15.172:0/3546466671 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0003bb0 con 0x7f68d4003ca0 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.990+0000 7f68f0779640 1 -- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68d4003ca0 msgr2=0x7f68d4004120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.990+0000 7f68f0779640 1 --2- 172.21.15.172:0/3546466671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68d4003ca0 0x7f68d4004120 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f68e000c320 tx=0x7f68d40057f0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.990+0000 7f68eed16640 1 -- 172.21.15.172:0/3546466671 reap_dead start 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.991+0000 7f68f0779640 1 -- 172.21.15.172:0/3546466671 shutdown_connections 2024-03-16T19:31:02.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.991+0000 7f68f0779640 1 -- 172.21.15.172:0/3546466671 >> 172.21.15.172:0/3546466671 conn(0x7f68e806bf60 msgr2=0x7f68e806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:02.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.993+0000 7f68f0779640 1 -- 172.21.15.172:0/3546466671 shutdown_connections 2024-03-16T19:31:02.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.993+0000 7f68f0779640 1 -- 172.21.15.172:0/3546466671 wait complete. 2024-03-16T19:31:02.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.994+0000 7f68f0779640 1 Processor -- start 2024-03-16T19:31:02.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.994+0000 7f68f0779640 1 -- start start 2024-03-16T19:31:02.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68f0779640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4003ca0 0x7f68e815ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68f0779640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e815e050 0x7f68e81471c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68f0779640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68e80c9a30 0x7f68e80c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68f0779640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e81f0ed0 con 0x7f68d4003ca0 2024-03-16T19:31:02.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68f0779640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815bd00 con 0x7f68e80c9a30 2024-03-16T19:31:02.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68f0779640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815bfe0 con 0x7f68e815e050 2024-03-16T19:31:02.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68ee515640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4003ca0 0x7f68e815ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68ee515640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4003ca0 0x7f68e815ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57130/0 (socket says 172.21.15.172:57130) 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68ee515640 1 -- 172.21.15.172:0/3199311659 learned_addr learned my addr 172.21.15.172:0/3199311659 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68eed16640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68e80c9a30 0x7f68e80c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68edd14640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e815e050 0x7f68e81471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68eed16640 1 -- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e815e050 msgr2=0x7f68e81471c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68eed16640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68e815e050 0x7f68e81471c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68eed16640 1 -- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4003ca0 msgr2=0x7f68e815ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.995+0000 7f68eed16640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4003ca0 0x7f68e815ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.996+0000 7f68eed16640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e81f13e0 con 0x7f68e80c9a30 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.996+0000 7f68ee515640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f68d4003ca0 0x7f68e815ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.996+0000 7f68eed16640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68e80c9a30 0x7f68e80c9eb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f68e0031000 tx=0x7f68e0001ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.996+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e003c070 con 0x7f68e80c9a30 2024-03-16T19:31:02.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.996+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e0003db0 con 0x7f68e80c9a30 2024-03-16T19:31:02.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.996+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0006d10 con 0x7f68e80c9a30 2024-03-16T19:31:02.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.996+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e81f1100 con 0x7f68e80c9a30 2024-03-16T19:31:02.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.996+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e815dbf0 con 0x7f68e80c9a30 2024-03-16T19:31:02.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.997+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f68e80b3ec0 con 0x7f68e80c9a30 2024-03-16T19:31:02.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.998+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f68e0032070 con 0x7f68e80c9a30 2024-03-16T19:31:02.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.998+0000 7f68deffd640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f68cc06b3e0 0x7f68cc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.998+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5828+0+0 (secure 0 0 0) 0x7f68e00aab40 con 0x7f68e80c9a30 2024-03-16T19:31:02.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.999+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v64) v1 ==== 54+0+16 (secure 0 0 0) 0x7f68e00b1070 con 0x7f68e80c9a30 2024-03-16T19:31:02.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.999+0000 7f68ee515640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f68cc06b3e0 0x7f68cc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.999+0000 7f68f0779640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f68e80c3560 0x7f68e815c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:02.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.999+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e815be40 con 0x7f68e80c3560 2024-03-16T19:31:02.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.999+0000 7f68edd14640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f68e80c3560 0x7f68e815c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:02.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:02.999+0000 7f68edd14640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f68e80c3560 0x7f68e815c470 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:02.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.000+0000 7f68ee515640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f68cc06b3e0 0x7f68cc06d8a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f68e4006d40 tx=0x7f68e4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:02.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.000+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f68d8007d30 con 0x7f68e80c3560 2024-03-16T19:31:02.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.016+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f68e815ba70 con 0x7f68e80c3560 2024-03-16T19:31:02.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.018+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f68d80079e0 con 0x7f68e80c3560 2024-03-16T19:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:03.000 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:03.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.019+0000 7f68f0779640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f68e81f39c0 0x7f68e81f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.019+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e8068d20 con 0x7f68e81f39c0 2024-03-16T19:31:03.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.020+0000 7f68ee515640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f68e81f39c0 0x7f68e81f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.020+0000 7f68ee515640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f68e81f39c0 0x7f68e81f3df0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:03.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.021+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f68e8068d20 con 0x7f68e81f39c0 2024-03-16T19:31:03.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.036+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f68e81f4580 con 0x7f68e81f39c0 2024-03-16T19:31:03.019 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.039+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f68e81f4580 con 0x7f68e81f39c0 2024-03-16T19:31:03.020 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.039+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=65}) v3 -- 0x7f68e80681c0 con 0x7f68e80c9a30 2024-03-16T19:31:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.039+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68e815c120 con 0x7f68e80c9a30 2024-03-16T19:31:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.040+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68e0006e70 con 0x7f68e80c9a30 2024-03-16T19:31:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.040+0000 7f68f0779640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f68e8062200 0x7f68e80c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.040+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e80cbbe0 con 0x7f68e8062200 2024-03-16T19:31:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.040+0000 7f68edd14640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f68e8062200 0x7f68e80c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.041+0000 7f68edd14640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f68e8062200 0x7f68e80c7860 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:03.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.042+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f68e80cbbe0 con 0x7f68e8062200 2024-03-16T19:31:03.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.056+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f68e80628e0 con 0x7f68e8062200 2024-03-16T19:31:03.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.057+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f68e80628e0 con 0x7f68e8062200 2024-03-16T19:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:03.038 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:03.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.058+0000 7f68f0779640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f68e802bde0 0x7f68e802c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.058+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e8147700 con 0x7f68e802bde0 2024-03-16T19:31:03.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.058+0000 7f68eed16640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f68e802bde0 0x7f68e802c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.058+0000 7f68eed16640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f68e802bde0 0x7f68e802c220 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:03.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.059+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f68e8147700 con 0x7f68e802bde0 2024-03-16T19:31:03.053 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.073+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f68e8159a80 con 0x7f68e802bde0 2024-03-16T19:31:03.054 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:03.074+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:03.055 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.075+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f68e8159a80 con 0x7f68e802bde0 2024-03-16T19:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:03.056 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:03.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.075+0000 7f68f0779640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f68e802cc70 0x7f68e8204d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.075+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e81594c0 con 0x7f68e802cc70 2024-03-16T19:31:03.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.075+0000 7f68ee515640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f68e802cc70 0x7f68e8204d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.056 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.076+0000 7f68ee515640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f68e802cc70 0x7f68e8204d40 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:03.057 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.077+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f68e81594c0 con 0x7f68e802cc70 2024-03-16T19:31:03.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.091+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f68e81594c0 con 0x7f68e802cc70 2024-03-16T19:31:03.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.092+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f68e81594c0 con 0x7f68e802cc70 2024-03-16T19:31:03.073 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:03.073 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:03.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.092+0000 7f68f0779640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f68e802d0b0 0x7f68e82077a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.092+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e820a570 con 0x7f68e802d0b0 2024-03-16T19:31:03.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.093+0000 7f68edd14640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f68e802d0b0 0x7f68e82077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.093+0000 7f68edd14640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f68e802d0b0 0x7f68e82077a0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:03.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.094+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f68e820a570 con 0x7f68e802d0b0 2024-03-16T19:31:03.076 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:03.076 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:03.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:03.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.112+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f68e8207ce0 con 0x7f68e802d0b0 2024-03-16T19:31:03.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.114+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f68e8207ce0 con 0x7f68e802d0b0 2024-03-16T19:31:03.094 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:03.095 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:03.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.114+0000 7f68f0779640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f68e8208360 0x7f68e820a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.114+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e820d630 con 0x7f68e8208360 2024-03-16T19:31:03.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.115+0000 7f68eed16640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f68e8208360 0x7f68e820a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.115+0000 7f68eed16640 1 --2- 172.21.15.172:0/3199311659 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f68e8208360 0x7f68e820a750 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:03.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.116+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f68e820d630 con 0x7f68e8208360 2024-03-16T19:31:03.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.143+0000 7f68f0779640 1 -- 172.21.15.172:0/3199311659 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f68e820ad10 con 0x7f68e8208360 2024-03-16T19:31:03.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.145+0000 7f68deffd640 1 -- 172.21.15.172:0/3199311659 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f68e820ad10 con 0x7f68e8208360 2024-03-16T19:31:03.125 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:03.125 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:03.129 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:03.155 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:03.175+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:03.202 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:03.202 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:03.211 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:03.256 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:03.276+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:03.319 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:03.319 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:03.328 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:03.357 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:03.377+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:03.438 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:03.438 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:03.447 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:03.458 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:03.478+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:03.558 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:03.558 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:03.558 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:03.578+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:03.567 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:03.659 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:03.679+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:03.673 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:03.673 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:03.682 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:03.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.772+0000 7fc978ff9640 1 -- 172.21.15.172:0/1236231942 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["3"]}]=0 marked out osd.3. v65) v1 ==== 87+0+0 (secure 0 0 0) 0x7fc970058330 con 0x7fc97c0caf40 2024-03-16T19:31:03.756 INFO:teuthology.orchestra.run.smithi172.stderr:marked out osd.3. 2024-03-16T19:31:03.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:31:03.760 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:03.780+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:03.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc96806b3e0 msgr2=0x7fc96806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:03.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc96806b3e0 0x7fc96806d8a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc96c007fd0 tx=0x7fc96c007de0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c0caf40 msgr2=0x7fc97c147870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:03.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c0caf40 0x7fc97c147870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc97000d5f0 tx=0x7fc97000dab0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 shutdown_connections 2024-03-16T19:31:03.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc97c154e50 0x7fc97c152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc96806b3e0 0x7fc96806d8a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c0caf40 0x7fc97c147870 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc97000d5f0 tx=0x7fc97000dab0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 --2- 172.21.15.172:0/1236231942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c0b5f10 0x7fc97c1599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.779+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 >> 172.21.15.172:0/1236231942 conn(0x7fc97c066c20 msgr2=0x7fc97c0c74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:03.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.781+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 shutdown_connections 2024-03-16T19:31:03.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.781+0000 7fc982671640 1 -- 172.21.15.172:0/1236231942 wait complete. 2024-03-16T19:31:03.797 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:03.797 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:03.805 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:03.890 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:03.910+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:03.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.940+0000 7fcc0c607640 1 Processor -- start 2024-03-16T19:31:03.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.940+0000 7fcc0c607640 1 -- start start 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc0c607640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcc04159160 0x7fcc040b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc0c607640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcc040cb5c0 0x7fcc0406afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc0c607640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc04159540 con 0x7fcc040cbe60 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc0c607640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc04162340 con 0x7fcc04159160 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc0c607640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc040cc780 con 0x7fcc040cb5c0 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc0a3a3640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcc04159160 0x7fcc040b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc09ba2640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fcc040cbe60 0x7fcc040caeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:54872/0 (socket says 172.21.15.172:54872) 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc09ba2640 1 -- 172.21.15.172:0/3552353127 learned_addr learned my addr 172.21.15.172:0/3552353127 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc0aba4640 1 --2- 172.21.15.172:0/3552353127 >> v2:172.21.15.174:3300/0 conn(0x7fcc040cb5c0 0x7fcc0406afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc093a1640 1 -- 172.21.15.172:0/3552353127 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483263544 0 0) 0x7fcc04159540 con 0x7fcc040cbe60 2024-03-16T19:31:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.941+0000 7fcc093a1640 1 -- 172.21.15.172:0/3552353127 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf0003520 con 0x7fcc040cbe60 2024-03-16T19:31:03.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.942+0000 7fcc0aba4640 1 -- 172.21.15.172:0/3552353127 >> v2:172.21.15.172:3300/0 conn(0x7fcc04159160 msgr2=0x7fcc040b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:03.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.942+0000 7fcc0aba4640 1 --2- 172.21.15.172:0/3552353127 >> v2:172.21.15.172:3300/0 conn(0x7fcc04159160 0x7fcc040b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.942+0000 7fcc0aba4640 1 -- 172.21.15.172:0/3552353127 >> v1:172.21.15.174:6789/0 conn(0x7fcc040cbe60 legacy=0x7fcc040caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:03.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.942+0000 7fcc0aba4640 1 -- 172.21.15.172:0/3552353127 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc041625e0 con 0x7fcc040cb5c0 2024-03-16T19:31:03.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.942+0000 7fcc0a3a3640 1 --2- 172.21.15.172:0/3552353127 >> v2:172.21.15.172:3300/0 conn(0x7fcc04159160 0x7fcc040b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:03.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.942+0000 7fcc0aba4640 1 --2- 172.21.15.172:0/3552353127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc040cb5c0 0x7fcc0406afe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcc040c76e0 tx=0x7fcbfc02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6835c79924b0b05d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:03.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.943+0000 7fcc093a1640 1 -- 172.21.15.172:0/3552353127 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc02cd00 con 0x7fcc040cb5c0 2024-03-16T19:31:03.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.943+0000 7fcc093a1640 1 -- 172.21.15.172:0/3552353127 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc02ce60 con 0x7fcc040cb5c0 2024-03-16T19:31:03.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.943+0000 7fcc093a1640 1 -- 172.21.15.172:0/3552353127 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc034660 con 0x7fcc040cb5c0 2024-03-16T19:31:03.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.943+0000 7fcc0c607640 1 -- 172.21.15.172:0/3552353127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc040cb5c0 msgr2=0x7fcc0406afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:03.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.943+0000 7fcc0c607640 1 --2- 172.21.15.172:0/3552353127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc040cb5c0 0x7fcc0406afe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcc040c76e0 tx=0x7fcbfc02bf50 comp rx=0 tx=0).stop 2024-03-16T19:31:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.943+0000 7fcc0c607640 1 -- 172.21.15.172:0/3552353127 shutdown_connections 2024-03-16T19:31:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.943+0000 7fcc0c607640 1 --2- 172.21.15.172:0/3552353127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc040cb5c0 0x7fcc0406afe0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.943+0000 7fcc0c607640 1 --2- 172.21.15.172:0/3552353127 >> v2:172.21.15.172:3300/0 conn(0x7fcc04159160 0x7fcc040b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.943+0000 7fcc0c607640 1 -- 172.21.15.172:0/3552353127 >> 172.21.15.172:0/3552353127 conn(0x7fcc04066c20 msgr2=0x7fcc04067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.944+0000 7fcc0c607640 1 -- 172.21.15.172:0/3552353127 shutdown_connections 2024-03-16T19:31:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.944+0000 7fcc0c607640 1 -- 172.21.15.172:0/3552353127 wait complete. 2024-03-16T19:31:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.945+0000 7fcc0c607640 1 Processor -- start 2024-03-16T19:31:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.945+0000 7fcc0c607640 1 -- start start 2024-03-16T19:31:03.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.946+0000 7fcc0c607640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc040cb5c0 0x7fcc041f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0c607640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc040cbe60 0x7fcc040c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0c607640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc04159160 0x7fcc041549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0c607640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040c6ab0 con 0x7fcc04159160 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0a3a3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc040cb5c0 0x7fcc041f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0a3a3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc040cb5c0 0x7fcc041f50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:49764/0 (socket says 172.21.15.172:49764) 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0a3a3640 1 -- 172.21.15.172:0/2546849058 learned_addr learned my addr 172.21.15.172:0/2546849058 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc09ba2640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc040cbe60 0x7fcc040c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0aba4640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc04159160 0x7fcc041549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0c607640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041f5630 con 0x7fcc040cbe60 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040c4730 con 0x7fcc040cb5c0 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0aba4640 1 -- 172.21.15.172:0/2546849058 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc040cb5c0 msgr2=0x7fcc041f50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0aba4640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc040cb5c0 0x7fcc041f50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0aba4640 1 -- 172.21.15.172:0/2546849058 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc040cbe60 msgr2=0x7fcc040c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0aba4640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc040cbe60 0x7fcc040c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.947+0000 7fcc0aba4640 1 -- 172.21.15.172:0/2546849058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc00009400 con 0x7fcc04159160 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.948+0000 7fcc0aba4640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc04159160 0x7fcc041549c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcc040cc8c0 tx=0x7fcbfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.948+0000 7fcbfaffd640 1 -- 172.21.15.172:0/2546849058 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc03d600 con 0x7fcc04159160 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.948+0000 7fcbfaffd640 1 -- 172.21.15.172:0/2546849058 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc03d780 con 0x7fcc04159160 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.948+0000 7fcbfaffd640 1 -- 172.21.15.172:0/2546849058 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc03db30 con 0x7fcc04159160 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.949+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbfc009a70 con 0x7fcc04159160 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.949+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc04071d60 con 0x7fcc04159160 2024-03-16T19:31:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.950+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc04072280 con 0x7fcc04159160 2024-03-16T19:31:03.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.951+0000 7fcbfaffd640 1 -- 172.21.15.172:0/2546849058 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fcbfc03b040 con 0x7fcc04159160 2024-03-16T19:31:03.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.951+0000 7fcbfaffd640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbe806b3b0 0x7fcbe806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:03.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.951+0000 7fcbfaffd640 1 -- 172.21.15.172:0/2546849058 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fcbfc0ab990 con 0x7fcc04159160 2024-03-16T19:31:03.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.951+0000 7fcc0a3a3640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbe806b3b0 0x7fcbe806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:03.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.952+0000 7fcc0a3a3640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbe806b3b0 0x7fcbe806d870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009d80 tx=0x7fcc00009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:03.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:03.956+0000 7fcbfaffd640 1 -- 172.21.15.172:0/2546849058 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbfc076640 con 0x7fcc04159160 2024-03-16T19:31:03.953 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:03.953 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:03.961 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:03.981+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:03.962 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:04.062 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:04.082+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:04.109 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:04.109 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:04.118 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:04.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.145+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fcc04075a70 con 0x7fcc04159160 2024-03-16T19:31:04.202 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:04.222+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:04.264 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:04.284+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:04.265 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:04.265 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:04.274 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:04.365 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:04.385+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:04.424 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:04.424 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:04.433 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:04.474 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:04.494+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:04.567 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:04.587+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:04.581 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:04.581 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:04.593 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:04.675 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:04.695+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:04.736 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:04.736 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:04.745 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:04.749 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.768+0000 7fcbfaffd640 1 -- 172.21.15.172:0/2546849058 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v66) v1 ==== 101+0+0 (secure 0 0 0) 0x7fcbfc039090 con 0x7fcc04159160 2024-03-16T19:31:04.749 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:31:04.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.771+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbe806b3b0 msgr2=0x7fcbe806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:04.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.771+0000 7fcc0c607640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbe806b3b0 0x7fcbe806d870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009d80 tx=0x7fcc00009190 comp rx=0 tx=0).stop 2024-03-16T19:31:04.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.771+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc04159160 msgr2=0x7fcc041549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:04.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.771+0000 7fcc0c607640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc04159160 0x7fcc041549c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcc040cc8c0 tx=0x7fcbfc004a40 comp rx=0 tx=0).stop 2024-03-16T19:31:04.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.772+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 shutdown_connections 2024-03-16T19:31:04.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.772+0000 7fcc0c607640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc04159160 0x7fcc041549c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:04.752 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.772+0000 7fcc0c607640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc040cbe60 0x7fcc040c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:04.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.772+0000 7fcc0c607640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcbe806b3b0 0x7fcbe806d870 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009d80 tx=0x7fcc00009190 comp rx=0 tx=0).stop 2024-03-16T19:31:04.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.772+0000 7fcc0c607640 1 --2- 172.21.15.172:0/2546849058 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc040cb5c0 0x7fcc041f50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:04.753 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.772+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 >> 172.21.15.172:0/2546849058 conn(0x7fcc04066c20 msgr2=0x7fcc040ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:04.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.773+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 shutdown_connections 2024-03-16T19:31:04.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:04.773+0000 7fcc0c607640 1 -- 172.21.15.172:0/2546849058 wait complete. 2024-03-16T19:31:04.787 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:04.807+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:04.869 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:04.889+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:04.893 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:04.893 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:04.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:04.970 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:04.990+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:05.071 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:05.091+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:05.074 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:05.074 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:05.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:05.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:31:05.172 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:05.192+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:05.260 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:05.260 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:05.273 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:05.293+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:05.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:05.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.305+0000 7f635bc5c640 1 Processor -- start 2024-03-16T19:31:05.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.305+0000 7f635bc5c640 1 -- start start 2024-03-16T19:31:05.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.306+0000 7f635bc5c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6354154ed0 0x7f635415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.306+0000 7f635bc5c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f63540b4700 0x7f63540b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.306+0000 7f635bc5c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63540b4270 con 0x7f63540b3630 2024-03-16T19:31:05.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.306+0000 7f635bc5c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63540b5050 con 0x7f63540b4700 2024-03-16T19:31:05.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.306+0000 7f635bc5c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63541552b0 con 0x7f6354154ed0 2024-03-16T19:31:05.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.306+0000 7f63599f8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6354154ed0 0x7f635415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.306+0000 7f63591f7640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f63540b3630 0x7f63540cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:54884/0 (socket says 172.21.15.172:54884) 2024-03-16T19:31:05.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.306+0000 7f63591f7640 1 -- 172.21.15.172:0/1854516868 learned_addr learned my addr 172.21.15.172:0/1854516868 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:05.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.307+0000 7f63599f8640 1 -- 172.21.15.172:0/1854516868 >> v2:172.21.15.172:3300/0 conn(0x7f63540b4700 msgr2=0x7f63540b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:05.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.307+0000 7f63589f6640 1 -- 172.21.15.172:0/1854516868 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323953872 0 0) 0x7f63540b4270 con 0x7f63540b3630 2024-03-16T19:31:05.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.307+0000 7f635a1f9640 1 --2- 172.21.15.172:0/1854516868 >> v2:172.21.15.172:3300/0 conn(0x7f63540b4700 0x7f63540b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.307+0000 7f63599f8640 1 --2- 172.21.15.172:0/1854516868 >> v2:172.21.15.172:3300/0 conn(0x7f63540b4700 0x7f63540b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:05.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.307+0000 7f63599f8640 1 -- 172.21.15.172:0/1854516868 >> v1:172.21.15.174:6789/0 conn(0x7f63540b3630 legacy=0x7f63540cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:05.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.307+0000 7f63599f8640 1 -- 172.21.15.172:0/1854516868 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635415b4b0 con 0x7f6354154ed0 2024-03-16T19:31:05.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.307+0000 7f635a1f9640 1 --2- 172.21.15.172:0/1854516868 >> v2:172.21.15.172:3300/0 conn(0x7f63540b4700 0x7f63540b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:31:05.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.308+0000 7f63599f8640 1 --2- 172.21.15.172:0/1854516868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6354154ed0 0x7f635415ab40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f63540cde30 tx=0x7f635002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cef384ac6f50603 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.308+0000 7f63589f6640 1 -- 172.21.15.172:0/1854516868 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350031830 con 0x7f6354154ed0 2024-03-16T19:31:05.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.308+0000 7f63589f6640 1 -- 172.21.15.172:0/1854516868 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6350031990 con 0x7f6354154ed0 2024-03-16T19:31:05.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.308+0000 7f63589f6640 1 -- 172.21.15.172:0/1854516868 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350031d20 con 0x7f6354154ed0 2024-03-16T19:31:05.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.309+0000 7f635bc5c640 1 -- 172.21.15.172:0/1854516868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6354154ed0 msgr2=0x7f635415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:05.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.309+0000 7f635bc5c640 1 --2- 172.21.15.172:0/1854516868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6354154ed0 0x7f635415ab40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f63540cde30 tx=0x7f635002b8d0 comp rx=0 tx=0).stop 2024-03-16T19:31:05.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.309+0000 7f635bc5c640 1 -- 172.21.15.172:0/1854516868 shutdown_connections 2024-03-16T19:31:05.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.309+0000 7f635bc5c640 1 --2- 172.21.15.172:0/1854516868 >> v2:172.21.15.172:3300/0 conn(0x7f63540b4700 0x7f63540b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:05.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.309+0000 7f635bc5c640 1 --2- 172.21.15.172:0/1854516868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6354154ed0 0x7f635415ab40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:05.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.309+0000 7f635bc5c640 1 -- 172.21.15.172:0/1854516868 >> 172.21.15.172:0/1854516868 conn(0x7f635406bf60 msgr2=0x7f635406d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:05.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.309+0000 7f635bc5c640 1 -- 172.21.15.172:0/1854516868 shutdown_connections 2024-03-16T19:31:05.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.309+0000 7f635bc5c640 1 -- 172.21.15.172:0/1854516868 wait complete. 2024-03-16T19:31:05.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.311+0000 7f635bc5c640 1 Processor -- start 2024-03-16T19:31:05.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.311+0000 7f635bc5c640 1 -- start start 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.313+0000 7f635bc5c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63540b3630 0x7f6354149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.313+0000 7f635bc5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63540b4700 0x7f635415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.313+0000 7f63599f8640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63540b3630 0x7f6354149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.313+0000 7f63599f8640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63540b3630 0x7f6354149650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57178/0 (socket says 172.21.15.172:57178) 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.313+0000 7f63599f8640 1 -- 172.21.15.172:0/3968345056 learned_addr learned my addr 172.21.15.172:0/3968345056 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.313+0000 7f635bc5c640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6354154ed0 0x7f635415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.313+0000 7f63591f7640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63540b4700 0x7f635415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.314+0000 7f635bc5c640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541f0e40 con 0x7f63540b3630 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.314+0000 7f635bc5c640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635415f030 con 0x7f6354154ed0 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.314+0000 7f635bc5c640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635415ea70 con 0x7f63540b4700 2024-03-16T19:31:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.314+0000 7f635a1f9640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6354154ed0 0x7f635415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.314+0000 7f635a1f9640 1 -- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63540b4700 msgr2=0x7f635415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.314+0000 7f635a1f9640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63540b4700 0x7f635415e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.314+0000 7f635a1f9640 1 -- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63540b3630 msgr2=0x7f6354149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.314+0000 7f635a1f9640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63540b3630 0x7f6354149650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.314+0000 7f635a1f9640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350009400 con 0x7f6354154ed0 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.315+0000 7f63599f8640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63540b3630 0x7f6354149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.315+0000 7f635a1f9640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6354154ed0 0x7f635415cf60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f634c00cdf0 tx=0x7f634c00a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.315+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c00d840 con 0x7f6354154ed0 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.315+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c00d9a0 con 0x7f6354154ed0 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.315+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c00dd50 con 0x7f6354154ed0 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.315+0000 7f635bc5c640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635415ed50 con 0x7f6354154ed0 2024-03-16T19:31:05.296 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.315+0000 7f635bc5c640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63541f0ad0 con 0x7f6354154ed0 2024-03-16T19:31:05.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.316+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f63541f0640 con 0x7f6354154ed0 2024-03-16T19:31:05.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.317+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f634c01f440 con 0x7f6354154ed0 2024-03-16T19:31:05.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.318+0000 7f634a7fc640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f633806b3b0 0x7f633806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.318+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f634c08a670 con 0x7f6354154ed0 2024-03-16T19:31:05.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.318+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v66) v1 ==== 54+0+16 (secure 0 0 0) 0x7f634c091070 con 0x7f6354154ed0 2024-03-16T19:31:05.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.318+0000 7f63599f8640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f633806b3b0 0x7f633806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.318+0000 7f6337fff640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f63540c9ae0 0x7f6354147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.299 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.319+0000 7f63599f8640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f633806b3b0 0x7f633806d870 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f635002b910 tx=0x7f635003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.319+0000 7f63591f7640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f63540c9ae0 0x7f6354147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.319+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63540b3c20 con 0x7f63540c9ae0 2024-03-16T19:31:05.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.319+0000 7f63591f7640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f63540c9ae0 0x7f6354147330 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.321+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f63440047e0 con 0x7f63540c9ae0 2024-03-16T19:31:05.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.350+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f635415ba70 con 0x7f63540c9ae0 2024-03-16T19:31:05.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.353+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6344004950 con 0x7f63540c9ae0 2024-03-16T19:31:05.333 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.353+0000 7f6337fff640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f63540c9ec0 0x7f63541f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.353+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6354068d20 con 0x7f63540c9ec0 2024-03-16T19:31:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.354+0000 7f63599f8640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f63540c9ec0 0x7f63541f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.334 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.354+0000 7f63599f8640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f63540c9ec0 0x7f63541f3be0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.355+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6354068d20 con 0x7f63540c9ec0 2024-03-16T19:31:05.366 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.386+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63540cd210 con 0x7f63540c9ec0 2024-03-16T19:31:05.368 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.388+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f63540cd210 con 0x7f63540c9ec0 2024-03-16T19:31:05.369 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:05.369 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:05.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.389+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=67}) v3 -- 0x7f63540681c0 con 0x7f6354154ed0 2024-03-16T19:31:05.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.389+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63540cbbe0 con 0x7f6354154ed0 2024-03-16T19:31:05.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.389+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f634c055250 con 0x7f6354154ed0 2024-03-16T19:31:05.370 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:05.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.389+0000 7f6337fff640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f63541f4630 0x7f6354061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.389+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63541f4a10 con 0x7f63541f4630 2024-03-16T19:31:05.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.389+0000 7f63591f7640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f63541f4630 0x7f6354061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.390+0000 7f63591f7640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f63541f4630 0x7f6354061930 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.391+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f63541f4a10 con 0x7f63541f4630 2024-03-16T19:31:05.374 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:05.394+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:05.400 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.420+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63540620c0 con 0x7f63541f4630 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.825481383, 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.401 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:31:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:31:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:31:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.035255212, 2024-03-16T19:31:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:31:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.03449497699999, 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:31:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:31:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.033681022, 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:31:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:31:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.03304109699999, 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:31:05.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:31:05.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:31:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.032383, 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.422 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:31:05.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.031850425, 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 107.973678522, 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.430 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:31:05.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 107.973076328, 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:31:05.434 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:31:05.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:31:05.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 107.97243338299999, 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:31:05.438 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:05.442 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 107.971929422, 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:31:05.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.813783156, 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:31:05.446 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:31:05.450 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.81325302, 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:31:05.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.061283760999999, 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:05.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:31:05.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:31:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:31:05.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.459 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.061259225000001, 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:05.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:31:05.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:31:05.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:31:05.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.464 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.061254695000002, 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.061244883000001, 2024-03-16T19:31:05.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:31:05.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:05.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.061241030000005, 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.479 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.061237714000001, 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:05.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:31:05.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:31:05.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.060042131000003, 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.487 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.488 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:05.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.422+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f63540620c0 con 0x7f63541f4630 2024-03-16T19:31:05.489 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:05.489 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:05.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.423+0000 7f6337fff640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6354062650 0x7f63541530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.423+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63541526b0 con 0x7f6354062650 2024-03-16T19:31:05.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.423+0000 7f635a1f9640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6354062650 0x7f63541530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.423+0000 7f635a1f9640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f6354062650 0x7f63541530e0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.424+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f63541526b0 con 0x7f6354062650 2024-03-16T19:31:05.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:05.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.453+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63541538f0 con 0x7f6354062650 2024-03-16T19:31:05.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.454+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f63541538f0 con 0x7f6354062650 2024-03-16T19:31:05.492 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:05.492 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:05.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.455+0000 7f6337fff640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6354207c50 0x7f635420a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.455+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63540cd770 con 0x7f6354207c50 2024-03-16T19:31:05.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.455+0000 7f63599f8640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6354207c50 0x7f635420a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.455+0000 7f63599f8640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f6354207c50 0x7f635420a040 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.456+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f63540cd770 con 0x7f6354207c50 2024-03-16T19:31:05.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.485+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63540cd770 con 0x7f6354207c50 2024-03-16T19:31:05.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.486+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f63540cd770 con 0x7f6354207c50 2024-03-16T19:31:05.493 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:05.494 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:05.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.486+0000 7f6337fff640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f635420aa90 0x7f635420ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.487+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6354153ba0 con 0x7f635420aa90 2024-03-16T19:31:05.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.487+0000 7f63591f7640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f635420aa90 0x7f635420ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.487+0000 7f63591f7640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f635420aa90 0x7f635420ce80 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.488+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6354153ba0 con 0x7f635420aa90 2024-03-16T19:31:05.495 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:05.509+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:05.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.517+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6354153ba0 con 0x7f635420aa90 2024-03-16T19:31:05.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.518+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6354153ba0 con 0x7f635420aa90 2024-03-16T19:31:05.499 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:05.499 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:05.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.519+0000 7f6337fff640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f635420d930 0x7f635420dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:05.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.519+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6354212e40 con 0x7f635420d930 2024-03-16T19:31:05.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.519+0000 7f635a1f9640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f635420d930 0x7f635420dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:05.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.519+0000 7f635a1f9640 1 --2- 172.21.15.172:0/3968345056 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f635420d930 0x7f635420dd70 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:05.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.520+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6354212e40 con 0x7f635420d930 2024-03-16T19:31:05.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.549+0000 7f6337fff640 1 -- 172.21.15.172:0/3968345056 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f635420e320 con 0x7f635420d930 2024-03-16T19:31:05.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:05.551+0000 7f634a7fc640 1 -- 172.21.15.172:0/3968345056 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f635420e320 con 0x7f635420d930 2024-03-16T19:31:05.531 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:05.531 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:05.536 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:05.589 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:05.609+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:05.628 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:05.629 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:05.629 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:05.629 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:05.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:05.691 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:05.710+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:05.747 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:05.747 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:05.747 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:05.747 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:05.747 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:05.758 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:05.791 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:05.811+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:05.891 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:05.911+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:05.913 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:05.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.574124691999998, 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.574107577999996, 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.574100298000005, 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:05.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:31:05.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:31:05.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:31:05.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.928 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.574086430999998, 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:31:05.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:31:05.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:31:05.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.933 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.573413595999995, 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:05.937 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.571115345999999, 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:31:05.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:05.941 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.571070245000001, 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.57104219, 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:31:05.945 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:05.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:31:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.571020775999997, 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:31:05.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:31:05.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.570998424999999, 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.570978026000006, 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:31:05.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:31:05.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.960 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.570872034000004, 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:31:05.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.569080322999994, 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:05.968 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.569032821999997, 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.568793571, 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.560295425999996, 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:31:05.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:31:05.980 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.558955549999993, 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.555964568, 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.555333765999997, 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:47 2.6 e40/18 ondisk, result = 0)", 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.553851356999999, 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017677, 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377611+0000", 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9599999999999998e-07 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377618+0000", 2024-03-16T19:31:05.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9990000000000002e-06 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377626+0000", 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4249999999999998e-06 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377632+0000", 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:05.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377707+0000", 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4506999999999997e-05 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377714+0000", 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5819999999999998e-06 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377787+0000", 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3232000000000006e-05 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:05.997 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:05.998 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:05.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:05.999 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:06.018+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:06.099 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:06.119+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:06.175 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:06.175 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:06.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:06.200 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:06.220+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:06.301 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:06.321+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:06.401 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:06.401 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:06.401 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:06.402 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:06.422+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:06.414 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:06.502 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:06.522+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:06.603 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:06.623+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.264751641000004, 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:31:06.607 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.264737167999996, 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:31:06.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:31:06.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.264232679000003, 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:31:06.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.616 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.264217282999994, 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:31:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:31:06.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.263758600000003, 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.621 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:31:06.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.263212429999996, 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.262655339000005, 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:06.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:31:06.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:31:06.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:31:06.633 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.261453591000006, 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.260874881999996, 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:06.638 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:31:06.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.641 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.260825354999994, 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.259864711999995, 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:31:06.648 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.256150513999998, 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:31:06.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:31:06.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 86.815749601999997, 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:06.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:31:06.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:31:06.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.250196599000006, 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:06.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.660 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 83.742597873999998, 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:06.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPull(2.e e67/66 cost 1048576)", 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.83792396499999999, 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026662799999999998, 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:06.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0500000000000003e-07 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784292+0000", 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5139e-05 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784304+0000", 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1501000000000001e-05 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784313+0000", 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8189999999999998e-06 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784384+0000", 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1192999999999994e-05 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:06.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:06.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:06.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784543+0000", 2024-03-16T19:31:06.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159371 2024-03-16T19:31:06.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:06.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:06.668 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:06.669 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:06.713 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:06.733+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:06.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:31:06.778 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:06.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:06.779 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:06.779 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:06.787 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:06.820 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:06.840+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:06.905 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:06.925+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:06.934 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:06.934 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:06.934 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:06.934 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:06.934 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:06.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.954+0000 7f2acaeb5640 1 Processor -- start 2024-03-16T19:31:06.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.954+0000 7f2acaeb5640 1 -- start start 2024-03-16T19:31:06.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2acaeb5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 0x7f2ac40cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:06.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2acaeb5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2ac40caf40 0x7f2ac40cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:06.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2acaeb5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac406b110 con 0x7f2ac40b5f10 2024-03-16T19:31:06.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2acaeb5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac415b8d0 con 0x7f2ac40caf40 2024-03-16T19:31:06.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2acaeb5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac415ad50 con 0x7f2ac4154e50 2024-03-16T19:31:06.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2ac9452640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2ac40caf40 0x7f2ac40cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:06.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2ac8c51640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 0x7f2ac40cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:06.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2ac9452640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2ac40caf40 0x7f2ac40cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:46390/0 (socket says 172.21.15.172:46390) 2024-03-16T19:31:06.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2ac9452640 1 -- 172.21.15.172:0/1465145485 learned_addr learned my addr 172.21.15.172:0/1465145485 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:06.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2ac9452640 1 -- 172.21.15.172:0/1465145485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 msgr2=0x7f2ac40cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:06.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2ac9452640 1 --2- 172.21.15.172:0/1465145485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 0x7f2ac40cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:06.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2ac9452640 1 -- 172.21.15.172:0/1465145485 >> v1:172.21.15.174:6789/0 conn(0x7f2ac40b5f10 legacy=0x7f2ac406b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:06.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.956+0000 7f2ac9452640 1 -- 172.21.15.172:0/1465145485 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac415b030 con 0x7f2ac40caf40 2024-03-16T19:31:06.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.957+0000 7f2ac9452640 1 --2- 172.21.15.172:0/1465145485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40caf40 0x7f2ac40cc170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009e50 tx=0x7f2abc02bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28763a9d322bf52f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:06.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.958+0000 7f2ac37fe640 1 -- 172.21.15.172:0/1465145485 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc02cb90 con 0x7f2ac40caf40 2024-03-16T19:31:06.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.958+0000 7f2ac37fe640 1 -- 172.21.15.172:0/1465145485 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc02ccf0 con 0x7f2ac40caf40 2024-03-16T19:31:06.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.958+0000 7f2ac37fe640 1 -- 172.21.15.172:0/1465145485 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc034610 con 0x7f2ac40caf40 2024-03-16T19:31:06.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.958+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1465145485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40caf40 msgr2=0x7f2ac40cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:06.939 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.958+0000 7f2acaeb5640 1 --2- 172.21.15.172:0/1465145485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40caf40 0x7f2ac40cc170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009e50 tx=0x7f2abc02bff0 comp rx=0 tx=0).stop 2024-03-16T19:31:06.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.959+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1465145485 shutdown_connections 2024-03-16T19:31:06.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.959+0000 7f2acaeb5640 1 --2- 172.21.15.172:0/1465145485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40caf40 0x7f2ac40cc170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:06.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.959+0000 7f2acaeb5640 1 --2- 172.21.15.172:0/1465145485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 0x7f2ac40cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:06.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.959+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1465145485 >> 172.21.15.172:0/1465145485 conn(0x7f2ac4066c20 msgr2=0x7f2ac4067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:06.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.959+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1465145485 shutdown_connections 2024-03-16T19:31:06.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.959+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1465145485 wait complete. 2024-03-16T19:31:06.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.960+0000 7f2acaeb5640 1 Processor -- start 2024-03-16T19:31:06.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.960+0000 7f2acaeb5640 1 -- start start 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2acaeb5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40b5f10 0x7f2ac4150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2acaeb5640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ac40caf40 0x7f2ac414fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2acaeb5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 0x7f2ac414beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2acaeb5640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac414f840 con 0x7f2ac40caf40 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2acaeb5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac414e5e0 con 0x7f2ac40b5f10 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2ac3fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ac40caf40 0x7f2ac414fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2ac8c51640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40b5f10 0x7f2ac4150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2ac3fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ac40caf40 0x7f2ac414fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57188/0 (socket says 172.21.15.172:57188) 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2ac3fff640 1 -- 172.21.15.172:0/1455024781 learned_addr learned my addr 172.21.15.172:0/1455024781 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2ac9452640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 0x7f2ac414beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.962+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac414d780 con 0x7f2ac4154e50 2024-03-16T19:31:06.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac8c51640 1 -- 172.21.15.172:0/1455024781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 msgr2=0x7f2ac414beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac8c51640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 0x7f2ac414beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac8c51640 1 -- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ac40caf40 msgr2=0x7f2ac414fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac8c51640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ac40caf40 0x7f2ac414fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac8c51640 1 -- 172.21.15.172:0/1455024781 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc009b30 con 0x7f2ac40b5f10 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac3fff640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ac40caf40 0x7f2ac414fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac8c51640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40b5f10 0x7f2ac4150960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800e870 tx=0x7f2ab800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac17fa640 1 -- 172.21.15.172:0/1455024781 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800ee60 con 0x7f2ac40b5f10 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac17fa640 1 -- 172.21.15.172:0/1455024781 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab800c940 con 0x7f2ac40b5f10 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2ac17fa640 1 -- 172.21.15.172:0/1455024781 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800ccd0 con 0x7f2ac40b5f10 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac415ab20 con 0x7f2ac40b5f10 2024-03-16T19:31:06.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.963+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac414dd40 con 0x7f2ac40b5f10 2024-03-16T19:31:06.947 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:06.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.965+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac40c9ab0 con 0x7f2ac40b5f10 2024-03-16T19:31:06.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.965+0000 7f2ac17fa640 1 -- 172.21.15.172:0/1455024781 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f2ab8013470 con 0x7f2ac40b5f10 2024-03-16T19:31:06.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.966+0000 7f2ac17fa640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2aa806b3b0 0x7f2aa806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:06.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.966+0000 7f2ac3fff640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2aa806b3b0 0x7f2aa806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:06.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.966+0000 7f2ac17fa640 1 -- 172.21.15.172:0/1455024781 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f2ab8089cb0 con 0x7f2ac40b5f10 2024-03-16T19:31:06.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.966+0000 7f2ac3fff640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2aa806b3b0 0x7f2aa806d870 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4150ea0 tx=0x7f2ab0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:06.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:06.970+0000 7f2ac17fa640 1 -- 172.21.15.172:0/1455024781 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ab8054890 con 0x7f2ac40b5f10 2024-03-16T19:31:07.031 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:07.051+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:07.093 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:07.093 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:07.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:07.093 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:07.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.093 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:07.093 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:07.093 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 95.755851738000004, 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:07.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.098 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 95.754624316000005, 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 95.754221610000002, 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:31:07.106 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 95.753612859, 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:31:07.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:31:07.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 95.753168990000006, 2024-03-16T19:31:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.115 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 95.753026214000002, 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:07.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:31:07.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 95.753011673000003, 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.119 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:31:07.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:31:07.123 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 95.750133977999994, 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:31:07.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.127 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:07.128 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:07.129 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:07.148+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:07.229 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:07.249+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:07.275 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:07.275 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:07.275 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:07.276 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:07.284 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:07.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.347+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f2ac40cba10 con 0x7f2ac40b5f10 2024-03-16T19:31:07.368 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:07.388+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:07.430 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:07.450+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:07.431 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:07.431 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:07.431 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:07.431 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:07.431 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:07.439 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:07.532 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:07.552+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:07.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:31:07.586 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:07.586 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:07.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:07.586 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:07.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 96.248255004000001, 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:07.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:31:07.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 96.248175885999999, 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.594 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 96.247561762000004, 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.598 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 96.247121692999997, 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:31:07.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.602 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 96.246880102999995, 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:31:07.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:07.606 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:07.607 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:07.607 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:07.633 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:07.653+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:07.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.709+0000 7fd7bd476640 1 Processor -- start 2024-03-16T19:31:07.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.709+0000 7fd7bd476640 1 -- start start 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7bd476640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b8154ed0 0x7fd7b815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7bd476640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd7b80b4700 0x7fd7b80b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7b67fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd7b80b3630 0x7fd7b80cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:41538/0 (socket says 172.21.15.172:41538) 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7b67fc640 1 -- 172.21.15.172:0/2855526135 learned_addr learned my addr 172.21.15.172:0/2855526135 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b8154ed0 0x7fd7b815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7b77fe640 1 --2- 172.21.15.172:0/2855526135 >> v2:172.21.15.174:3300/0 conn(0x7fd7b80b4700 0x7fd7b80b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7bd476640 1 -- 172.21.15.172:0/2855526135 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b80b4270 con 0x7fd7b80b3630 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7bd476640 1 -- 172.21.15.172:0/2855526135 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b81552b0 con 0x7fd7b80b4700 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7bd476640 1 -- 172.21.15.172:0/2855526135 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b815b4b0 con 0x7fd7b8154ed0 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348368159 0 0) 0x7fd7b80b4270 con 0x7fd7b80b3630 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.711+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a0003520 con 0x7fd7b80b3630 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 39659001 0 0) 0x7fd7a0003520 con 0x7fd7b80b3630 2024-03-16T19:31:07.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b8154ed0 msgr2=0x7fd7b815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b8154ed0 0x7fd7b815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 >> v2:172.21.15.174:3300/0 conn(0x7fd7b80b4700 msgr2=0x7fd7b80b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 --2- 172.21.15.172:0/2855526135 >> v2:172.21.15.174:3300/0 conn(0x7fd7b80b4700 0x7fd7b80b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b815b790 con 0x7fd7b80b3630 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7fd7ac001670 con 0x7fd7b80b3630 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 >> v1:172.21.15.172:6789/0 conn(0x7fd7b80b3630 legacy=0x7fd7b80cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b8154ed0 0x7fd7a0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7a0003f40 0x7fd7a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7a0004d00 0x7fd7a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b8154ed0 0x7fd7a0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b81552b0 con 0x7fd7a0003f40 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b815b4b0 con 0x7fd7a0004d00 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a000b980 con 0x7fd7b8154ed0 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b67fc640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7a0003f40 0x7fd7a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.712+0000 7fd7b77fe640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7a0004d00 0x7fd7a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.713+0000 7fd7b6ffd640 1 -- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7a0004d00 msgr2=0x7fd7a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.713+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7a0004d00 0x7fd7a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.713+0000 7fd7b6ffd640 1 -- 172.21.15.172:0/2855526135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7a0003f40 msgr2=0x7fd7a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.713+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7a0003f40 0x7fd7a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.713+0000 7fd7b6ffd640 1 -- 172.21.15.172:0/2855526135 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0003520 con 0x7fd7b8154ed0 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.714+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b8154ed0 0x7fd7a0003970 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009af0 tx=0x7fd7a00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2973db4a44e8bc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.714+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a802c710 con 0x7fd7b8154ed0 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.714+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a802c870 con 0x7fd7b8154ed0 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.714+0000 7fd7b5ffb640 1 -- 172.21.15.172:0/2855526135 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a802cc00 con 0x7fd7b8154ed0 2024-03-16T19:31:07.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.715+0000 7fd7bd476640 1 -- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b8154ed0 msgr2=0x7fd7a0003970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:07.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.715+0000 7fd7bd476640 1 --2- 172.21.15.172:0/2855526135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b8154ed0 0x7fd7a0003970 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009af0 tx=0x7fd7a00057f0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.715+0000 7fd7b77fe640 1 -- 172.21.15.172:0/2855526135 reap_dead start 2024-03-16T19:31:07.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.715+0000 7fd7bd476640 1 -- 172.21.15.172:0/2855526135 shutdown_connections 2024-03-16T19:31:07.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.715+0000 7fd7bd476640 1 -- 172.21.15.172:0/2855526135 >> 172.21.15.172:0/2855526135 conn(0x7fd7b806bf60 msgr2=0x7fd7b806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:07.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.715+0000 7fd7bd476640 1 -- 172.21.15.172:0/2855526135 shutdown_connections 2024-03-16T19:31:07.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.715+0000 7fd7bd476640 1 -- 172.21.15.172:0/2855526135 wait complete. 2024-03-16T19:31:07.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.717+0000 7fd7bd476640 1 Processor -- start 2024-03-16T19:31:07.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.717+0000 7fd7bd476640 1 -- start start 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7bd476640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7b8154ed0 0x7fd7b81f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7bd476640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b80c47f0 0x7fd7b815c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7b6ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7b8154ed0 0x7fd7b81f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7b6ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7b8154ed0 0x7fd7b81f0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57204/0 (socket says 172.21.15.172:57204) 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7b6ffd640 1 -- 172.21.15.172:0/3659091521 learned_addr learned my addr 172.21.15.172:0/3659091521 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7bd476640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7b815bd00 0x7fd7b815caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b81f08a0 con 0x7fd7b8154ed0 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b80c6170 con 0x7fd7b815bd00 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b80c6410 con 0x7fd7b80c47f0 2024-03-16T19:31:07.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.718+0000 7fd7b67fc640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b80c47f0 0x7fd7b815c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd7b77fe640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7b815bd00 0x7fd7b815caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd7b67fc640 1 -- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7b815bd00 msgr2=0x7fd7b815caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:07.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd7b67fc640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7b815bd00 0x7fd7b815caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd7b67fc640 1 -- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7b8154ed0 msgr2=0x7fd7b81f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:07.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd7b67fc640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7b8154ed0 0x7fd7b81f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd7b67fc640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b8154560 con 0x7fd7b80c47f0 2024-03-16T19:31:07.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7b8154ed0 0x7fd7b81f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:07.700 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd7b67fc640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7b80c47f0 0x7fd7b815c5b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac004ea0 tx=0x7fd7ac004f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac00f7a0 con 0x7fd7b80c47f0 2024-03-16T19:31:07.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ac00f920 con 0x7fd7b80c47f0 2024-03-16T19:31:07.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.719+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b80c66b0 con 0x7fd7b80c47f0 2024-03-16T19:31:07.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.720+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac00fcd0 con 0x7fd7b80c47f0 2024-03-16T19:31:07.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.720+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b80c6c70 con 0x7fd7b80c47f0 2024-03-16T19:31:07.701 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.721+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd7b8068d20 con 0x7fd7b80c47f0 2024-03-16T19:31:07.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.721+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd7ac01d460 con 0x7fd7b80c47f0 2024-03-16T19:31:07.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.722+0000 7fd79f7fe640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd7a406b3b0 0x7fd7a406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.702 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.722+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd7a406b3b0 0x7fd7a406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.723+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6002+0+0 (secure 0 0 0) 0x7fd7ac089c10 con 0x7fd7b80c47f0 2024-03-16T19:31:07.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.723+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd7a406b3b0 0x7fd7a406d870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8000c00 tx=0x7fd7a8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.723+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v67) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd7ac08f220 con 0x7fd7b80c47f0 2024-03-16T19:31:07.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.723+0000 7fd7bd476640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd7b81f0470 0x7fd7b8146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.723+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7b81f0a10 con 0x7fd7b81f0470 2024-03-16T19:31:07.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.723+0000 7fd7b77fe640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd7b81f0470 0x7fd7b8146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.724+0000 7fd7b77fe640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd7b81f0470 0x7fd7b8146fe0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.725+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd7b81f0a10 con 0x7fd7b81f0470 2024-03-16T19:31:07.714 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:07.714 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:07.723 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:07.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.754+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd7b80b3f00 con 0x7fd7b81f0470 2024-03-16T19:31:07.735 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.755+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd7b000f040 con 0x7fd7b81f0470 2024-03-16T19:31:07.736 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:07.736 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:07.736 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.756+0000 7fd7bd476640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd7b81f3990 0x7fd7b81f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.756+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7b80681c0 con 0x7fd7b81f3990 2024-03-16T19:31:07.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.756+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd7b81f3990 0x7fd7b81f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.756+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd7b81f3990 0x7fd7b81f3dd0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.737 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.757+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd7b80681c0 con 0x7fd7b81f3990 2024-03-16T19:31:07.766 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:07.786+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:07.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.787+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd7b814b9d0 con 0x7fd7b81f3990 2024-03-16T19:31:07.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.789+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd7b814b9d0 con 0x7fd7b81f3990 2024-03-16T19:31:07.769 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.789+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=68}) v3 -- 0x7fd7b8150520 con 0x7fd7b80c47f0 2024-03-16T19:31:07.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.789+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd7b8153430 con 0x7fd7b80c47f0 2024-03-16T19:31:07.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.790+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7ac054880 con 0x7fd7b80c47f0 2024-03-16T19:31:07.771 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:07.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.790+0000 7fd7bd476640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd7b8061e40 0x7fd7b8062290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.790+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7b80ccd60 con 0x7fd7b8061e40 2024-03-16T19:31:07.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.790+0000 7fd7b67fc640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd7b8061e40 0x7fd7b8062290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.790+0000 7fd7b67fc640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd7b8061e40 0x7fd7b8062290 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.791+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd7b80ccd60 con 0x7fd7b8061e40 2024-03-16T19:31:07.775 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.795+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== mon.2 v2:172.21.15.174:3301/0 8 ==== osd_map(68..68 src has 1..68) v4 ==== 288+0+0 (secure 0 0 0) 0x7fd7ac0585e0 con 0x7fd7b80c47f0 2024-03-16T19:31:07.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.797+0000 7f2ac17fa640 1 -- 172.21.15.172:0/1455024781 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v68) v1 ==== 87+0+0 (secure 0 0 0) 0x7f2ab8021090 con 0x7f2ac40b5f10 2024-03-16T19:31:07.778 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2aa806b3b0 msgr2=0x7f2aa806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2aa806b3b0 0x7f2aa806d870 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4150ea0 tx=0x7f2ab0009040 comp rx=0 tx=0).stop 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40b5f10 msgr2=0x7f2ac4150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40b5f10 0x7f2ac4150960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800e870 tx=0x7f2ab800ed30 comp rx=0 tx=0).stop 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 shutdown_connections 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2aa806b3b0 0x7f2aa806d870 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ac4154e50 0x7f2ac414beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ac40caf40 0x7f2ac414fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 --2- 172.21.15.172:0/1455024781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac40b5f10 0x7f2ac4150960 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 >> 172.21.15.172:0/1455024781 conn(0x7f2ac4066c20 msgr2=0x7f2ac406b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:07.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 shutdown_connections 2024-03-16T19:31:07.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.801+0000 7f2acaeb5640 1 -- 172.21.15.172:0/1455024781 wait complete. 2024-03-16T19:31:07.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.813+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd7b80627d0 con 0x7fd7b8061e40 2024-03-16T19:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:07.795 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:07.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.815+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd7b80627d0 con 0x7fd7b8061e40 2024-03-16T19:31:07.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.815+0000 7fd7bd476640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd7b80cc4b0 0x7fd7b802bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.815+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7b80cc8d0 con 0x7fd7b80cc4b0 2024-03-16T19:31:07.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.815+0000 7fd7b77fe640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd7b80cc4b0 0x7fd7b802bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.815+0000 7fd7b77fe640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd7b80cc4b0 0x7fd7b802bde0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.816+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd7b80cc8d0 con 0x7fd7b80cc4b0 2024-03-16T19:31:07.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.830+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd7b802c320 con 0x7fd7b80cc4b0 2024-03-16T19:31:07.812 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.832+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd7b802c320 con 0x7fd7b80cc4b0 2024-03-16T19:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:07.813 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:07.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.832+0000 7fd7bd476640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd7b802ca00 0x7fd7b802ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.832+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7b82075a0 con 0x7fd7b802ca00 2024-03-16T19:31:07.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.832+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd7b802ca00 0x7fd7b802ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.833+0000 7fd7b6ffd640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd7b802ca00 0x7fd7b802ce80 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.834+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd7b82075a0 con 0x7fd7b802ca00 2024-03-16T19:31:07.834 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:07.854+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:07.836 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:07.836 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:07.836 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:07.836 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:07.836 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:07.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.861+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd7b8062940 con 0x7fd7b802ca00 2024-03-16T19:31:07.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.863+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd7b8062940 con 0x7fd7b802ca00 2024-03-16T19:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:07.843 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.863+0000 7fd7bd476640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd7b8205220 0x7fd7b8207610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.863+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7b820a3e0 con 0x7fd7b8205220 2024-03-16T19:31:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.863+0000 7fd7b67fc640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd7b8205220 0x7fd7b8207610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.864+0000 7fd7b67fc640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd7b8205220 0x7fd7b8207610 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.845 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:07.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.865+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd7b820a3e0 con 0x7fd7b8205220 2024-03-16T19:31:07.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.893+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd7b8207b50 con 0x7fd7b8205220 2024-03-16T19:31:07.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.895+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd7b8207b50 con 0x7fd7b8205220 2024-03-16T19:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:07.876 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:07.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.895+0000 7fd7bd476640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd7b82081d0 0x7fd7b820a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:07.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.895+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7b820d390 con 0x7fd7b82081d0 2024-03-16T19:31:07.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.896+0000 7fd7b77fe640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd7b82081d0 0x7fd7b820a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:07.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.896+0000 7fd7b77fe640 1 --2- 172.21.15.172:0/3659091521 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd7b82081d0 0x7fd7b820a5c0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:07.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.897+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd7b820d390 con 0x7fd7b82081d0 2024-03-16T19:31:07.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.918+0000 7fd7bd476640 1 -- 172.21.15.172:0/3659091521 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd7b820ab00 con 0x7fd7b82081d0 2024-03-16T19:31:07.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:07.920+0000 7fd79f7fe640 1 -- 172.21.15.172:0/3659091521 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd7b820ab00 con 0x7fd7b82081d0 2024-03-16T19:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:07.900 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:07.905 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:07.934 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:07.954+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:08.024 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:08.024 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:08.036 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:08.036 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:08.055+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:08.136 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:08.156+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:08.185 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:08.186 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:08.195 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:08.239 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:08.259+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:08.304 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:08.304 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:08.313 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:08.354 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:08.374+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:08.438 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:08.459+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:08.463 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:08.463 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:08.474 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:08.540 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:08.560+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:08.586 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:08.586 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:08.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:08.641 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:08.661+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:08.707 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:08.707 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:08.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:08.741 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:08.762+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:08.781 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-16T19:31:08.781 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:31:08.781 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-16T19:31:08.781 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-16T19:31:08.781 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-16T19:31:08.782 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-03-16T19:31:08.829 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:08.830 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:08.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:08.842 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:08.862+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:08.915 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:08.915 INFO:teuthology.orchestra.run.smithi174.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-16T19:31:08.915 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:08.943 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:08.963+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:08.950 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:08.950 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:08.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:09.044 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:09.064+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:09.119 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:09.119 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:09.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:09.145 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:09.165+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:09.246 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:09.266+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:09.281 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:09.281 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:09.290 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:09.347 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:09.367+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:09.399 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:09.399 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:09.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:09.453 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:09.473+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:09.518 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:09.519 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:09.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:09.569 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:09.589+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:09.649 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:09.670+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:09.715 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:09.715 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:09.725 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:09.750 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:09.770+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:09.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:31:09.832 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:09.832 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:09.840 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:09.884 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:09.904+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:09.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:31:09.951 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:09.951 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:09.951 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:09.971+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:09.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.974+0000 7faf3986b640 1 Processor -- start 2024-03-16T19:31:09.954 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.974+0000 7faf3986b640 1 -- start start 2024-03-16T19:31:09.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.975+0000 7faf3986b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faf340c3ea0 0x7faf34072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:09.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.975+0000 7faf3986b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faf340c3930 0x7faf340c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:09.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.975+0000 7faf3986b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf34072b10 con 0x7faf34151910 2024-03-16T19:31:09.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.975+0000 7faf3986b640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf340c2330 con 0x7faf340c3ea0 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.975+0000 7faf3986b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf340c7da0 con 0x7faf340c3930 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.976+0000 7faf327fc640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faf340c3ea0 0x7faf34072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.976+0000 7faf32ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7faf34151910 0x7faf340ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:47656/0 (socket says 172.21.15.172:47656) 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.976+0000 7faf32ffd640 1 -- 172.21.15.172:0/1425392581 learned_addr learned my addr 172.21.15.172:0/1425392581 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.976+0000 7faf337fe640 1 --2- 172.21.15.172:0/1425392581 >> v2:172.21.15.174:3300/0 conn(0x7faf340c3930 0x7faf340c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.976+0000 7faf327fc640 1 -- 172.21.15.172:0/1425392581 >> v2:172.21.15.174:3300/0 conn(0x7faf340c3930 msgr2=0x7faf340c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.976+0000 7faf327fc640 1 --2- 172.21.15.172:0/1425392581 >> v2:172.21.15.174:3300/0 conn(0x7faf340c3930 0x7faf340c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.976+0000 7faf327fc640 1 -- 172.21.15.172:0/1425392581 >> v1:172.21.15.172:6789/0 conn(0x7faf34151910 legacy=0x7faf340ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.976+0000 7faf327fc640 1 -- 172.21.15.172:0/1425392581 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf34071b10 con 0x7faf340c3ea0 2024-03-16T19:31:09.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.976+0000 7faf337fe640 1 --2- 172.21.15.172:0/1425392581 >> v2:172.21.15.174:3300/0 conn(0x7faf340c3930 0x7faf340c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:31:09.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.977+0000 7faf327fc640 1 --2- 172.21.15.172:0/1425392581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 0x7faf34072fa0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7faf340b2b30 tx=0x7faf28034390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3330ee9d564049d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:09.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.977+0000 7faf31ffb640 1 -- 172.21.15.172:0/1425392581 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28039710 con 0x7faf340c3ea0 2024-03-16T19:31:09.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.977+0000 7faf31ffb640 1 -- 172.21.15.172:0/1425392581 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf28037050 con 0x7faf340c3ea0 2024-03-16T19:31:09.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.977+0000 7faf31ffb640 1 -- 172.21.15.172:0/1425392581 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28039710 con 0x7faf340c3ea0 2024-03-16T19:31:09.960 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.978+0000 7faf3986b640 1 -- 172.21.15.172:0/1425392581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 msgr2=0x7faf34072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.978+0000 7faf3986b640 1 --2- 172.21.15.172:0/1425392581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 0x7faf34072fa0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7faf340b2b30 tx=0x7faf28034390 comp rx=0 tx=0).stop 2024-03-16T19:31:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.979+0000 7faf3986b640 1 -- 172.21.15.172:0/1425392581 shutdown_connections 2024-03-16T19:31:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.979+0000 7faf3986b640 1 --2- 172.21.15.172:0/1425392581 >> v2:172.21.15.174:3300/0 conn(0x7faf340c3930 0x7faf340c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.979+0000 7faf3986b640 1 --2- 172.21.15.172:0/1425392581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 0x7faf34072fa0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.979+0000 7faf3986b640 1 -- 172.21.15.172:0/1425392581 >> 172.21.15.172:0/1425392581 conn(0x7faf3406db70 msgr2=0x7faf3406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.979+0000 7faf3986b640 1 -- 172.21.15.172:0/1425392581 shutdown_connections 2024-03-16T19:31:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.979+0000 7faf3986b640 1 -- 172.21.15.172:0/1425392581 wait complete. 2024-03-16T19:31:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.980+0000 7faf3986b640 1 Processor -- start 2024-03-16T19:31:09.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.981+0000 7faf3986b640 1 -- start start 2024-03-16T19:31:09.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf3986b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf340c3930 0x7faf341ec270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:09.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf3986b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 0x7faf341454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:09.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf3986b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf34151910 0x7faf34145a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:09.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf327fc640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 0x7faf341454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:09.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf327fc640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 0x7faf341454d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42518/0 (socket says 172.21.15.172:42518) 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf327fc640 1 -- 172.21.15.172:0/1809127170 learned_addr learned my addr 172.21.15.172:0/1809127170 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf340726f0 con 0x7faf340c3ea0 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf32ffd640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf340c3930 0x7faf341ec270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf34158250 con 0x7faf340c3930 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf341f05e0 con 0x7faf34151910 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.982+0000 7faf337fe640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf34151910 0x7faf34145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf32ffd640 1 -- 172.21.15.172:0/1809127170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf34151910 msgr2=0x7faf34145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf32ffd640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf34151910 0x7faf34145a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf32ffd640 1 -- 172.21.15.172:0/1809127170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 msgr2=0x7faf341454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf32ffd640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 0x7faf341454d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf32ffd640 1 -- 172.21.15.172:0/1809127170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280107f0 con 0x7faf340c3930 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf32ffd640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf340c3930 0x7faf341ec270 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf2400c3a0 tx=0x7faf2400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf1bfff640 1 -- 172.21.15.172:0/1809127170 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24005ce0 con 0x7faf340c3930 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf1bfff640 1 -- 172.21.15.172:0/1809127170 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf24005e40 con 0x7faf340c3930 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf1bfff640 1 -- 172.21.15.172:0/1809127170 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf240072f0 con 0x7faf340c3930 2024-03-16T19:31:09.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf34157c90 con 0x7faf340c3930 2024-03-16T19:31:09.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.983+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf341555e0 con 0x7faf340c3930 2024-03-16T19:31:09.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.985+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf34075a70 con 0x7faf340c3930 2024-03-16T19:31:09.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.985+0000 7faf1bfff640 1 -- 172.21.15.172:0/1809127170 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7faf24007450 con 0x7faf340c3930 2024-03-16T19:31:09.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.986+0000 7faf1bfff640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faf2006b3b0 0x7faf2006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:09.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.986+0000 7faf1bfff640 1 -- 172.21.15.172:0/1809127170 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6002+0+0 (secure 0 0 0) 0x7faf240892e0 con 0x7faf340c3930 2024-03-16T19:31:09.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.986+0000 7faf327fc640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faf2006b3b0 0x7faf2006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:09.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.987+0000 7faf327fc640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faf2006b3b0 0x7faf2006d870 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faf28000c00 tx=0x7faf28039000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:09.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:09.990+0000 7faf1bfff640 1 -- 172.21.15.172:0/1809127170 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf24053e20 con 0x7faf340c3930 2024-03-16T19:31:10.052 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:10.072+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:10.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.079+0000 7ff1adf56640 1 Processor -- start 2024-03-16T19:31:10.059 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.079+0000 7ff1adf56640 1 -- start start 2024-03-16T19:31:10.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.080+0000 7ff1adf56640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 0x7ff1a81592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.060 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.080+0000 7ff1adf56640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff1a80ca750 0x7ff1a80cec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.080+0000 7ff1a6ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff1a80ca750 0x7ff1a80cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.080+0000 7ff1a77fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 0x7ff1a81592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.080+0000 7ff1a7fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff1a815f7a0 0x7ff1a8070880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:47668/0 (socket says 172.21.15.172:47668) 2024-03-16T19:31:10.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.080+0000 7ff1a7fff640 1 -- 172.21.15.172:0/89311212 learned_addr learned my addr 172.21.15.172:0/89311212 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:10.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.081+0000 7ff1adf56640 1 -- 172.21.15.172:0/89311212 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a815f310 con 0x7ff1a815f7a0 2024-03-16T19:31:10.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.081+0000 7ff1adf56640 1 -- 172.21.15.172:0/89311212 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a806bfe0 con 0x7ff1a80ca750 2024-03-16T19:31:10.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.081+0000 7ff1adf56640 1 -- 172.21.15.172:0/89311212 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a815efc0 con 0x7ff1a8158ef0 2024-03-16T19:31:10.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.081+0000 7ff1a67fc640 1 -- 172.21.15.172:0/89311212 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098431861 0 0) 0x7ff1a815f310 con 0x7ff1a815f7a0 2024-03-16T19:31:10.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.081+0000 7ff1a67fc640 1 -- 172.21.15.172:0/89311212 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff190003520 con 0x7ff1a815f7a0 2024-03-16T19:31:10.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.081+0000 7ff1a77fe640 1 -- 172.21.15.172:0/89311212 >> v2:172.21.15.174:3300/0 conn(0x7ff1a80ca750 msgr2=0x7ff1a80cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:10.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.081+0000 7ff1a77fe640 1 --2- 172.21.15.172:0/89311212 >> v2:172.21.15.174:3300/0 conn(0x7ff1a80ca750 0x7ff1a80cec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.081+0000 7ff1a77fe640 1 -- 172.21.15.172:0/89311212 >> v1:172.21.15.172:6789/0 conn(0x7ff1a815f7a0 legacy=0x7ff1a8070880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:10.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.081+0000 7ff1a77fe640 1 -- 172.21.15.172:0/89311212 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a8162970 con 0x7ff1a8158ef0 2024-03-16T19:31:10.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.082+0000 7ff1a77fe640 1 --2- 172.21.15.172:0/89311212 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 0x7ff1a81592d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff1980097a0 tx=0x7ff19802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4073887040380915 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.082+0000 7ff1a67fc640 1 -- 172.21.15.172:0/89311212 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19802d040 con 0x7ff1a8158ef0 2024-03-16T19:31:10.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1a67fc640 1 -- 172.21.15.172:0/89311212 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff198031810 con 0x7ff1a8158ef0 2024-03-16T19:31:10.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1a67fc640 1 -- 172.21.15.172:0/89311212 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198031ba0 con 0x7ff1a8158ef0 2024-03-16T19:31:10.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1adf56640 1 -- 172.21.15.172:0/89311212 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 msgr2=0x7ff1a81592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:10.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1adf56640 1 --2- 172.21.15.172:0/89311212 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 0x7ff1a81592d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff1980097a0 tx=0x7ff19802b8b0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1adf56640 1 -- 172.21.15.172:0/89311212 shutdown_connections 2024-03-16T19:31:10.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1adf56640 1 --2- 172.21.15.172:0/89311212 >> v2:172.21.15.174:3300/0 conn(0x7ff1a80ca750 0x7ff1a80cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1adf56640 1 --2- 172.21.15.172:0/89311212 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 0x7ff1a81592d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1adf56640 1 -- 172.21.15.172:0/89311212 >> 172.21.15.172:0/89311212 conn(0x7ff1a806e880 msgr2=0x7ff1a806ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:10.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1adf56640 1 -- 172.21.15.172:0/89311212 shutdown_connections 2024-03-16T19:31:10.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.083+0000 7ff1adf56640 1 -- 172.21.15.172:0/89311212 wait complete. 2024-03-16T19:31:10.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.085+0000 7ff1adf56640 1 Processor -- start 2024-03-16T19:31:10.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.085+0000 7ff1adf56640 1 -- start start 2024-03-16T19:31:10.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.086+0000 7ff1adf56640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a80ca750 0x7ff1a80ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1adf56640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 0x7ff1a80cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1adf56640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1a815f7a0 0x7ff1a81f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a80ca750 0x7ff1a80ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a80ca750 0x7ff1a80ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40834/0 (socket says 172.21.15.172:40834) 2024-03-16T19:31:10.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a6ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 0x7ff1a80cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a6ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 0x7ff1a80cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59012/0 (socket says 172.21.15.172:59012) 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1a815f7a0 0x7ff1a81f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1a815f7a0 0x7ff1a81f8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42534/0 (socket says 172.21.15.172:42534) 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a77fe640 1 -- 172.21.15.172:0/3308014752 learned_addr learned my addr 172.21.15.172:0/3308014752 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1adf56640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a815bff0 con 0x7ff1a815f7a0 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1adf56640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a8161eb0 con 0x7ff1a80ca750 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1adf56640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80ce1c0 con 0x7ff1a8158ef0 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a77fe640 1 -- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 msgr2=0x7ff1a80cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a77fe640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a8158ef0 0x7ff1a80cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.087+0000 7ff1a77fe640 1 -- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1a815f7a0 msgr2=0x7ff1a81f8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.088+0000 7ff1a77fe640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1a815f7a0 0x7ff1a81f8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.088+0000 7ff1a77fe640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff198009400 con 0x7ff1a80ca750 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.088+0000 7ff1a7fff640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1a815f7a0 0x7ff1a81f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.088+0000 7ff1a77fe640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a80ca750 0x7ff1a80ceb70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff1980098c0 tx=0x7ff1980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.088+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19803ad60 con 0x7ff1a80ca750 2024-03-16T19:31:10.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.088+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff198044400 con 0x7ff1a80ca750 2024-03-16T19:31:10.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.088+0000 7ff1adf56640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a80cf0b0 con 0x7ff1a80ca750 2024-03-16T19:31:10.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.088+0000 7ff1adf56640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a80b4190 con 0x7ff1a80ca750 2024-03-16T19:31:10.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.088+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1980445b0 con 0x7ff1a80ca750 2024-03-16T19:31:10.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.089+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff1a8068d20 con 0x7ff1a80ca750 2024-03-16T19:31:10.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.090+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff19804d440 con 0x7ff1a80ca750 2024-03-16T19:31:10.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.091+0000 7ff18ffff640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff19406b3b0 0x7ff19406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.091+0000 7ff1a6ffd640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff19406b3b0 0x7ff19406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.091+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6002+0+0 (secure 0 0 0) 0x7ff1980abe60 con 0x7ff1a80ca750 2024-03-16T19:31:10.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.091+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v68) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff1980ae480 con 0x7ff1a80ca750 2024-03-16T19:31:10.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.092+0000 7ff18dffb640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ff1a81f87b0 0x7ff1a8151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.092+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1a8070f90 con 0x7ff1a81f87b0 2024-03-16T19:31:10.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.092+0000 7ff1a6ffd640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff19406b3b0 0x7ff19406d870 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff1a81544a0 tx=0x7ff19c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.092+0000 7ff1a7fff640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ff1a81f87b0 0x7ff1a8151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.092+0000 7ff1a7fff640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ff1a81f87b0 0x7ff1a8151460 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.093+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff1a000f1d0 con 0x7ff1a81f87b0 2024-03-16T19:31:10.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.123+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1a8163210 con 0x7ff1a81f87b0 2024-03-16T19:31:10.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.125+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff1a000f620 con 0x7ff1a81f87b0 2024-03-16T19:31:10.106 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:10.106 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:10.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.126+0000 7ff18dffb640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ff1a80cc0d0 0x7ff1a80cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.126+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1a8151eb0 con 0x7ff1a80cc0d0 2024-03-16T19:31:10.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.126+0000 7ff1a77fe640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ff1a80cc0d0 0x7ff1a80cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.126+0000 7ff1a77fe640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ff1a80cc0d0 0x7ff1a80cc4b0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.127+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff1a8151eb0 con 0x7ff1a80cc0d0 2024-03-16T19:31:10.108 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:10.108 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:10.117 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:10.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.155+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1a80ce800 con 0x7ff1a80cc0d0 2024-03-16T19:31:10.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.157+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff1a80ce800 con 0x7ff1a80cc0d0 2024-03-16T19:31:10.137 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:10.138 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:10.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.157+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=69}) v3 -- 0x7ff1a80cbb70 con 0x7ff1a80ca750 2024-03-16T19:31:10.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.157+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff1a8157f70 con 0x7ff1a80ca750 2024-03-16T19:31:10.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.158+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff19802d040 con 0x7ff1a80ca750 2024-03-16T19:31:10.139 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:10.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.158+0000 7ff18dffb640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ff1a8061930 0x7ff1a8061d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.158+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1a80ccf00 con 0x7ff1a8061930 2024-03-16T19:31:10.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.158+0000 7ff1a6ffd640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ff1a8061930 0x7ff1a8061d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.158+0000 7ff1a6ffd640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ff1a8061930 0x7ff1a8061d10 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.159+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff1a80ccf00 con 0x7ff1a8061930 2024-03-16T19:31:10.154 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:10.174+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:10.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.184+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1a80624a0 con 0x7ff1a8061930 2024-03-16T19:31:10.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.186+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff1a80624a0 con 0x7ff1a8061930 2024-03-16T19:31:10.166 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:10.167 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:10.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.186+0000 7ff18dffb640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff1a81528e0 0x7ff1a8152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.186+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1a8157c90 con 0x7ff1a81528e0 2024-03-16T19:31:10.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.186+0000 7ff1a7fff640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff1a81528e0 0x7ff1a8152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.187+0000 7ff1a7fff640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff1a81528e0 0x7ff1a8152d40 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.187+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff1a8157c90 con 0x7ff1a81528e0 2024-03-16T19:31:10.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.215+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1a806ee90 con 0x7ff1a81528e0 2024-03-16T19:31:10.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.217+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff1a806ee90 con 0x7ff1a81528e0 2024-03-16T19:31:10.197 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:10.198 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:10.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.217+0000 7ff18dffb640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ff1a8207740 0x7ff1a8209b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.217+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1a820c900 con 0x7ff1a8207740 2024-03-16T19:31:10.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.217+0000 7ff1a77fe640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ff1a8207740 0x7ff1a8209b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.218+0000 7ff1a77fe640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ff1a8207740 0x7ff1a8209b30 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.219+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff1a820c900 con 0x7ff1a8207740 2024-03-16T19:31:10.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.248+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1a820a070 con 0x7ff1a8207740 2024-03-16T19:31:10.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.250+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff1a820a070 con 0x7ff1a8207740 2024-03-16T19:31:10.230 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:10.231 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:10.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.250+0000 7ff18dffb640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff1a820a8c0 0x7ff1a820ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.250+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1a820fcd0 con 0x7ff1a820a8c0 2024-03-16T19:31:10.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.251+0000 7ff1a6ffd640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff1a820a8c0 0x7ff1a820ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.251+0000 7ff1a6ffd640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff1a820a8c0 0x7ff1a820ad40 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.252+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff1a820fcd0 con 0x7ff1a820a8c0 2024-03-16T19:31:10.239 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:10.239 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:10.239 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:10.239 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:10.248 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:10.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.280+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1a820b280 con 0x7ff1a820a8c0 2024-03-16T19:31:10.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.282+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff1a820b280 con 0x7ff1a820a8c0 2024-03-16T19:31:10.262 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.282+0000 7ff18dffb640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff1a820c620 0x7ff1a820b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.282+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1a820c0c0 con 0x7ff1a820c620 2024-03-16T19:31:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.282+0000 7ff1a7fff640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff1a820c620 0x7ff1a820b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:10.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.283+0000 7ff1a7fff640 1 --2- 172.21.15.172:0/3308014752 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff1a820c620 0x7ff1a820b960 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:10.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.284+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff1a820c0c0 con 0x7ff1a820c620 2024-03-16T19:31:10.292 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:10.312+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:10.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.312+0000 7ff18dffb640 1 -- 172.21.15.172:0/3308014752 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1a820c0c0 con 0x7ff1a820c620 2024-03-16T19:31:10.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.314+0000 7ff18ffff640 1 -- 172.21.15.172:0/3308014752 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff1a820c0c0 con 0x7ff1a820c620 2024-03-16T19:31:10.294 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:10.294 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:10.300 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:10.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.362+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7faf3406bda0 con 0x7faf340c3930 2024-03-16T19:31:10.356 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:10.376+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:10.360 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:10.360 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:10.360 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:10.360 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:10.360 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:10.369 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:10.456 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:10.476+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:10.517 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.179499070000006, 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:10.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:31:10.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:31:10.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.179419952000003, 2024-03-16T19:31:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.178805827999994, 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:10.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:10.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.530 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.178365759000002, 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.534 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.178124169, 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:10.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:10.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:10.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:31:10.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:31:10.539 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.539 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:10.539 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.539 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.539 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:10.539 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:10.540 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:10.558 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:10.578+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:10.658 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:10.678+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:10.687 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:10.687 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:10.687 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:10.687 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:10.696 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:10.759 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:10.778+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.826+0000 7faf1bfff640 1 -- 172.21.15.172:0/1809127170 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v69) v1 ==== 97+0+0 (secure 0 0 0) 0x7faf24017090 con 0x7faf340c3930 2024-03-16T19:31:10.806 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:31:10.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faf2006b3b0 msgr2=0x7faf2006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faf2006b3b0 0x7faf2006d870 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faf28000c00 tx=0x7faf28039000 comp rx=0 tx=0).stop 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf340c3930 msgr2=0x7faf341ec270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf340c3930 0x7faf341ec270 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf2400c3a0 tx=0x7faf2400c860 comp rx=0 tx=0).stop 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 shutdown_connections 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faf2006b3b0 0x7faf2006d870 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf34151910 0x7faf34145a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf340c3ea0 0x7faf341454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 --2- 172.21.15.172:0/1809127170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf340c3930 0x7faf341ec270 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 >> 172.21.15.172:0/1809127170 conn(0x7faf3406db70 msgr2=0x7faf3414f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 shutdown_connections 2024-03-16T19:31:10.810 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:10.829+0000 7faf3986b640 1 -- 172.21.15.172:0/1809127170 wait complete. 2024-03-16T19:31:10.818 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:10.818 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:10.818 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:10.818 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:10.818 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:10.827 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:10.868 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:10.887+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:10.961 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:10.981+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:10.979 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:10.979 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:10.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:10.979 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:10.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.979 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:10.979 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:10.979 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.640798781000001, 2024-03-16T19:31:10.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:31:10.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:31:10.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:10.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.640774608000001, 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:10.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:31:10.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:10.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:31:10.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.640768781000006, 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:31:10.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:31:10.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.640764868999995, 2024-03-16T19:31:10.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:31:10.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:31:10.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.001 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.640758887000004, 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:31:11.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.640755221000006, 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:31:11.007 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:31:11.012 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.640744561000005, 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:31:11.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.640740672000007, 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:31:11.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.637223981000005, 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:11.021 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.637153545999993, 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:31:11.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.636771198999995, 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:31:11.029 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:11.033 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.635004916, 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:31:11.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:31:11.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.633191994000001, 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:31:11.037 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:31:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.040 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 99.632336245000005, 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:11.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:31:11.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:31:11.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:31:11.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 94.626475479000007, 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:31:11.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:31:11.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 91.193924271, 2024-03-16T19:31:11.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:11.055 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 89.626430565000007, 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:31:11.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:31:11.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:31:11.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.060 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 88.118818313999995, 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:11.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:31:11.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 51.621955088999997, 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.069 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.621419173, 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:31:11.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:31:11.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:31:11.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.074 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.074 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:11.075 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:11.076 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:11.095+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:11.176 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:11.195+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:11.186 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:11.186 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:11.186 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:11.187 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:11.195 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:11.279 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:11.298+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:11.342 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:11.342 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:11.342 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:11.342 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:11.342 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:11.350 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:11.392 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:11.411+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:11.478 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:11.497+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:11.499 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:11.499 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 115.288598838, 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:31:11.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:31:11.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 114.911903291, 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:31:11.509 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 114.13592684699999, 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:31:11.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:31:11.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 114.135842434, 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:31:11.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:31:11.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:31:11.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 114.135835078, 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.521 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:31:11.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:31:11.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.943295886, 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:31:11.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.534 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.94329007899999, 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:31:11.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:11.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:11.540 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.94317194600001, 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:31:11.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:31:11.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.943076749, 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.546 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.551 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.943072738, 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:31:11.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:31:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:31:11.556 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.524140864, 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:31:11.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:31:11.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.562 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.52409149499999, 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:11.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:31:11.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:31:11.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:31:11.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:31:11.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.52401266299999, 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.568 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:31:11.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.573 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.524009452, 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.524006499, 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.579 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:31:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.523891525, 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:31:11.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:31:11.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.523888173, 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.590 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:31:11.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.595 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.523885057, 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.523824588, 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.601 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:31:11.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:31:11.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 112.91514453800001, 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:31:11.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:31:11.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:11.612 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:11.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:11.614 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:11.632+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:11.714 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:11.733+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:11.787 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:11.788 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:11.788 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:11.788 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:11.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:11.815 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:11.834+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:11.916 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:11.935+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:11.971 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:11.971 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:11.971 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:11.971 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:11.971 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:11.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:12.017 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:12.036+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:12.119 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:12.138+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:12.155 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.58086021, 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:31:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.790634039, 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:31:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.789873804, 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:31:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:31:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:31:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.789059849, 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:31:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.788419924, 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:31:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.787761827, 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:31:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:31:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.787229252, 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:31:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:31:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.729057349, 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:31:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.72845515500001, 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:31:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:31:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.72781221, 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:31:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.727308249, 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:31:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 113.569161983, 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:31:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:31:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:31:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 113.56863184700001, 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:12.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:31:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.212 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.816662588, 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:31:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:31:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.816638052, 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:31:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:31:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.816633522, 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:31:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:31:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:31:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.81662371, 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:12.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:31:12.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:12.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.81661985700001, 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.816616541, 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.815420958, 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:12.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:12.250 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:12.268+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:12.300 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:31:12.379 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:12.398+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:12.435 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:12.435 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:12.435 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:12.435 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:12.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.466+0000 7f3beb6a7640 1 Processor -- start 2024-03-16T19:31:12.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:12.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.466+0000 7f3beb6a7640 1 -- start start 2024-03-16T19:31:12.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.467+0000 7f3beb6a7640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3be4162fb0 0x7f3be40b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.467+0000 7f3beb6a7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 0x7f3be40b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.467+0000 7f3be8c42640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3be4162fb0 0x7f3be40b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.467+0000 7f3be9443640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3be40ca2d0 0x7f3be40d0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:47682/0 (socket says 172.21.15.172:47682) 2024-03-16T19:31:12.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.467+0000 7f3be9443640 1 -- 172.21.15.172:0/2278697768 learned_addr learned my addr 172.21.15.172:0/2278697768 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:12.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.467+0000 7f3be9c44640 1 --2- 172.21.15.172:0/2278697768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 0x7f3be40b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.467+0000 7f3beb6a7640 1 -- 172.21.15.172:0/2278697768 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be40b3c80 con 0x7f3be40ca2d0 2024-03-16T19:31:12.450 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.467+0000 7f3beb6a7640 1 -- 172.21.15.172:0/2278697768 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be4162b90 con 0x7f3be4162fb0 2024-03-16T19:31:12.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.467+0000 7f3beb6a7640 1 -- 172.21.15.172:0/2278697768 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be4163420 con 0x7f3be40b3310 2024-03-16T19:31:12.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.468+0000 7f3bdbfff640 1 -- 172.21.15.172:0/2278697768 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197453744 0 0) 0x7f3be0001110 con 0x7f3be40ca2d0 2024-03-16T19:31:12.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.468+0000 7f3bdbfff640 1 -- 172.21.15.172:0/2278697768 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd0003520 con 0x7f3be40ca2d0 2024-03-16T19:31:12.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.468+0000 7f3be8c42640 1 -- 172.21.15.172:0/2278697768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 msgr2=0x7f3be40b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:12.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.468+0000 7f3be8c42640 1 --2- 172.21.15.172:0/2278697768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 0x7f3be40b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.468+0000 7f3be8c42640 1 -- 172.21.15.172:0/2278697768 >> v1:172.21.15.172:6789/0 conn(0x7f3be40ca2d0 legacy=0x7f3be40d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:12.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.468+0000 7f3be8c42640 1 -- 172.21.15.172:0/2278697768 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be40ca6b0 con 0x7f3be4162fb0 2024-03-16T19:31:12.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.468+0000 7f3be9c44640 1 --2- 172.21.15.172:0/2278697768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 0x7f3be40b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:12.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.469+0000 7f3be8c42640 1 --2- 172.21.15.172:0/2278697768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be4162fb0 0x7f3be40b3e40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40097a0 tx=0x7f3bd402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef775f054e250fdc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.469+0000 7f3bdbfff640 1 -- 172.21.15.172:0/2278697768 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4031780 con 0x7f3be4162fb0 2024-03-16T19:31:12.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.469+0000 7f3bdbfff640 1 -- 172.21.15.172:0/2278697768 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd40318e0 con 0x7f3be4162fb0 2024-03-16T19:31:12.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.469+0000 7f3bdbfff640 1 -- 172.21.15.172:0/2278697768 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4031c50 con 0x7f3be4162fb0 2024-03-16T19:31:12.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.470+0000 7f3beb6a7640 1 -- 172.21.15.172:0/2278697768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be4162fb0 msgr2=0x7f3be40b3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:12.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.470+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/2278697768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be4162fb0 0x7f3be40b3e40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40097a0 tx=0x7f3bd402b820 comp rx=0 tx=0).stop 2024-03-16T19:31:12.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.470+0000 7f3beb6a7640 1 -- 172.21.15.172:0/2278697768 shutdown_connections 2024-03-16T19:31:12.453 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.470+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/2278697768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 0x7f3be40b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.470+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/2278697768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be4162fb0 0x7f3be40b3e40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.470+0000 7f3beb6a7640 1 -- 172.21.15.172:0/2278697768 >> 172.21.15.172:0/2278697768 conn(0x7f3be406bf60 msgr2=0x7f3be406d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:12.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.471+0000 7f3beb6a7640 1 -- 172.21.15.172:0/2278697768 shutdown_connections 2024-03-16T19:31:12.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.471+0000 7f3beb6a7640 1 -- 172.21.15.172:0/2278697768 wait complete. 2024-03-16T19:31:12.454 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:12.471+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:12.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.472+0000 7f3beb6a7640 1 Processor -- start 2024-03-16T19:31:12.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.473+0000 7f3beb6a7640 1 -- start start 2024-03-16T19:31:12.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3beb6a7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 0x7f3be40c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3beb6a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3be40ca2d0 0x7f3be415eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3beb6a7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be4162fb0 0x7f3be415a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3be9443640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 0x7f3be40c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3be9443640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 0x7f3be40c2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59032/0 (socket says 172.21.15.172:59032) 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3be8c42640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3be40ca2d0 0x7f3be415eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3be8c42640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3be40ca2d0 0x7f3be415eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40850/0 (socket says 172.21.15.172:40850) 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3be9443640 1 -- 172.21.15.172:0/1328626357 learned_addr learned my addr 172.21.15.172:0/1328626357 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be415cdf0 con 0x7f3be4162fb0 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be415b910 con 0x7f3be40ca2d0 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be415c490 con 0x7f3be40b3310 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be4162fb0 0x7f3be415a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.474+0000 7f3be8c42640 1 -- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 msgr2=0x7f3be40c2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.475+0000 7f3be8c42640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40b3310 0x7f3be40c2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.475+0000 7f3be8c42640 1 -- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be4162fb0 msgr2=0x7f3be415a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:12.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.475+0000 7f3be8c42640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be4162fb0 0x7f3be415a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.475+0000 7f3be8c42640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc009a70 con 0x7f3be40ca2d0 2024-03-16T19:31:12.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.475+0000 7f3be8c42640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3be40ca2d0 0x7f3be415eb80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3be406d270 tx=0x7f3bd40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.475+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4003a20 con 0x7f3be40ca2d0 2024-03-16T19:31:12.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.475+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd4040070 con 0x7f3be40ca2d0 2024-03-16T19:31:12.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.475+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4003dc0 con 0x7f3be40ca2d0 2024-03-16T19:31:12.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.475+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd4009400 con 0x7f3be40ca2d0 2024-03-16T19:31:12.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.476+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be415bb60 con 0x7f3be40ca2d0 2024-03-16T19:31:12.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.477+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3be4068d20 con 0x7f3be40ca2d0 2024-03-16T19:31:12.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.478+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3bd403c400 con 0x7f3be40ca2d0 2024-03-16T19:31:12.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.478+0000 7f3bd9ffb640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3bc806b3b0 0x7f3bc806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.478+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f3bd4042070 con 0x7f3be40ca2d0 2024-03-16T19:31:12.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.479+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v69) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3bd4079ac0 con 0x7f3be40ca2d0 2024-03-16T19:31:12.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.479+0000 7f3be9443640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3bc806b3b0 0x7f3bc806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.479+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3be415a9f0 0x7f3be4160380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.479+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3be415a9f0 0x7f3be4160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.479+0000 7f3be9443640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3bc806b3b0 0x7f3bc806d870 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3be0001370 tx=0x7f3be0007c70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.479+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3be40cc920 con 0x7f3be415a9f0 2024-03-16T19:31:12.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.480+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3be415a9f0 0x7f3be4160380 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.481+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3bdc00f040 con 0x7f3be415a9f0 2024-03-16T19:31:12.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.510+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3be415c5d0 con 0x7f3be415a9f0 2024-03-16T19:31:12.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.513+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3bdc011630 con 0x7f3be415a9f0 2024-03-16T19:31:12.494 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:12.494 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:12.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.513+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3be40c2780 0x7f3be41f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.513+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3be40688d0 con 0x7f3be40c2780 2024-03-16T19:31:12.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.513+0000 7f3be9443640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3be40c2780 0x7f3be41f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.513+0000 7f3be9443640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3be40c2780 0x7f3be41f8280 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.514+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3be40688d0 con 0x7f3be40c2780 2024-03-16T19:31:12.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.529+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3be40c8160 con 0x7f3be40c2780 2024-03-16T19:31:12.512 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.531+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3be40c8160 con 0x7f3be40c2780 2024-03-16T19:31:12.512 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:12.513 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:12.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.531+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=70}) v3 -- 0x7f3be40681c0 con 0x7f3be40ca2d0 2024-03-16T19:31:12.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.531+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3be40cbd00 con 0x7f3be40ca2d0 2024-03-16T19:31:12.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.531+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bd4031eb0 con 0x7f3be40ca2d0 2024-03-16T19:31:12.513 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:12.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.532+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3be4159100 0x7f3be4159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.532+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3be40d08b0 con 0x7f3be4159100 2024-03-16T19:31:12.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.532+0000 7f3be8c42640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3be4159100 0x7f3be4159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.532+0000 7f3be8c42640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3be4159100 0x7f3be4159510 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.514 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.533+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3be40d08b0 con 0x7f3be4159100 2024-03-16T19:31:12.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.548+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3be4159a50 con 0x7f3be4159100 2024-03-16T19:31:12.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.549+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3be4159a50 con 0x7f3be4159100 2024-03-16T19:31:12.530 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:12.531 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:12.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.549+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3be4061e40 0x7f3be40622a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.550+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3be40627e0 con 0x7f3be4061e40 2024-03-16T19:31:12.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.550+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3be4061e40 0x7f3be40622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.550+0000 7f3be9c44640 1 -- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3be4061e40 msgr2=0x7f3be40622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 24 2024-03-16T19:31:12.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.550+0000 7f3be9c44640 1 -- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3be4061e40 msgr2=0x7f3be40622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:31:12.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.550+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3be4061e40 0x7f3be40622a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:31:12.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.550+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3be4061e40 0x7f3be40622a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:31:12.553 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:12.572+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:12.599 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:12.599 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:12.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:12.692 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:12.711+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:12.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.751+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3be4061e40 0x7f3be40622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.732 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.751+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3be4061e40 0x7f3be40622a0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.752+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3be40627e0 con 0x7f3be4061e40 2024-03-16T19:31:12.755 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:12.774+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:12.757 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:12.758 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:12.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.781+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3be415da60 con 0x7f3be4061e40 2024-03-16T19:31:12.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.782+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3be415da60 con 0x7f3be4061e40 2024-03-16T19:31:12.764 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:12.764 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:12.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.783+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3be415def0 0x7f3be415e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.783+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3be420a610 con 0x7f3be415def0 2024-03-16T19:31:12.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.783+0000 7f3be9443640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3be415def0 0x7f3be415e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.783+0000 7f3be9443640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3be415def0 0x7f3be415e330 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.784+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3be420a610 con 0x7f3be415def0 2024-03-16T19:31:12.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:12.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.813+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3be420a5e0 con 0x7f3be415def0 2024-03-16T19:31:12.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.814+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3be420a5e0 con 0x7f3be415def0 2024-03-16T19:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:12.797 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:12.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.815+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3be4209930 0x7f3be4209d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.815+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3be420d9a0 con 0x7f3be4209930 2024-03-16T19:31:12.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.815+0000 7f3be8c42640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3be4209930 0x7f3be4209d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.815+0000 7f3be8c42640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3be4209930 0x7f3be4209d90 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.816+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3be420d9a0 con 0x7f3be4209930 2024-03-16T19:31:12.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:31:12.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.845+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3be420c820 con 0x7f3be4209930 2024-03-16T19:31:12.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.847+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3be420c820 con 0x7f3be4209930 2024-03-16T19:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:12.828 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:12.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.847+0000 7f3beb6a7640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3be420cea0 0x7f3be420d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.847+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3be4210d30 con 0x7f3be420cea0 2024-03-16T19:31:12.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.848+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3be420cea0 0x7f3be420d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.848+0000 7f3be9c44640 1 --2- 172.21.15.172:0/1328626357 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3be420cea0 0x7f3be420d2e0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.849+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3be4210d30 con 0x7f3be420cea0 2024-03-16T19:31:12.855 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:12.875+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:12.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.879+0000 7f3beb6a7640 1 -- 172.21.15.172:0/1328626357 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3be402bde0 con 0x7f3be420cea0 2024-03-16T19:31:12.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:12.881+0000 7f3bd9ffb640 1 -- 172.21.15.172:0/1328626357 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3be402bde0 con 0x7f3be420cea0 2024-03-16T19:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:12.863 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:12.867 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:12.920 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:12.920 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:12.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:12.957 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:12.976+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:12.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.001+0000 7f1c4f0c9640 1 Processor -- start 2024-03-16T19:31:12.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.001+0000 7f1c4f0c9640 1 -- start start 2024-03-16T19:31:12.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.003+0000 7f1c4f0c9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c4815c340 0x7f1c480c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.003+0000 7f1c4f0c9640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c480c9af0 0x7f1c4806e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.003+0000 7f1c4ce65640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1c48152ab0 0x7f1c4815bc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:47694/0 (socket says 172.21.15.172:47694) 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.003+0000 7f1c4ce65640 1 -- 172.21.15.172:0/4090209875 learned_addr learned my addr 172.21.15.172:0/4090209875 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.003+0000 7f1c4d666640 1 --2- 172.21.15.172:0/4090209875 >> v2:172.21.15.172:3300/0 conn(0x7f1c480c9af0 0x7f1c4806e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.003+0000 7f1c47fff640 1 --2- 172.21.15.172:0/4090209875 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c4815c340 0x7f1c480c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.003+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/4090209875 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c48152e90 con 0x7f1c48152ab0 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.003+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/4090209875 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c480c85e0 con 0x7f1c480c9af0 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.003+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/4090209875 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480c30e0 con 0x7f1c4815c340 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.004+0000 7f1c477fe640 1 -- 172.21.15.172:0/4090209875 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266421122 0 0) 0x7f1c3c001110 con 0x7f1c48152ab0 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.004+0000 7f1c477fe640 1 -- 172.21.15.172:0/4090209875 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c38003520 con 0x7f1c48152ab0 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.004+0000 7f1c477fe640 1 -- 172.21.15.172:0/4090209875 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3406747912 0 0) 0x7f1c38003520 con 0x7f1c48152ab0 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.004+0000 7f1c4d666640 1 -- 172.21.15.172:0/4090209875 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c4815c340 msgr2=0x7f1c480c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:12.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.004+0000 7f1c4d666640 1 --2- 172.21.15.172:0/4090209875 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c4815c340 0x7f1c480c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.004+0000 7f1c4d666640 1 -- 172.21.15.172:0/4090209875 >> v1:172.21.15.172:6789/0 conn(0x7f1c48152ab0 legacy=0x7f1c4815bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:12.987 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.004+0000 7f1c4d666640 1 -- 172.21.15.172:0/4090209875 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480c33c0 con 0x7f1c480c9af0 2024-03-16T19:31:12.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.005+0000 7f1c4d666640 1 --2- 172.21.15.172:0/4090209875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c480c9af0 0x7f1c4806e300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009d90 tx=0x7f1c4002bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823c7081130cd96c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.005+0000 7f1c477fe640 1 -- 172.21.15.172:0/4090209875 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c001800 con 0x7f1c480c9af0 2024-03-16T19:31:12.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.005+0000 7f1c477fe640 1 -- 172.21.15.172:0/4090209875 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4002c8a0 con 0x7f1c480c9af0 2024-03-16T19:31:12.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.005+0000 7f1c477fe640 1 -- 172.21.15.172:0/4090209875 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4002cc30 con 0x7f1c480c9af0 2024-03-16T19:31:12.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.006+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/4090209875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c480c9af0 msgr2=0x7f1c4806e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:12.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.006+0000 7f1c4f0c9640 1 --2- 172.21.15.172:0/4090209875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c480c9af0 0x7f1c4806e300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009d90 tx=0x7f1c4002bd70 comp rx=0 tx=0).stop 2024-03-16T19:31:12.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.007+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/4090209875 shutdown_connections 2024-03-16T19:31:12.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.007+0000 7f1c4f0c9640 1 --2- 172.21.15.172:0/4090209875 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c4815c340 0x7f1c480c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.007+0000 7f1c4f0c9640 1 --2- 172.21.15.172:0/4090209875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c480c9af0 0x7f1c4806e300 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.007+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/4090209875 >> 172.21.15.172:0/4090209875 conn(0x7f1c48070810 msgr2=0x7f1c48070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:12.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.007+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/4090209875 shutdown_connections 2024-03-16T19:31:12.989 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.007+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/4090209875 wait complete. 2024-03-16T19:31:12.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.008+0000 7f1c4f0c9640 1 Processor -- start 2024-03-16T19:31:12.990 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.009+0000 7f1c4f0c9640 1 -- start start 2024-03-16T19:31:12.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.010+0000 7f1c4f0c9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c480c9af0 0x7f1c480cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.010+0000 7f1c4f0c9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c48152ab0 0x7f1c48153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.010+0000 7f1c4f0c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c480ca9a0 0x7f1c481ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.010+0000 7f1c4ce65640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c480c9af0 0x7f1c480cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.010+0000 7f1c4ce65640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c480c9af0 0x7f1c480cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59044/0 (socket says 172.21.15.172:59044) 2024-03-16T19:31:12.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.010+0000 7f1c47fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c48152ab0 0x7f1c48153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.010+0000 7f1c47fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c48152ab0 0x7f1c48153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42574/0 (socket says 172.21.15.172:42574) 2024-03-16T19:31:12.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4ce65640 1 -- 172.21.15.172:0/3006131372 learned_addr learned my addr 172.21.15.172:0/3006131372 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:12.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4d666640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c480ca9a0 0x7f1c481ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48153c80 con 0x7f1c48152ab0 2024-03-16T19:31:12.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c481f2590 con 0x7f1c480ca9a0 2024-03-16T19:31:12.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c481f2100 con 0x7f1c480c9af0 2024-03-16T19:31:12.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4ce65640 1 -- 172.21.15.172:0/3006131372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c480ca9a0 msgr2=0x7f1c481ee0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4ce65640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c480ca9a0 0x7f1c481ee0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4ce65640 1 -- 172.21.15.172:0/3006131372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c48152ab0 msgr2=0x7f1c48153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4ce65640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c48152ab0 0x7f1c48153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4ce65640 1 -- 172.21.15.172:0/3006131372 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c40009a70 con 0x7f1c480c9af0 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.011+0000 7f1c4d666640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c480ca9a0 0x7f1c481ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.012+0000 7f1c4ce65640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c480c9af0 0x7f1c480cd190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00c750 tx=0x7f1c3c00cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.012+0000 7f1c457fa640 1 -- 172.21.15.172:0/3006131372 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c005ad0 con 0x7f1c480c9af0 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.012+0000 7f1c457fa640 1 -- 172.21.15.172:0/3006131372 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c005c50 con 0x7f1c480c9af0 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.012+0000 7f1c457fa640 1 -- 172.21.15.172:0/3006131372 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c007070 con 0x7f1c480c9af0 2024-03-16T19:31:12.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.012+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c481539a0 con 0x7f1c480c9af0 2024-03-16T19:31:12.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.012+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c481f1ab0 con 0x7f1c480c9af0 2024-03-16T19:31:12.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.013+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48075a70 con 0x7f1c480c9af0 2024-03-16T19:31:12.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.014+0000 7f1c457fa640 1 -- 172.21.15.172:0/3006131372 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1c3c0071d0 con 0x7f1c480c9af0 2024-03-16T19:31:12.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.015+0000 7f1c457fa640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2c06b3e0 0x7f1c2c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:12.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.015+0000 7f1c457fa640 1 -- 172.21.15.172:0/3006131372 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f1c3c0892d0 con 0x7f1c480c9af0 2024-03-16T19:31:12.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.015+0000 7f1c47fff640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2c06b3e0 0x7f1c2c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:12.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.016+0000 7f1c47fff640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2c06b3e0 0x7f1c2c06d8a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1c4815a010 tx=0x7f1c34009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:13.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.019+0000 7f1c457fa640 1 -- 172.21.15.172:0/3006131372 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c3c053e20 con 0x7f1c480c9af0 2024-03-16T19:31:13.045 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:13.045 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:13.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:13.058 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:13.077+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:13.159 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:13.178+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:13.164 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:13.164 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:13.174 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:13.260 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:13.279+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:13.282 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:13.282 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:13.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.308+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f1c48152e90 con 0x7f1c480c9af0 2024-03-16T19:31:13.291 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:13.361 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:13.380+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:13.437 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:13.437 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:13.446 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:13.462 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:13.481+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:13.556 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:13.556 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:13.563 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:13.583+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:13.565 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:13.664 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:13.683+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:13.671 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:13.672 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:13.680 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:13.765 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:13.784+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:13.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.830+0000 7f1c457fa640 1 -- 172.21.15.172:0/3006131372 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v70) v1 ==== 91+0+0 (secure 0 0 0) 0x7f1c48152e90 con 0x7f1c480c9af0 2024-03-16T19:31:13.811 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:31:13.813 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2c06b3e0 msgr2=0x7f1c2c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2c06b3e0 0x7f1c2c06d8a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1c4815a010 tx=0x7f1c34009190 comp rx=0 tx=0).stop 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c480c9af0 msgr2=0x7f1c480cd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c480c9af0 0x7f1c480cd190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00c750 tx=0x7f1c3c00cc10 comp rx=0 tx=0).stop 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 shutdown_connections 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c480ca9a0 0x7f1c481ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c48152ab0 0x7f1c48153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2c06b3e0 0x7f1c2c06d8a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 --2- 172.21.15.172:0/3006131372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c480c9af0 0x7f1c480cd190 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 >> 172.21.15.172:0/3006131372 conn(0x7f1c48070810 msgr2=0x7f1c480cbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 shutdown_connections 2024-03-16T19:31:13.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:13.832+0000 7f1c4f0c9640 1 -- 172.21.15.172:0/3006131372 wait complete. 2024-03-16T19:31:13.830 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:13.830 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:13.839 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:13.884 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:13.903+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:13.924 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-16T19:31:13.924 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:31:13.924 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-16T19:31:13.924 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-16T19:31:13.946 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:13.946 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:13.954 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:13.996 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:14.015+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:14.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.056+0000 7f32a8ffd640 1 Processor -- start 2024-03-16T19:31:14.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.056+0000 7f32a8ffd640 1 -- start start 2024-03-16T19:31:14.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.057+0000 7f32a8ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a4154e50 0x7f32a40cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.057+0000 7f32a8ffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32a40b5f10 0x7f32a40cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.057+0000 7f32a8ffd640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a406b110 con 0x7f32a40caf40 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.057+0000 7f32a8ffd640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a415b8d0 con 0x7f32a40b5f10 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a8ffd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a415ad50 con 0x7f32a4154e50 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a1d9b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32a40b5f10 0x7f32a40cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a1d9b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f32a40b5f10 0x7f32a40cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42586/0 (socket says 172.21.15.172:42586) 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a4154e50 0x7f32a40cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a1d9b640 1 -- 172.21.15.172:0/1016480781 learned_addr learned my addr 172.21.15.172:0/1016480781 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a159a640 1 -- 172.21.15.172:0/1016480781 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187244497 0 0) 0x7f32a406b110 con 0x7f32a40caf40 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a159a640 1 -- 172.21.15.172:0/1016480781 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f328c003520 con 0x7f32a40caf40 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a1d9b640 1 -- 172.21.15.172:0/1016480781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a4154e50 msgr2=0x7f32a40cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a1d9b640 1 --2- 172.21.15.172:0/1016480781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a4154e50 0x7f32a40cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a1d9b640 1 -- 172.21.15.172:0/1016480781 >> v1:172.21.15.174:6789/0 conn(0x7f32a40caf40 legacy=0x7f32a406b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.058+0000 7f32a1d9b640 1 -- 172.21.15.172:0/1016480781 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a415b030 con 0x7f32a40b5f10 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.059+0000 7f32a1d9b640 1 --2- 172.21.15.172:0/1016480781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a40b5f10 0x7f32a40cc170 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3298009870 tx=0x7f329802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54fdde3e7a4e924 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.059+0000 7f32a159a640 1 -- 172.21.15.172:0/1016480781 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329802f830 con 0x7f32a40b5f10 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.059+0000 7f32a159a640 1 -- 172.21.15.172:0/1016480781 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329802f990 con 0x7f32a40b5f10 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.059+0000 7f32a159a640 1 -- 172.21.15.172:0/1016480781 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329802fd20 con 0x7f32a40b5f10 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.060+0000 7f32a8ffd640 1 -- 172.21.15.172:0/1016480781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a40b5f10 msgr2=0x7f32a40cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.060+0000 7f32a8ffd640 1 --2- 172.21.15.172:0/1016480781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a40b5f10 0x7f32a40cc170 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3298009870 tx=0x7f329802ba10 comp rx=0 tx=0).stop 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.060+0000 7f32a8ffd640 1 -- 172.21.15.172:0/1016480781 shutdown_connections 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.060+0000 7f32a8ffd640 1 --2- 172.21.15.172:0/1016480781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a40b5f10 0x7f32a40cc170 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.060+0000 7f32a8ffd640 1 --2- 172.21.15.172:0/1016480781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a4154e50 0x7f32a40cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.060+0000 7f32a8ffd640 1 -- 172.21.15.172:0/1016480781 >> 172.21.15.172:0/1016480781 conn(0x7f32a4066c20 msgr2=0x7f32a4067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.060+0000 7f32a8ffd640 1 -- 172.21.15.172:0/1016480781 shutdown_connections 2024-03-16T19:31:14.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.060+0000 7f32a8ffd640 1 -- 172.21.15.172:0/1016480781 wait complete. 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.061+0000 7f32a8ffd640 1 Processor -- start 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.061+0000 7f32a8ffd640 1 -- start start 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a8ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a40b5f10 0x7f32a40c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a8ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a40caf40 0x7f32a41e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a8ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32a4154e50 0x7f32a414d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a8ffd640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a414ee30 con 0x7f32a40caf40 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a8ffd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a414cd20 con 0x7f32a4154e50 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a8ffd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a4155ad0 con 0x7f32a40b5f10 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a1d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a40caf40 0x7f32a41e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a40b5f10 0x7f32a40c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a40b5f10 0x7f32a40c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59052/0 (socket says 172.21.15.172:59052) 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a259c640 1 -- 172.21.15.172:0/594764261 learned_addr learned my addr 172.21.15.172:0/594764261 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a2d9d640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32a4154e50 0x7f32a414d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a1d9b640 1 -- 172.21.15.172:0/594764261 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a40b5f10 msgr2=0x7f32a40c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a1d9b640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a40b5f10 0x7f32a40c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a1d9b640 1 -- 172.21.15.172:0/594764261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32a4154e50 msgr2=0x7f32a414d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a1d9b640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32a4154e50 0x7f32a414d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.062+0000 7f32a1d9b640 1 -- 172.21.15.172:0/594764261 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32980094d0 con 0x7f32a40caf40 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.063+0000 7f32a259c640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32a40b5f10 0x7f32a40c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.063+0000 7f32a1d9b640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a40caf40 0x7f32a41e6f00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3298033000 tx=0x7f3298004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.063+0000 7f3292ffd640 1 -- 172.21.15.172:0/594764261 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298037c40 con 0x7f32a40caf40 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.063+0000 7f3292ffd640 1 -- 172.21.15.172:0/594764261 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3298037da0 con 0x7f32a40caf40 2024-03-16T19:31:14.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.063+0000 7f32a8ffd640 1 -- 172.21.15.172:0/594764261 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a415ab20 con 0x7f32a40caf40 2024-03-16T19:31:14.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.063+0000 7f3292ffd640 1 -- 172.21.15.172:0/594764261 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298041640 con 0x7f32a40caf40 2024-03-16T19:31:14.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.063+0000 7f32a8ffd640 1 -- 172.21.15.172:0/594764261 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a4153fb0 con 0x7f32a40caf40 2024-03-16T19:31:14.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.064+0000 7f32a8ffd640 1 -- 172.21.15.172:0/594764261 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32a415a9a0 con 0x7f32a40caf40 2024-03-16T19:31:14.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.065+0000 7f3292ffd640 1 -- 172.21.15.172:0/594764261 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f32980417a0 con 0x7f32a40caf40 2024-03-16T19:31:14.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.065+0000 7f3292ffd640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f328406b3e0 0x7f328406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.065+0000 7f3292ffd640 1 -- 172.21.15.172:0/594764261 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f329803c070 con 0x7f32a40caf40 2024-03-16T19:31:14.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.065+0000 7f32a259c640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f328406b3e0 0x7f328406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.065+0000 7f3292ffd640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f32840737e0 0x7f3284075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.065+0000 7f3292ffd640 1 -- 172.21.15.172:0/594764261 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3284076330 con 0x7f32840737e0 2024-03-16T19:31:14.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.065+0000 7f3292ffd640 1 -- 172.21.15.172:0/594764261 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32980aedd0 con 0x7f32a40caf40 2024-03-16T19:31:14.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.065+0000 7f32a2d9d640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f32840737e0 0x7f3284075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.066+0000 7f32a2d9d640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f32840737e0 0x7f3284075c60 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.066+0000 7f32a259c640 1 --2- 172.21.15.172:0/594764261 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f328406b3e0 0x7f328406d8a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3294009f50 tx=0x7f3294009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.066+0000 7f3292ffd640 1 -- 172.21.15.172:0/594764261 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3284076330 con 0x7f32840737e0 2024-03-16T19:31:14.068 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:14.087+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:14.073 INFO:teuthology.orchestra.run.smithi172.stderr:no valid command found; 10 closest matches: 2024-03-16T19:31:14.073 INFO:teuthology.orchestra.run.smithi172.stderr:0 2024-03-16T19:31:14.073 INFO:teuthology.orchestra.run.smithi172.stderr:1 2024-03-16T19:31:14.073 INFO:teuthology.orchestra.run.smithi172.stderr:2 2024-03-16T19:31:14.073 INFO:teuthology.orchestra.run.smithi172.stderr:abort 2024-03-16T19:31:14.074 INFO:teuthology.orchestra.run.smithi172.stderr:assert 2024-03-16T19:31:14.074 INFO:teuthology.orchestra.run.smithi172.stderr:bench [] [] [] [] 2024-03-16T19:31:14.074 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs debug_inject_read_zeros 2024-03-16T19:31:14.074 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs files list 2024-03-16T19:31:14.074 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs stats 2024-03-16T19:31:14.074 INFO:teuthology.orchestra.run.smithi172.stderr:bluestore allocator dump block 2024-03-16T19:31:14.074 INFO:teuthology.orchestra.run.smithi172.stderr:Error EINVAL: invalid command 2024-03-16T19:31:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-16T19:31:14.078 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-16T19:31:14.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-16T19:31:14.103 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:14.103 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:14.112 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:14.169 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:14.188+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:14.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.200+0000 7f17d9bf4640 1 Processor -- start 2024-03-16T19:31:14.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.200+0000 7f17d9bf4640 1 -- start start 2024-03-16T19:31:14.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d9bf4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17d40caf40 0x7f17d4166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d9bf4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d406b110 con 0x7f17d4154e50 2024-03-16T19:31:14.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d9bf4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d415b8d0 con 0x7f17d40b5f10 2024-03-16T19:31:14.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d9bf4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d415ad50 con 0x7f17d40caf40 2024-03-16T19:31:14.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d37fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f17d4154e50 0x7f17d40cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:47704/0 (socket says 172.21.15.172:47704) 2024-03-16T19:31:14.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d37fe640 1 -- 172.21.15.172:0/638761975 learned_addr learned my addr 172.21.15.172:0/638761975 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:14.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d3fff640 1 --2- 172.21.15.172:0/638761975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17d40caf40 0x7f17d4166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d3fff640 1 -- 172.21.15.172:0/638761975 >> v1:172.21.15.174:6789/0 conn(0x7f17d40b5f10 legacy=0x7f17d406b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d3fff640 1 -- 172.21.15.172:0/638761975 >> v1:172.21.15.172:6789/0 conn(0x7f17d4154e50 legacy=0x7f17d40cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.201+0000 7f17d3fff640 1 -- 172.21.15.172:0/638761975 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d415b030 con 0x7f17d40caf40 2024-03-16T19:31:14.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.203+0000 7f17d3fff640 1 --2- 172.21.15.172:0/638761975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17d40caf40 0x7f17d4166270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009dc0 tx=0x7f17cc02bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec2ad0652d07b094 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.203+0000 7f17d27fc640 1 -- 172.21.15.172:0/638761975 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc02cd50 con 0x7f17d40caf40 2024-03-16T19:31:14.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.203+0000 7f17d27fc640 1 -- 172.21.15.172:0/638761975 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc034440 con 0x7f17d40caf40 2024-03-16T19:31:14.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.203+0000 7f17d27fc640 1 -- 172.21.15.172:0/638761975 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0347d0 con 0x7f17d40caf40 2024-03-16T19:31:14.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.204+0000 7f17d9bf4640 1 -- 172.21.15.172:0/638761975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17d40caf40 msgr2=0x7f17d4166270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.204+0000 7f17d9bf4640 1 --2- 172.21.15.172:0/638761975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17d40caf40 0x7f17d4166270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009dc0 tx=0x7f17cc02bfa0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.204+0000 7f17d9bf4640 1 -- 172.21.15.172:0/638761975 shutdown_connections 2024-03-16T19:31:14.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.204+0000 7f17d9bf4640 1 --2- 172.21.15.172:0/638761975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17d40caf40 0x7f17d4166270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.204+0000 7f17d9bf4640 1 -- 172.21.15.172:0/638761975 >> 172.21.15.172:0/638761975 conn(0x7f17d4066c20 msgr2=0x7f17d4067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:14.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.204+0000 7f17d9bf4640 1 -- 172.21.15.172:0/638761975 shutdown_connections 2024-03-16T19:31:14.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.204+0000 7f17d9bf4640 1 -- 172.21.15.172:0/638761975 wait complete. 2024-03-16T19:31:14.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.205+0000 7f17d9bf4640 1 Processor -- start 2024-03-16T19:31:14.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.206+0000 7f17d9bf4640 1 -- start start 2024-03-16T19:31:14.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d9bf4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d40b5f10 0x7f17d4157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d9bf4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17d40caf40 0x7f17d41558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d9bf4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17d4154e50 0x7f17d40c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d40b5f10 0x7f17d4157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d9bf4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d40c4ff0 con 0x7f17d40b5f10 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d9bf4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d41583e0 con 0x7f17d40caf40 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d9bf4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d41563b0 con 0x7f17d4154e50 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d40b5f10 0x7f17d4157ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42598/0 (socket says 172.21.15.172:42598) 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d37fe640 1 -- 172.21.15.172:0/842197626 learned_addr learned my addr 172.21.15.172:0/842197626 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d2ffd640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17d40caf40 0x7f17d41558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d3fff640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17d4154e50 0x7f17d40c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d3fff640 1 -- 172.21.15.172:0/842197626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17d40caf40 msgr2=0x7f17d41558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d3fff640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17d40caf40 0x7f17d41558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d3fff640 1 -- 172.21.15.172:0/842197626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d40b5f10 msgr2=0x7f17d4157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d3fff640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d40b5f10 0x7f17d4157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d3fff640 1 -- 172.21.15.172:0/842197626 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc009a70 con 0x7f17d4154e50 2024-03-16T19:31:14.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.207+0000 7f17d37fe640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f17d40b5f10 0x7f17d4157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:14.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.208+0000 7f17d3fff640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17d4154e50 0x7f17d40c5480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f17cc033000 tx=0x7f17cc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.208+0000 7f17bbfff640 1 -- 172.21.15.172:0/842197626 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc040450 con 0x7f17d4154e50 2024-03-16T19:31:14.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.208+0000 7f17bbfff640 1 -- 172.21.15.172:0/842197626 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc0405b0 con 0x7f17d4154e50 2024-03-16T19:31:14.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.208+0000 7f17bbfff640 1 -- 172.21.15.172:0/842197626 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc040940 con 0x7f17d4154e50 2024-03-16T19:31:14.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.208+0000 7f17d9bf4640 1 -- 172.21.15.172:0/842197626 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d415ab20 con 0x7f17d4154e50 2024-03-16T19:31:14.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.208+0000 7f17d9bf4640 1 -- 172.21.15.172:0/842197626 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d40c4c80 con 0x7f17d4154e50 2024-03-16T19:31:14.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.209+0000 7f17d9bf4640 1 -- 172.21.15.172:0/842197626 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f17d406d1e0 con 0x7f17d4154e50 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.210+0000 7f17bbfff640 1 -- 172.21.15.172:0/842197626 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f17cc040aa0 con 0x7f17d4154e50 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.210+0000 7f17bbfff640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f17c006b3b0 0x7f17c006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.210+0000 7f17bbfff640 1 -- 172.21.15.172:0/842197626 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f17cc0ac3b0 con 0x7f17d4154e50 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.210+0000 7f17d37fe640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f17c006b3b0 0x7f17c006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.210+0000 7f17bbfff640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f17c00737b0 0x7f17c0075c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.210+0000 7f17bbfff640 1 -- 172.21.15.172:0/842197626 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17c0076300 con 0x7f17c00737b0 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.210+0000 7f17bbfff640 1 -- 172.21.15.172:0/842197626 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17cc0aea70 con 0x7f17d4154e50 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.211+0000 7f17d2ffd640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f17c00737b0 0x7f17c0075c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.211+0000 7f17d37fe640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f17c006b3b0 0x7f17c006d870 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f17c4005880 tx=0x7f17c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.211+0000 7f17d2ffd640 1 --2- 172.21.15.172:0/842197626 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f17c00737b0 0x7f17c0075c30 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.212+0000 7f17bbfff640 1 -- 172.21.15.172:0/842197626 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f17c0076300 con 0x7f17c00737b0 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:no valid command found; 10 closest matches: 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:0 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:1 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:2 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:abort 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:assert 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:bench [] [] [] [] 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs debug_inject_read_zeros 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs files list 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs stats 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:bluestore allocator dump block 2024-03-16T19:31:14.219 INFO:teuthology.orchestra.run.smithi172.stderr:Error EINVAL: invalid command 2024-03-16T19:31:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-16T19:31:14.223 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-16T19:31:14.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-16T19:31:14.260 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:14.260 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:14.269 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:14.270 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:14.289+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:14.367 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.386+0000 7fc121276640 1 Processor -- start 2024-03-16T19:31:14.367 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.386+0000 7fc121276640 1 -- start start 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc121276640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c157ec0 0x7fc11c15b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc121276640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc11c15b540 0x7fc11c0c9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc121276640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c15b990 con 0x7fc11c06dc20 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc121276640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c0cba70 con 0x7fc11c15b540 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc121276640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c06bfe0 con 0x7fc11c157ec0 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc11a7fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc11c06dc20 0x7fc11c0c9080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:47718/0 (socket says 172.21.15.172:47718) 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc11a7fc640 1 -- 172.21.15.172:0/952418563 learned_addr learned my addr 172.21.15.172:0/952418563 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc11affd640 1 --2- 172.21.15.172:0/952418563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c157ec0 0x7fc11c15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc11b7fe640 1 --2- 172.21.15.172:0/952418563 >> v2:172.21.15.174:3300/0 conn(0x7fc11c15b540 0x7fc11c0c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc119ffb640 1 -- 172.21.15.172:0/952418563 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704702138 0 0) 0x7fc110001110 con 0x7fc11c06dc20 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc119ffb640 1 -- 172.21.15.172:0/952418563 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc104003520 con 0x7fc11c06dc20 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc11b7fe640 1 -- 172.21.15.172:0/952418563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c157ec0 msgr2=0x7fc11c15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:14.369 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc11b7fe640 1 --2- 172.21.15.172:0/952418563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c157ec0 0x7fc11c15b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc11b7fe640 1 -- 172.21.15.172:0/952418563 >> v1:172.21.15.172:6789/0 conn(0x7fc11c06dc20 legacy=0x7fc11c0c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc11b7fe640 1 -- 172.21.15.172:0/952418563 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c0700f0 con 0x7fc11c15b540 2024-03-16T19:31:14.370 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.387+0000 7fc11affd640 1 --2- 172.21.15.172:0/952418563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c157ec0 0x7fc11c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:14.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.388+0000 7fc11b7fe640 1 --2- 172.21.15.172:0/952418563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc11c15b540 0x7fc11c0c9790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc114009d90 tx=0x7fc11402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=936babfe362555d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.388+0000 7fc119ffb640 1 -- 172.21.15.172:0/952418563 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11402cce0 con 0x7fc11c15b540 2024-03-16T19:31:14.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.388+0000 7fc119ffb640 1 -- 172.21.15.172:0/952418563 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11402ce40 con 0x7fc11c15b540 2024-03-16T19:31:14.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.388+0000 7fc119ffb640 1 -- 172.21.15.172:0/952418563 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114034640 con 0x7fc11c15b540 2024-03-16T19:31:14.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.389+0000 7fc121276640 1 -- 172.21.15.172:0/952418563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc11c15b540 msgr2=0x7fc11c0c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.389+0000 7fc121276640 1 --2- 172.21.15.172:0/952418563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc11c15b540 0x7fc11c0c9790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc114009d90 tx=0x7fc11402bf30 comp rx=0 tx=0).stop 2024-03-16T19:31:14.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.389+0000 7fc121276640 1 -- 172.21.15.172:0/952418563 shutdown_connections 2024-03-16T19:31:14.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.389+0000 7fc121276640 1 --2- 172.21.15.172:0/952418563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc11c15b540 0x7fc11c0c9790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.389+0000 7fc121276640 1 --2- 172.21.15.172:0/952418563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c157ec0 0x7fc11c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.389+0000 7fc121276640 1 -- 172.21.15.172:0/952418563 >> 172.21.15.172:0/952418563 conn(0x7fc11c070810 msgr2=0x7fc11c070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:14.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.389+0000 7fc121276640 1 -- 172.21.15.172:0/952418563 shutdown_connections 2024-03-16T19:31:14.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.389+0000 7fc121276640 1 -- 172.21.15.172:0/952418563 wait complete. 2024-03-16T19:31:14.372 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:14.390+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:14.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.390+0000 7fc121276640 1 Processor -- start 2024-03-16T19:31:14.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.390+0000 7fc121276640 1 -- start start 2024-03-16T19:31:14.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc121276640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c06dc20 0x7fc11c0c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc121276640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc11c157ec0 0x7fc11c156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc121276640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc11c15b540 0x7fc11c1587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc121276640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c1f12e0 con 0x7fc11c15b540 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc121276640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c15a8e0 con 0x7fc11c157ec0 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc121276640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0c7ef0 con 0x7fc11c06dc20 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc11b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc11c15b540 0x7fc11c1587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc11b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc11c15b540 0x7fc11c1587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42604/0 (socket says 172.21.15.172:42604) 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc11b7fe640 1 -- 172.21.15.172:0/1297173700 learned_addr learned my addr 172.21.15.172:0/1297173700 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc11a7fc640 1 --2- 172.21.15.172:0/1297173700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc11c157ec0 0x7fc11c156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.391+0000 7fc11affd640 1 --2- 172.21.15.172:0/1297173700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c06dc20 0x7fc11c0c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc11b7fe640 1 -- 172.21.15.172:0/1297173700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c06dc20 msgr2=0x7fc11c0c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc11b7fe640 1 --2- 172.21.15.172:0/1297173700 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc11c06dc20 0x7fc11c0c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc11b7fe640 1 -- 172.21.15.172:0/1297173700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc11c157ec0 msgr2=0x7fc11c156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc11b7fe640 1 --2- 172.21.15.172:0/1297173700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc11c157ec0 0x7fc11c156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc11b7fe640 1 -- 172.21.15.172:0/1297173700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c009400 con 0x7fc11c15b540 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc11b7fe640 1 --2- 172.21.15.172:0/1297173700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc11c15b540 0x7fc11c1587b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc114009eb0 tx=0x7fc114004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc1037fe640 1 -- 172.21.15.172:0/1297173700 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11403d600 con 0x7fc11c15b540 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc1037fe640 1 -- 172.21.15.172:0/1297173700 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11403d780 con 0x7fc11c15b540 2024-03-16T19:31:14.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc1037fe640 1 -- 172.21.15.172:0/1297173700 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11403db30 con 0x7fc11c15b540 2024-03-16T19:31:14.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc121276640 1 -- 172.21.15.172:0/1297173700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc114009a70 con 0x7fc11c15b540 2024-03-16T19:31:14.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.392+0000 7fc121276640 1 -- 172.21.15.172:0/1297173700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c1f1000 con 0x7fc11c15b540 2024-03-16T19:31:14.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.393+0000 7fc121276640 1 -- 172.21.15.172:0/1297173700 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc11c06c280 con 0x7fc11c15b540 2024-03-16T19:31:14.376 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:14.376 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:14.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.395+0000 7fc1037fe640 1 -- 172.21.15.172:0/1297173700 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc11403b040 con 0x7fc11c15b540 2024-03-16T19:31:14.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.395+0000 7fc1037fe640 1 --2- 172.21.15.172:0/1297173700 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc10806b3b0 0x7fc10806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.395+0000 7fc1037fe640 1 -- 172.21.15.172:0/1297173700 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6002+0+0 (secure 0 0 0) 0x7fc1140ac720 con 0x7fc11c15b540 2024-03-16T19:31:14.377 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.395+0000 7fc11affd640 1 --2- 172.21.15.172:0/1297173700 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc10806b3b0 0x7fc10806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.377 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:14.380 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:14.380 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-16T19:31:14.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-03-16T19:31:14.385 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:14.471 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:14.490+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:14.491 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:14.491 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:14.500 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:14.524 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.543+0000 7f28b3c9d640 1 Processor -- start 2024-03-16T19:31:14.524 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.543+0000 7f28b3c9d640 1 -- start start 2024-03-16T19:31:14.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.544+0000 7f28b3c9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f28ac06ba30 0x7f28ac166280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.544+0000 7f28b3c9d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ac0719e0 con 0x7f28ac158230 2024-03-16T19:31:14.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.544+0000 7f28b3c9d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ac0c7b50 con 0x7f28ac157e50 2024-03-16T19:31:14.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.544+0000 7f28b3c9d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28ac06aed0 con 0x7f28ac06ba30 2024-03-16T19:31:14.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b1238640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f28ac158230 0x7f28ac06b320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:47722/0 (socket says 172.21.15.172:47722) 2024-03-16T19:31:14.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b1238640 1 -- 172.21.15.172:0/1015368706 learned_addr learned my addr 172.21.15.172:0/1015368706 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:14.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b223a640 1 --2- 172.21.15.172:0/1015368706 >> v2:172.21.15.174:3300/0 conn(0x7f28ac06ba30 0x7f28ac166280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b0a37640 1 -- 172.21.15.172:0/1015368706 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184172309 0 0) 0x7f28ac0719e0 con 0x7f28ac158230 2024-03-16T19:31:14.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b0a37640 1 -- 172.21.15.172:0/1015368706 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2898003520 con 0x7f28ac158230 2024-03-16T19:31:14.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b0a37640 1 -- 172.21.15.172:0/1015368706 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122147515 0 0) 0x7f28ac0c7b50 con 0x7f28ac157e50 2024-03-16T19:31:14.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b0a37640 1 -- 172.21.15.172:0/1015368706 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ac0719e0 con 0x7f28ac157e50 2024-03-16T19:31:14.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b223a640 1 -- 172.21.15.172:0/1015368706 >> v1:172.21.15.174:6789/0 conn(0x7f28ac157e50 legacy=0x7f28ac0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b223a640 1 -- 172.21.15.172:0/1015368706 >> v1:172.21.15.172:6789/0 conn(0x7f28ac158230 legacy=0x7f28ac06b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.545+0000 7f28b223a640 1 -- 172.21.15.172:0/1015368706 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac0c7e10 con 0x7f28ac06ba30 2024-03-16T19:31:14.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.546+0000 7f28b223a640 1 --2- 172.21.15.172:0/1015368706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28ac06ba30 0x7f28ac166280 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009d90 tx=0x7f28a402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f4baa8d44261728 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.546+0000 7f28b0a37640 1 -- 172.21.15.172:0/1015368706 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a402cd00 con 0x7f28ac06ba30 2024-03-16T19:31:14.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.546+0000 7f28b0a37640 1 -- 172.21.15.172:0/1015368706 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a402ce60 con 0x7f28ac06ba30 2024-03-16T19:31:14.527 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.546+0000 7f28b0a37640 1 -- 172.21.15.172:0/1015368706 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a4034660 con 0x7f28ac06ba30 2024-03-16T19:31:14.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.547+0000 7f28b3c9d640 1 -- 172.21.15.172:0/1015368706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28ac06ba30 msgr2=0x7f28ac166280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.547+0000 7f28b3c9d640 1 --2- 172.21.15.172:0/1015368706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28ac06ba30 0x7f28ac166280 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009d90 tx=0x7f28a402bf50 comp rx=0 tx=0).stop 2024-03-16T19:31:14.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.547+0000 7f28b3c9d640 1 -- 172.21.15.172:0/1015368706 shutdown_connections 2024-03-16T19:31:14.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.547+0000 7f28b3c9d640 1 --2- 172.21.15.172:0/1015368706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28ac06ba30 0x7f28ac166280 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.547+0000 7f28b3c9d640 1 -- 172.21.15.172:0/1015368706 >> 172.21.15.172:0/1015368706 conn(0x7f28ac06db70 msgr2=0x7f28ac06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:14.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.547+0000 7f28b3c9d640 1 -- 172.21.15.172:0/1015368706 shutdown_connections 2024-03-16T19:31:14.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.547+0000 7f28b3c9d640 1 -- 172.21.15.172:0/1015368706 wait complete. 2024-03-16T19:31:14.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.549+0000 7f28b3c9d640 1 Processor -- start 2024-03-16T19:31:14.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.549+0000 7f28b3c9d640 1 -- start start 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b3c9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28ac06ba30 0x7f28ac14e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b3c9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28ac157e50 0x7f28ac1f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b3c9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28ac158230 0x7f28ac0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b3c9d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac14ea80 con 0x7f28ac157e50 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b3c9d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac14e860 con 0x7f28ac158230 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b3c9d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac151290 con 0x7f28ac06ba30 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1238640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28ac157e50 0x7f28ac1f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1a39640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28ac06ba30 0x7f28ac14e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1238640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28ac157e50 0x7f28ac1f4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42614/0 (socket says 172.21.15.172:42614) 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1238640 1 -- 172.21.15.172:0/1048570421 learned_addr learned my addr 172.21.15.172:0/1048570421 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:14.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b223a640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28ac158230 0x7f28ac0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1a39640 1 -- 172.21.15.172:0/1048570421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28ac158230 msgr2=0x7f28ac0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1a39640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28ac158230 0x7f28ac0cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1a39640 1 -- 172.21.15.172:0/1048570421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28ac157e50 msgr2=0x7f28ac1f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1a39640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28ac157e50 0x7f28ac1f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1a39640 1 -- 172.21.15.172:0/1048570421 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a4009a70 con 0x7f28ac06ba30 2024-03-16T19:31:14.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b223a640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28ac158230 0x7f28ac0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:14.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b1a39640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28ac06ba30 0x7f28ac14e320 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f28a800c3f0 tx=0x7f28a800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28a27fc640 1 -- 172.21.15.172:0/1048570421 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80052d0 con 0x7f28ac06ba30 2024-03-16T19:31:14.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28a27fc640 1 -- 172.21.15.172:0/1048570421 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a8005450 con 0x7f28ac06ba30 2024-03-16T19:31:14.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b3c9d640 1 -- 172.21.15.172:0/1048570421 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ac1f45d0 con 0x7f28ac06ba30 2024-03-16T19:31:14.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28a27fc640 1 -- 172.21.15.172:0/1048570421 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8003830 con 0x7f28ac06ba30 2024-03-16T19:31:14.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.550+0000 7f28b3c9d640 1 -- 172.21.15.172:0/1048570421 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ac14f400 con 0x7f28ac06ba30 2024-03-16T19:31:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.551+0000 7f28b3c9d640 1 -- 172.21.15.172:0/1048570421 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f28ac0c4180 con 0x7f28ac06ba30 2024-03-16T19:31:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.552+0000 7f28a27fc640 1 -- 172.21.15.172:0/1048570421 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f28a8003990 con 0x7f28ac06ba30 2024-03-16T19:31:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.552+0000 7f28a27fc640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f289006b3b0 0x7f289006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.553+0000 7f28a27fc640 1 -- 172.21.15.172:0/1048570421 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f28a8089290 con 0x7f28ac06ba30 2024-03-16T19:31:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.553+0000 7f28b1238640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f289006b3b0 0x7f289006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.553+0000 7f28a27fc640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f2890073750 0x7f2890075bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.553+0000 7f28a27fc640 1 -- 172.21.15.172:0/1048570421 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28900762a0 con 0x7f2890073750 2024-03-16T19:31:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.553+0000 7f28a27fc640 1 -- 172.21.15.172:0/1048570421 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28a808b920 con 0x7f28ac06ba30 2024-03-16T19:31:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.553+0000 7f28b223a640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f2890073750 0x7f2890075bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.553+0000 7f28b1238640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f289006b3b0 0x7f289006d870 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f28ac06cd20 tx=0x7f289c007e10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.553+0000 7f28b223a640 1 --2- 172.21.15.172:0/1048570421 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f2890073750 0x7f2890075bd0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.554+0000 7f28a27fc640 1 -- 172.21.15.172:0/1048570421 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f28900762a0 con 0x7f2890073750 2024-03-16T19:31:14.561 INFO:teuthology.orchestra.run.smithi172.stderr:no valid command found; 10 closest matches: 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:0 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:1 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:2 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:abort 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:assert 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:bench [] [] [] [] 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs debug_inject_read_zeros 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs files list 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs stats 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:bluestore allocator dump block 2024-03-16T19:31:14.562 INFO:teuthology.orchestra.run.smithi172.stderr:Error EINVAL: invalid command 2024-03-16T19:31:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-16T19:31:14.564 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-16T19:31:14.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-03-16T19:31:14.583 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:14.602+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:14.646 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:14.646 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:14.656 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:14.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.687+0000 7f3f438d0640 1 Processor -- start 2024-03-16T19:31:14.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.688+0000 7f3f438d0640 1 -- start start 2024-03-16T19:31:14.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.688+0000 7f3f438d0640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3f3c154e50 0x7f3f3c0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.688+0000 7f3f438d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 0x7f3f3c0cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.688+0000 7f3f438d0640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c06b110 con 0x7f3f3c0caf40 2024-03-16T19:31:14.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.688+0000 7f3f438d0640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c15b8d0 con 0x7f3f3c154e50 2024-03-16T19:31:14.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.688+0000 7f3f438d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c15ad50 con 0x7f3f3c0b5f10 2024-03-16T19:31:14.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.689+0000 7f3f4166c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3f3c154e50 0x7f3f3c0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.689+0000 7f3f41e6d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3f3c0caf40 0x7f3f3c06b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:47724/0 (socket says 172.21.15.172:47724) 2024-03-16T19:31:14.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.689+0000 7f3f41e6d640 1 -- 172.21.15.172:0/1575177523 learned_addr learned my addr 172.21.15.172:0/1575177523 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:14.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.689+0000 7f3f40e6b640 1 --2- 172.21.15.172:0/1575177523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 0x7f3f3c0cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.689+0000 7f3f4166c640 1 -- 172.21.15.172:0/1575177523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 msgr2=0x7f3f3c0cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:14.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.689+0000 7f3f4166c640 1 --2- 172.21.15.172:0/1575177523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 0x7f3f3c0cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.689+0000 7f3f4166c640 1 -- 172.21.15.172:0/1575177523 >> v1:172.21.15.172:6789/0 conn(0x7f3f3c0caf40 legacy=0x7f3f3c06b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.689+0000 7f3f33fff640 1 -- 172.21.15.172:0/1575177523 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695871023 0 0) 0x7f3f3c06b110 con 0x7f3f3c0caf40 2024-03-16T19:31:14.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.689+0000 7f3f4166c640 1 -- 172.21.15.172:0/1575177523 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c15b030 con 0x7f3f3c154e50 2024-03-16T19:31:14.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.690+0000 7f3f4166c640 1 --2- 172.21.15.172:0/1575177523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f3c154e50 0x7f3f3c0cbc30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3f380097a0 tx=0x7f3f3802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b7e6f7e25c10ce server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.690+0000 7f3f33fff640 1 -- 172.21.15.172:0/1575177523 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38030610 con 0x7f3f3c154e50 2024-03-16T19:31:14.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.690+0000 7f3f33fff640 1 -- 172.21.15.172:0/1575177523 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f38030770 con 0x7f3f3c154e50 2024-03-16T19:31:14.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.690+0000 7f3f33fff640 1 -- 172.21.15.172:0/1575177523 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38030b00 con 0x7f3f3c154e50 2024-03-16T19:31:14.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.691+0000 7f3f438d0640 1 -- 172.21.15.172:0/1575177523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f3c154e50 msgr2=0x7f3f3c0cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.691+0000 7f3f438d0640 1 --2- 172.21.15.172:0/1575177523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f3c154e50 0x7f3f3c0cbc30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3f380097a0 tx=0x7f3f3802b960 comp rx=0 tx=0).stop 2024-03-16T19:31:14.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.691+0000 7f3f438d0640 1 -- 172.21.15.172:0/1575177523 shutdown_connections 2024-03-16T19:31:14.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.691+0000 7f3f438d0640 1 --2- 172.21.15.172:0/1575177523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 0x7f3f3c0cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.691+0000 7f3f438d0640 1 --2- 172.21.15.172:0/1575177523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f3c154e50 0x7f3f3c0cbc30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.691+0000 7f3f438d0640 1 -- 172.21.15.172:0/1575177523 >> 172.21.15.172:0/1575177523 conn(0x7f3f3c066c20 msgr2=0x7f3f3c067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:14.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.691+0000 7f3f438d0640 1 -- 172.21.15.172:0/1575177523 shutdown_connections 2024-03-16T19:31:14.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.691+0000 7f3f438d0640 1 -- 172.21.15.172:0/1575177523 wait complete. 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.692+0000 7f3f438d0640 1 Processor -- start 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.692+0000 7f3f438d0640 1 -- start start 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f438d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 0x7f3f3c1577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f438d0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f3c0caf40 0x7f3f3c155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f438d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f3c154e50 0x7f3f3c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f40e6b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f3c0caf40 0x7f3f3c155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f40e6b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f3c0caf40 0x7f3f3c155f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42624/0 (socket says 172.21.15.172:42624) 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f4166c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 0x7f3f3c1577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f4166c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 0x7f3f3c1577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59092/0 (socket says 172.21.15.172:59092) 2024-03-16T19:31:14.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f40e6b640 1 -- 172.21.15.172:0/959301415 learned_addr learned my addr 172.21.15.172:0/959301415 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f438d0640 1 -- 172.21.15.172:0/959301415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c156b10 con 0x7f3f3c0caf40 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f438d0640 1 -- 172.21.15.172:0/959301415 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c158800 con 0x7f3f3c154e50 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f438d0640 1 -- 172.21.15.172:0/959301415 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c1553b0 con 0x7f3f3c0b5f10 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f41e6d640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f3c154e50 0x7f3f3c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f40e6b640 1 -- 172.21.15.172:0/959301415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 msgr2=0x7f3f3c1577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f40e6b640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 0x7f3f3c1577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f40e6b640 1 -- 172.21.15.172:0/959301415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f3c154e50 msgr2=0x7f3f3c1564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f40e6b640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f3c154e50 0x7f3f3c1564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f40e6b640 1 -- 172.21.15.172:0/959301415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f38009400 con 0x7f3f3c0caf40 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f4166c640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f3c0b5f10 0x7f3f3c1577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.693+0000 7f3f40e6b640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f3c0caf40 0x7f3f3c155f70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00e9e0 tx=0x7f3f2c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.694+0000 7f3f31ffb640 1 -- 172.21.15.172:0/959301415 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c009670 con 0x7f3f3c0caf40 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.694+0000 7f3f31ffb640 1 -- 172.21.15.172:0/959301415 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c00c9d0 con 0x7f3f3c0caf40 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.694+0000 7f3f438d0640 1 -- 172.21.15.172:0/959301415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c15ab20 con 0x7f3f3c0caf40 2024-03-16T19:31:14.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.694+0000 7f3f31ffb640 1 -- 172.21.15.172:0/959301415 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c00cd80 con 0x7f3f3c0caf40 2024-03-16T19:31:14.677 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.694+0000 7f3f438d0640 1 -- 172.21.15.172:0/959301415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c155670 con 0x7f3f3c0caf40 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.695+0000 7f3f438d0640 1 -- 172.21.15.172:0/959301415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f3c0ca310 con 0x7f3f3c0caf40 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.695+0000 7f3f31ffb640 1 -- 172.21.15.172:0/959301415 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3f2c013470 con 0x7f3f3c0caf40 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.696+0000 7f3f31ffb640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3f2006b3b0 0x7f3f2006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.696+0000 7f3f31ffb640 1 -- 172.21.15.172:0/959301415 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f3f2c089eb0 con 0x7f3f3c0caf40 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.696+0000 7f3f4166c640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3f2006b3b0 0x7f3f2006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.696+0000 7f3f31ffb640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3f20073750 0x7f3f20075bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.696+0000 7f3f31ffb640 1 -- 172.21.15.172:0/959301415 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f200762a0 con 0x7f3f20073750 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.696+0000 7f3f31ffb640 1 -- 172.21.15.172:0/959301415 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f2c08c570 con 0x7f3f3c0caf40 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.696+0000 7f3f41e6d640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3f20073750 0x7f3f20075bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.696+0000 7f3f4166c640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3f2006b3b0 0x7f3f2006d870 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3f38036000 tx=0x7f3f38008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.697+0000 7f3f41e6d640 1 --2- 172.21.15.172:0/959301415 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3f20073750 0x7f3f20075bd0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.697+0000 7f3f31ffb640 1 -- 172.21.15.172:0/959301415 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3f200762a0 con 0x7f3f20073750 2024-03-16T19:31:14.697 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:14.716+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:no valid command found; 10 closest matches: 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:0 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:1 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:2 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:abort 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:assert 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:bench [] [] [] [] 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs debug_inject_read_zeros 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs files list 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs stats 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:bluestore allocator dump block 2024-03-16T19:31:14.704 INFO:teuthology.orchestra.run.smithi172.stderr:Error EINVAL: invalid command 2024-03-16T19:31:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-16T19:31:14.707 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-16T19:31:14.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-03-16T19:31:14.775 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:14.794+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:14.804 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:14.804 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:14.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:14.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.869+0000 7f4685b25640 1 Processor -- start 2024-03-16T19:31:14.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.869+0000 7f4685b25640 1 -- start start 2024-03-16T19:31:14.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.871+0000 7f4685b25640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4680151910 0x7f4680072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.871+0000 7f467f7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4680151910 0x7f4680072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.871+0000 7f467f7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4680151910 0x7f4680072fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42632/0 (socket says 172.21.15.172:42632) 2024-03-16T19:31:14.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.871+0000 7f467f7fe640 1 -- 172.21.15.172:0/3993455009 learned_addr learned my addr 172.21.15.172:0/3993455009 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:14.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.871+0000 7f4685b25640 1 --2- 172.21.15.172:0/3993455009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3930 0x7f46800c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.871+0000 7f4685b25640 1 -- 172.21.15.172:0/3993455009 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4680072b10 con 0x7f46800c3ea0 2024-03-16T19:31:14.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.871+0000 7f4685b25640 1 -- 172.21.15.172:0/3993455009 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46800c2330 con 0x7f4680151910 2024-03-16T19:31:14.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.871+0000 7f4685b25640 1 -- 172.21.15.172:0/3993455009 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46800c7da0 con 0x7f46800c3930 2024-03-16T19:31:14.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.871+0000 7f467ffff640 1 --2- 172.21.15.172:0/3993455009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3930 0x7f46800c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.872+0000 7f467ffff640 1 -- 172.21.15.172:0/3993455009 >> v2:172.21.15.172:3300/0 conn(0x7f4680151910 msgr2=0x7f4680072fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:14.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.872+0000 7f467e7fc640 1 -- 172.21.15.172:0/3993455009 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384358437 0 0) 0x7f4680072b10 con 0x7f46800c3ea0 2024-03-16T19:31:14.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.872+0000 7f467ffff640 1 --2- 172.21.15.172:0/3993455009 >> v2:172.21.15.172:3300/0 conn(0x7f4680151910 0x7f4680072fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.872+0000 7f467ffff640 1 -- 172.21.15.172:0/3993455009 >> v1:172.21.15.174:6789/0 conn(0x7f46800c3ea0 legacy=0x7f46800ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.872+0000 7f467ffff640 1 -- 172.21.15.172:0/3993455009 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4680071b10 con 0x7f46800c3930 2024-03-16T19:31:14.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.872+0000 7f467f7fe640 1 --2- 172.21.15.172:0/3993455009 >> v2:172.21.15.172:3300/0 conn(0x7f4680151910 0x7f4680072fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:14.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.873+0000 7f467ffff640 1 --2- 172.21.15.172:0/3993455009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3930 0x7f46800c8340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4678009e10 tx=0x7f467802e330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61ba80bd2fcb9a1f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.873+0000 7f467e7fc640 1 -- 172.21.15.172:0/3993455009 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678036640 con 0x7f46800c3930 2024-03-16T19:31:14.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.873+0000 7f467e7fc640 1 -- 172.21.15.172:0/3993455009 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46780367a0 con 0x7f46800c3930 2024-03-16T19:31:14.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.873+0000 7f467e7fc640 1 -- 172.21.15.172:0/3993455009 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678036b30 con 0x7f46800c3930 2024-03-16T19:31:14.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.874+0000 7f4685b25640 1 -- 172.21.15.172:0/3993455009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3930 msgr2=0x7f46800c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.874+0000 7f4685b25640 1 --2- 172.21.15.172:0/3993455009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3930 0x7f46800c8340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4678009e10 tx=0x7f467802e330 comp rx=0 tx=0).stop 2024-03-16T19:31:14.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.874+0000 7f4685b25640 1 -- 172.21.15.172:0/3993455009 shutdown_connections 2024-03-16T19:31:14.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.874+0000 7f4685b25640 1 --2- 172.21.15.172:0/3993455009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3930 0x7f46800c8340 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.874+0000 7f4685b25640 1 --2- 172.21.15.172:0/3993455009 >> v2:172.21.15.172:3300/0 conn(0x7f4680151910 0x7f4680072fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.874+0000 7f4685b25640 1 -- 172.21.15.172:0/3993455009 >> 172.21.15.172:0/3993455009 conn(0x7f468006db70 msgr2=0x7f468006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:14.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.874+0000 7f4685b25640 1 -- 172.21.15.172:0/3993455009 shutdown_connections 2024-03-16T19:31:14.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.874+0000 7f4685b25640 1 -- 172.21.15.172:0/3993455009 wait complete. 2024-03-16T19:31:14.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.876+0000 7f4685b25640 1 Processor -- start 2024-03-16T19:31:14.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.876+0000 7f4685b25640 1 -- start start 2024-03-16T19:31:14.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.877+0000 7f4685b25640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46800c3930 0x7f46801efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.877+0000 7f4685b25640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3ea0 0x7f46801ec780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f4685b25640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4680151910 0x7f468014dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46800c3930 0x7f46801efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46800c3930 0x7f46801efd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40944/0 (socket says 172.21.15.172:40944) 2024-03-16T19:31:14.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467effd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3ea0 0x7f46801ec780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467effd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3ea0 0x7f46801ec780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:59114/0 (socket says 172.21.15.172:59114) 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f4685b25640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680156f30 con 0x7f4680151910 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4680151910 0x7f468014dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4680151910 0x7f468014dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42646/0 (socket says 172.21.15.172:42646) 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467ffff640 1 -- 172.21.15.172:0/2741086053 learned_addr learned my addr 172.21.15.172:0/2741086053 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f4685b25640 1 -- 172.21.15.172:0/2741086053 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680155040 con 0x7f46800c3930 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f4685b25640 1 -- 172.21.15.172:0/2741086053 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46801ec2f0 con 0x7f46800c3ea0 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467effd640 1 -- 172.21.15.172:0/2741086053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46800c3930 msgr2=0x7f46801efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467effd640 1 --2- 172.21.15.172:0/2741086053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46800c3930 0x7f46801efd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467effd640 1 -- 172.21.15.172:0/2741086053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4680151910 msgr2=0x7f468014dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467effd640 1 --2- 172.21.15.172:0/2741086053 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4680151910 0x7f468014dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.878+0000 7f467effd640 1 -- 172.21.15.172:0/2741086053 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46700094d0 con 0x7f46800c3ea0 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.879+0000 7f467effd640 1 --2- 172.21.15.172:0/2741086053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46800c3ea0 0x7f46801ec780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f46740017a0 tx=0x7f4674001c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.879+0000 7f4667fff640 1 -- 172.21.15.172:0/2741086053 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674013070 con 0x7f46800c3ea0 2024-03-16T19:31:14.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.879+0000 7f4667fff640 1 -- 172.21.15.172:0/2741086053 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467400ee60 con 0x7f46800c3ea0 2024-03-16T19:31:14.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.879+0000 7f4685b25640 1 -- 172.21.15.172:0/2741086053 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4678009a70 con 0x7f46800c3ea0 2024-03-16T19:31:14.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.879+0000 7f4667fff640 1 -- 172.21.15.172:0/2741086053 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46740050d0 con 0x7f46800c3ea0 2024-03-16T19:31:14.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.879+0000 7f4685b25640 1 -- 172.21.15.172:0/2741086053 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4680154d60 con 0x7f46800c3ea0 2024-03-16T19:31:14.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.880+0000 7f4665ffb640 1 -- 172.21.15.172:0/2741086053 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f468006cc00 con 0x7f46800c3ea0 2024-03-16T19:31:14.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.881+0000 7f4667fff640 1 -- 172.21.15.172:0/2741086053 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4674005230 con 0x7f46800c3ea0 2024-03-16T19:31:14.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.882+0000 7f4667fff640 1 --2- 172.21.15.172:0/2741086053 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f466c06b3b0 0x7f466c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.882+0000 7f4667fff640 1 -- 172.21.15.172:0/2741086053 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f467408d800 con 0x7f46800c3ea0 2024-03-16T19:31:14.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.882+0000 7f467f7fe640 1 --2- 172.21.15.172:0/2741086053 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f466c06b3b0 0x7f466c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.882+0000 7f4667fff640 1 --2- 172.21.15.172:0/2741086053 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f466c0737b0 0x7f466c075c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:14.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.882+0000 7f4667fff640 1 -- 172.21.15.172:0/2741086053 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f466c076300 con 0x7f466c0737b0 2024-03-16T19:31:14.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.882+0000 7f4667fff640 1 -- 172.21.15.172:0/2741086053 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f467408db70 con 0x7f46800c3ea0 2024-03-16T19:31:14.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.883+0000 7f467ffff640 1 --2- 172.21.15.172:0/2741086053 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f466c0737b0 0x7f466c075c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:14.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.883+0000 7f467f7fe640 1 --2- 172.21.15.172:0/2741086053 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f466c06b3b0 0x7f466c06d870 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4670006ca0 tx=0x7f4670009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.883+0000 7f467ffff640 1 --2- 172.21.15.172:0/2741086053 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f466c0737b0 0x7f466c075c30 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:14.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:14.884+0000 7f4667fff640 1 -- 172.21.15.172:0/2741086053 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f466c076300 con 0x7f466c0737b0 2024-03-16T19:31:14.869 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:31:14.914 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:14.933+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:14.917 INFO:teuthology.orchestra.run.smithi172.stderr:no valid command found; 10 closest matches: 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:0 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:1 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:2 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:abort 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:assert 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:bench [] [] [] [] 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs debug_inject_read_zeros 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs files list 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs stats 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:bluestore allocator dump block 2024-03-16T19:31:14.918 INFO:teuthology.orchestra.run.smithi172.stderr:Error EINVAL: invalid command 2024-03-16T19:31:14.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-16T19:31:14.922 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-16T19:31:14.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-03-16T19:31:14.963 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:14.963 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:14.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:14.978 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:14.996+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:15.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.035+0000 7fe70234b640 1 Processor -- start 2024-03-16T19:31:15.017 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.035+0000 7fe70234b640 1 -- start start 2024-03-16T19:31:15.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe70234b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc158ef0 0x7fe6fc0c9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe70234b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe6fc06cd40 0x7fe6fc0c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe70234b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6fc163270 con 0x7fe6fc1592d0 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe70234b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6fc162430 con 0x7fe6fc06cd40 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe70234b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc1626f0 con 0x7fe6fc158ef0 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe6fb7fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe6fc1592d0 0x7fe6fc06c630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:42176/0 (socket says 172.21.15.172:42176) 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe6fb7fe640 1 -- 172.21.15.172:0/575447866 learned_addr learned my addr 172.21.15.172:0/575447866 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe7008e8640 1 --2- 172.21.15.172:0/575447866 >> v2:172.21.15.174:3300/0 conn(0x7fe6fc06cd40 0x7fe6fc0c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe6faffd640 1 -- 172.21.15.172:0/575447866 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104151991 0 0) 0x7fe6fc163270 con 0x7fe6fc1592d0 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe6faffd640 1 -- 172.21.15.172:0/575447866 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e4003520 con 0x7fe6fc1592d0 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe7008e8640 1 -- 172.21.15.172:0/575447866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc158ef0 msgr2=0x7fe6fc0c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe6fbfff640 1 --2- 172.21.15.172:0/575447866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc158ef0 0x7fe6fc0c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe7008e8640 1 --2- 172.21.15.172:0/575447866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc158ef0 0x7fe6fc0c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe7008e8640 1 -- 172.21.15.172:0/575447866 >> v1:172.21.15.174:6789/0 conn(0x7fe6fc1592d0 legacy=0x7fe6fc06c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.037+0000 7fe7008e8640 1 -- 172.21.15.172:0/575447866 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc1629d0 con 0x7fe6fc06cd40 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.038+0000 7fe6fbfff640 1 --2- 172.21.15.172:0/575447866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc158ef0 0x7fe6fc0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.038+0000 7fe7008e8640 1 --2- 172.21.15.172:0/575447866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6fc06cd40 0x7fe6fc0c95c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009d90 tx=0x7fe6f402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd794f7f18f6bbb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.039+0000 7fe6faffd640 1 -- 172.21.15.172:0/575447866 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f402cc80 con 0x7fe6fc06cd40 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.039+0000 7fe6faffd640 1 -- 172.21.15.172:0/575447866 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f402cde0 con 0x7fe6fc06cd40 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.039+0000 7fe6faffd640 1 -- 172.21.15.172:0/575447866 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4034640 con 0x7fe6fc06cd40 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.040+0000 7fe70234b640 1 -- 172.21.15.172:0/575447866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6fc06cd40 msgr2=0x7fe6fc0c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.040+0000 7fe70234b640 1 --2- 172.21.15.172:0/575447866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6fc06cd40 0x7fe6fc0c95c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009d90 tx=0x7fe6f402bed0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.040+0000 7fe70234b640 1 -- 172.21.15.172:0/575447866 shutdown_connections 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.040+0000 7fe70234b640 1 --2- 172.21.15.172:0/575447866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6fc06cd40 0x7fe6fc0c95c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.040+0000 7fe70234b640 1 --2- 172.21.15.172:0/575447866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc158ef0 0x7fe6fc0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.040+0000 7fe70234b640 1 -- 172.21.15.172:0/575447866 >> 172.21.15.172:0/575447866 conn(0x7fe6fc066c20 msgr2=0x7fe6fc067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.040+0000 7fe70234b640 1 -- 172.21.15.172:0/575447866 shutdown_connections 2024-03-16T19:31:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.040+0000 7fe70234b640 1 -- 172.21.15.172:0/575447866 wait complete. 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.042+0000 7fe70234b640 1 Processor -- start 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.042+0000 7fe70234b640 1 -- start start 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.043+0000 7fe70234b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc06cd40 0x7fe6fc15bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.043+0000 7fe70234b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6fc158ef0 0x7fe6fc159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.043+0000 7fe70234b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6fc1592d0 0x7fe6fc159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.043+0000 7fe70234b640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc15d080 con 0x7fe6fc1592d0 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.043+0000 7fe70234b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0b2d70 con 0x7fe6fc158ef0 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.043+0000 7fe70234b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc15dc10 con 0x7fe6fc06cd40 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe6fb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6fc158ef0 0x7fe6fc159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe6fb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6fc158ef0 0x7fe6fc159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40974/0 (socket says 172.21.15.172:40974) 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe6fb7fe640 1 -- 172.21.15.172:0/3255318426 learned_addr learned my addr 172.21.15.172:0/3255318426 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:15.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe7008e8640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6fc1592d0 0x7fe6fc159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe6fbfff640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc06cd40 0x7fe6fc15bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe6fb7fe640 1 -- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc06cd40 msgr2=0x7fe6fc15bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe6fb7fe640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6fc06cd40 0x7fe6fc15bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe6fb7fe640 1 -- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6fc1592d0 msgr2=0x7fe6fc159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe6fb7fe640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6fc1592d0 0x7fe6fc159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.044+0000 7fe6fb7fe640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4009a70 con 0x7fe6fc158ef0 2024-03-16T19:31:15.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.045+0000 7fe6fb7fe640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6fc158ef0 0x7fe6fc159a00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000c4d0 tx=0x7fe6f000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.045+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0005da0 con 0x7fe6fc158ef0 2024-03-16T19:31:15.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.045+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f0007070 con 0x7fe6fc158ef0 2024-03-16T19:31:15.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.045+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f00073e0 con 0x7fe6fc158ef0 2024-03-16T19:31:15.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.046+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc0b1e50 con 0x7fe6fc158ef0 2024-03-16T19:31:15.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.046+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc0b23d0 con 0x7fe6fc158ef0 2024-03-16T19:31:15.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.047+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe6fc159720 con 0x7fe6fc158ef0 2024-03-16T19:31:15.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.049+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fe6f0007540 con 0x7fe6fc158ef0 2024-03-16T19:31:15.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.049+0000 7fe6f8ff9640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe6e806b3e0 0x7fe6e806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.049+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6002+0+0 (secure 0 0 0) 0x7fe6f00891e0 con 0x7fe6fc158ef0 2024-03-16T19:31:15.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.050+0000 7fe6fbfff640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe6e806b3e0 0x7fe6e806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.050+0000 7fe6fbfff640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe6e806b3e0 0x7fe6e806d8a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00ab60 tx=0x7fe6ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.050+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v70) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe6fc159720 con 0x7fe6fc158ef0 2024-03-16T19:31:15.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.051+0000 7fe70234b640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fe6fc0b1910 0x7fe6fc0c4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.051+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6fc0cba10 con 0x7fe6fc0b1910 2024-03-16T19:31:15.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.051+0000 7fe7008e8640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fe6fc0b1910 0x7fe6fc0c4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.052+0000 7fe7008e8640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fe6fc0b1910 0x7fe6fc0c4670 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.053+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe6fc0cba10 con 0x7fe6fc0b1910 2024-03-16T19:31:15.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.083+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6fc0b2b00 con 0x7fe6fc0b1910 2024-03-16T19:31:15.067 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.086+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe6fc0b2b00 con 0x7fe6fc0b1910 2024-03-16T19:31:15.068 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:15.068 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:15.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.087+0000 7fe70234b640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fe6fc0c4bb0 0x7fe6fc154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.087+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6fc162f90 con 0x7fe6fc0c4bb0 2024-03-16T19:31:15.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.087+0000 7fe6fbfff640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fe6fc0c4bb0 0x7fe6fc154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.087+0000 7fe6fbfff640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fe6fc0c4bb0 0x7fe6fc154770 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.089+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe6ec00c040 con 0x7fe6fc0c4bb0 2024-03-16T19:31:15.077 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:15.097+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:15.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.118+0000 7f0e75d27640 1 Processor -- start 2024-03-16T19:31:15.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.118+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6fc0c8d40 con 0x7fe6fc0c4bb0 2024-03-16T19:31:15.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.118+0000 7f0e75d27640 1 -- start start 2024-03-16T19:31:15.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.119+0000 7f0e75d27640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 0x7f0e700c9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.119+0000 7f0e75d27640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0e7006dc20 0x7f0e700c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.119+0000 7f0e75d27640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e700cba70 con 0x7f0e70157ec0 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.119+0000 7f0e75d27640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e7006bfe0 con 0x7f0e7006dc20 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.119+0000 7f0e75d27640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700700f0 con 0x7f0e7015b710 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.119+0000 7f0e6effd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 0x7f0e700c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.119+0000 7f0e6f7fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f0e70157ec0 0x7f0e7015b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:42178/0 (socket says 172.21.15.172:42178) 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.120+0000 7f0e6ffff640 1 --2- 172.21.15.172:0/2544229057 >> v2:172.21.15.174:3300/0 conn(0x7f0e7006dc20 0x7f0e700c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.120+0000 7f0e6f7fe640 1 -- 172.21.15.172:0/2544229057 learned_addr learned my addr 172.21.15.172:0/2544229057 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.120+0000 7f0e6e7fc640 1 -- 172.21.15.172:0/2544229057 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639941133 0 0) 0x7f0e700cba70 con 0x7f0e70157ec0 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.120+0000 7f0e6e7fc640 1 -- 172.21.15.172:0/2544229057 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e58003520 con 0x7f0e70157ec0 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.120+0000 7f0e6effd640 1 -- 172.21.15.172:0/2544229057 >> v2:172.21.15.174:3300/0 conn(0x7f0e7006dc20 msgr2=0x7f0e700c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.120+0000 7f0e6effd640 1 --2- 172.21.15.172:0/2544229057 >> v2:172.21.15.174:3300/0 conn(0x7f0e7006dc20 0x7f0e700c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.120+0000 7f0e6effd640 1 -- 172.21.15.172:0/2544229057 >> v1:172.21.15.174:6789/0 conn(0x7f0e70157ec0 legacy=0x7f0e7015b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.120+0000 7f0e6effd640 1 -- 172.21.15.172:0/2544229057 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e700c8ac0 con 0x7f0e7015b710 2024-03-16T19:31:15.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.121+0000 7f0e6effd640 1 --2- 172.21.15.172:0/2544229057 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 0x7f0e700c9080 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0e640097a0 tx=0x7f0e6402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bbda8f692499a82 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.121+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe6ec00fa40 con 0x7fe6fc0c4bb0 2024-03-16T19:31:15.104 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:15.104 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:15.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.121+0000 7f0e6e7fc640 1 -- 172.21.15.172:0/2544229057 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64030800 con 0x7f0e7015b710 2024-03-16T19:31:15.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.121+0000 7f0e6e7fc640 1 -- 172.21.15.172:0/2544229057 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e64030960 con 0x7f0e7015b710 2024-03-16T19:31:15.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.121+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=71}) v3 -- 0x7fe6fc159720 con 0x7fe6fc158ef0 2024-03-16T19:31:15.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.121+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe6fc1605a0 con 0x7fe6fc158ef0 2024-03-16T19:31:15.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.121+0000 7f0e6e7fc640 1 -- 172.21.15.172:0/2544229057 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64030cf0 con 0x7f0e7015b710 2024-03-16T19:31:15.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6f0053db0 con 0x7fe6fc158ef0 2024-03-16T19:31:15.106 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:15.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7fe70234b640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fe6fc0b2f30 0x7fe6fc1f8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6fc0c32f0 con 0x7fe6fc0b2f30 2024-03-16T19:31:15.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7f0e75d27640 1 -- 172.21.15.172:0/2544229057 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 msgr2=0x7f0e700c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7f0e75d27640 1 --2- 172.21.15.172:0/2544229057 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 0x7f0e700c9080 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0e640097a0 tx=0x7f0e6402b940 comp rx=0 tx=0).stop 2024-03-16T19:31:15.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7fe6fb7fe640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fe6fc0b2f30 0x7fe6fc1f8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7f0e75d27640 1 -- 172.21.15.172:0/2544229057 shutdown_connections 2024-03-16T19:31:15.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7f0e75d27640 1 --2- 172.21.15.172:0/2544229057 >> v2:172.21.15.174:3300/0 conn(0x7f0e7006dc20 0x7f0e700c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7f0e75d27640 1 --2- 172.21.15.172:0/2544229057 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 0x7f0e700c9080 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7f0e75d27640 1 -- 172.21.15.172:0/2544229057 >> 172.21.15.172:0/2544229057 conn(0x7f0e70070810 msgr2=0x7f0e70070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:15.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7f0e75d27640 1 -- 172.21.15.172:0/2544229057 shutdown_connections 2024-03-16T19:31:15.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.122+0000 7f0e75d27640 1 -- 172.21.15.172:0/2544229057 wait complete. 2024-03-16T19:31:15.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.123+0000 7fe6fb7fe640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fe6fc0b2f30 0x7fe6fc1f8780 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.124+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe6fc0c32f0 con 0x7fe6fc0b2f30 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.124+0000 7f0e75d27640 1 Processor -- start 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.124+0000 7f0e75d27640 1 -- start start 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e75d27640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e7006dc20 0x7f0e700c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e75d27640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e70157ec0 0x7f0e70153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e75d27640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 0x7f0e700c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e75d27640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700b2720 con 0x7f0e70157ec0 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e6effd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e70157ec0 0x7f0e70153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e6effd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e70157ec0 0x7f0e70153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42674/0 (socket says 172.21.15.172:42674) 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e6effd640 1 -- 172.21.15.172:0/612433762 learned_addr learned my addr 172.21.15.172:0/612433762 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:15.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e75d27640 1 -- 172.21.15.172:0/612433762 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e701f06f0 con 0x7f0e7006dc20 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e75d27640 1 -- 172.21.15.172:0/612433762 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700c6f00 con 0x7f0e7015b710 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.125+0000 7f0e6f7fe640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e7006dc20 0x7f0e700c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.126+0000 7f0e6ffff640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 0x7f0e700c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.126+0000 7f0e6effd640 1 -- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 msgr2=0x7f0e700c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.126+0000 7f0e6effd640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 0x7f0e700c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.126+0000 7f0e6effd640 1 -- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e7006dc20 msgr2=0x7f0e700c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.126+0000 7f0e6effd640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e7006dc20 0x7f0e700c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.126+0000 7f0e6effd640 1 -- 172.21.15.172:0/612433762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e64009400 con 0x7f0e70157ec0 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.126+0000 7f0e6ffff640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0e7015b710 0x7f0e700c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.126+0000 7f0e6effd640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0e70157ec0 0x7f0e70153df0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0e640098c0 tx=0x7f0e640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.127+0000 7f0e57fff640 1 -- 172.21.15.172:0/612433762 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6403e070 con 0x7f0e70157ec0 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.127+0000 7f0e57fff640 1 -- 172.21.15.172:0/612433762 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e64039990 con 0x7f0e70157ec0 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.127+0000 7f0e57fff640 1 -- 172.21.15.172:0/612433762 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64039d40 con 0x7f0e70157ec0 2024-03-16T19:31:15.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.127+0000 7f0e75d27640 1 -- 172.21.15.172:0/612433762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e701f0990 con 0x7f0e70157ec0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.127+0000 7f0e75d27640 1 -- 172.21.15.172:0/612433762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e701f0f50 con 0x7f0e70157ec0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.129+0000 7f0e75d27640 1 -- 172.21.15.172:0/612433762 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e7006c560 con 0x7f0e70157ec0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.129+0000 7f0e57fff640 1 -- 172.21.15.172:0/612433762 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0e640423f0 con 0x7f0e70157ec0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.129+0000 7f0e57fff640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0e5c06b3e0 0x7f0e5c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.130+0000 7f0e57fff640 1 -- 172.21.15.172:0/612433762 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f0e640ab7a0 con 0x7f0e70157ec0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.130+0000 7f0e57fff640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f0e5c0737e0 0x7f0e5c075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.130+0000 7f0e57fff640 1 -- 172.21.15.172:0/612433762 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e5c076330 con 0x7f0e5c0737e0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.130+0000 7f0e57fff640 1 -- 172.21.15.172:0/612433762 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e640ade60 con 0x7f0e70157ec0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.130+0000 7f0e6f7fe640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0e5c06b3e0 0x7f0e5c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.130+0000 7f0e6ffff640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f0e5c0737e0 0x7f0e5c075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.131+0000 7f0e6f7fe640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0e5c06b3e0 0x7f0e5c06d8a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0e60005880 tx=0x7f0e60009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.131+0000 7f0e6ffff640 1 --2- 172.21.15.172:0/612433762 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f0e5c0737e0 0x7f0e5c075c60 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.131+0000 7f0e57fff640 1 -- 172.21.15.172:0/612433762 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0e5c076330 con 0x7f0e5c0737e0 2024-03-16T19:31:15.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.153+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6fc1f8f90 con 0x7fe6fc0b2f30 2024-03-16T19:31:15.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.155+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe6fc1f8f90 con 0x7fe6fc0b2f30 2024-03-16T19:31:15.137 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:15.137 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:15.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.156+0000 7fe70234b640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fe6fc062200 0x7fe6fc0cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.156+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6fc068950 con 0x7fe6fc062200 2024-03-16T19:31:15.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.156+0000 7fe7008e8640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fe6fc062200 0x7fe6fc0cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.157+0000 7fe7008e8640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fe6fc062200 0x7fe6fc0cc0d0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.157+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe6fc068950 con 0x7fe6fc062200 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:no valid command found; 10 closest matches: 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:0 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:1 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:2 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:abort 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:assert 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:bench [] [] [] [] 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs debug_inject_read_zeros 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs files list 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:bluefs stats 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:bluestore allocator dump block 2024-03-16T19:31:15.141 INFO:teuthology.orchestra.run.smithi172.stderr:Error EINVAL: invalid command 2024-03-16T19:31:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-16T19:31:15.144 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-16T19:31:15.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.185+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6fc0628b0 con 0x7fe6fc062200 2024-03-16T19:31:15.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.186+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe6fc0628b0 con 0x7fe6fc062200 2024-03-16T19:31:15.168 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:15.168 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:15.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.187+0000 7fe70234b640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fe6fc02bde0 0x7fe6fc02c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.187+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6fc0b0bc0 con 0x7fe6fc02bde0 2024-03-16T19:31:15.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.187+0000 7fe6fbfff640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fe6fc02bde0 0x7fe6fc02c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.187+0000 7fe6fbfff640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fe6fc02bde0 0x7fe6fc02c200 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.188+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe6fc0b0bc0 con 0x7fe6fc02bde0 2024-03-16T19:31:15.178 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:15.197+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:15.189 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:15.189 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:15.189 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:15.189 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:15.189 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:15.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.217+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6fc0b0bc0 con 0x7fe6fc02bde0 2024-03-16T19:31:15.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.219+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe6fc0b0bc0 con 0x7fe6fc02bde0 2024-03-16T19:31:15.201 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:15.201 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:15.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.220+0000 7fe70234b640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fe6fc02cc50 0x7fe6fc20b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.220+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6fc0ccee0 con 0x7fe6fc02cc50 2024-03-16T19:31:15.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.220+0000 7fe6fb7fe640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fe6fc02cc50 0x7fe6fc20b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.220+0000 7fe6fb7fe640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fe6fc02cc50 0x7fe6fc20b640 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:15.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.221+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe6fc0ccee0 con 0x7fe6fc02cc50 2024-03-16T19:31:15.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.250+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6fc0ccee0 con 0x7fe6fc02cc50 2024-03-16T19:31:15.233 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:15.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.252+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe6fc0ccee0 con 0x7fe6fc02cc50 2024-03-16T19:31:15.234 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:15.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.252+0000 7fe70234b640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fe6fc02d100 0x7fe6fc20e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.252+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6fc210f70 con 0x7fe6fc02d100 2024-03-16T19:31:15.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.252+0000 7fe7008e8640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fe6fc02d100 0x7fe6fc20e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.252+0000 7fe7008e8640 1 --2- 172.21.15.172:0/3255318426 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fe6fc02d100 0x7fe6fc20e0a0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.253+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe6fc210f70 con 0x7fe6fc02d100 2024-03-16T19:31:15.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.282+0000 7fe70234b640 1 -- 172.21.15.172:0/3255318426 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6fc0cc6f0 con 0x7fe6fc02d100 2024-03-16T19:31:15.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:15.283+0000 7fe6f8ff9640 1 -- 172.21.15.172:0/3255318426 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe6fc0cc6f0 con 0x7fe6fc02d100 2024-03-16T19:31:15.265 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:15.265 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:15.270 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:15.312 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:15.331+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:15.380 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:15.399+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:15.406 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:15.406 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.831145952, 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.407 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.411 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.040919781, 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:15.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:31:15.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.417 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.040159546, 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:31:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:31:15.422 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.039345591, 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:31:15.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:31:15.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.038705666, 2024-03-16T19:31:15.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:31:15.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.038047569, 2024-03-16T19:31:15.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:31:15.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.437 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.03751499400001, 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:31:15.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:15.442 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.979343091, 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:15.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.97874089699999, 2024-03-16T19:31:15.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:31:15.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.978097952, 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:31:15.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:31:15.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.97759399100001, 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:31:15.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:31:15.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.461 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 116.819447725, 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:31:15.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:31:15.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 116.81891758899999, 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.467 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:31:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.06694833, 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.472 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:31:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:31:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.066923794, 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:31:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.06691926400001, 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:31:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:31:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.066909452, 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:31:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:31:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.06690559899999, 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:31:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:31:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.066902283, 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:31:15.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.06570670000001, 2024-03-16T19:31:15.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:31:15.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:31:15.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:31:15.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.510 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:15.510 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:15.510 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:15.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:15.512 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:15.530+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:15.612 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:15.631+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:15.713 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:15.732+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:15.721 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:15.722 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:15.722 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:15.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:15.818 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:15.837+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:15.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:31:15.893 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:15.893 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:15.893 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:15.894 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:15.894 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:15.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:15.948 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:15.967+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:15.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.010+0000 7fa6c6ddd640 1 Processor -- start 2024-03-16T19:31:15.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.010+0000 7fa6c6ddd640 1 -- start start 2024-03-16T19:31:15.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.012+0000 7fa6c6ddd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa6c00cb560 0x7fa6c015e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.012+0000 7fa6bffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa6c00cb560 0x7fa6c015e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.012+0000 7fa6bffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa6c00cb560 0x7fa6c015e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42676/0 (socket says 172.21.15.172:42676) 2024-03-16T19:31:15.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.012+0000 7fa6c4b79640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fa6c0154b90 0x7fa6c00cd1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:42184/0 (socket says 172.21.15.172:42184) 2024-03-16T19:31:15.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.013+0000 7fa6c6ddd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa6c006bc60 0x7fa6c006c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:15.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.013+0000 7fa6bffff640 1 -- 172.21.15.172:0/3417972935 learned_addr learned my addr 172.21.15.172:0/3417972935 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:15.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.013+0000 7fa6c537a640 1 --2- 172.21.15.172:0/3417972935 >> v2:172.21.15.174:3300/0 conn(0x7fa6c006bc60 0x7fa6c006c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:15.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.013+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3417972935 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c00cb360 con 0x7fa6c0154b90 2024-03-16T19:31:15.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.013+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3417972935 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c015ede0 con 0x7fa6c00cb560 2024-03-16T19:31:15.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.013+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3417972935 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c00cb9a0 con 0x7fa6c006bc60 2024-03-16T19:31:15.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.013+0000 7fa6bf7fe640 1 -- 172.21.15.172:0/3417972935 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421575390 0 0) 0x7fa6c00cb360 con 0x7fa6c0154b90 2024-03-16T19:31:15.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.013+0000 7fa6bf7fe640 1 -- 172.21.15.172:0/3417972935 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b0003520 con 0x7fa6c0154b90 2024-03-16T19:31:15.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.013+0000 7fa6bffff640 1 -- 172.21.15.172:0/3417972935 >> v2:172.21.15.174:3300/0 conn(0x7fa6c006bc60 msgr2=0x7fa6c006c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:15.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.014+0000 7fa6bffff640 1 --2- 172.21.15.172:0/3417972935 >> v2:172.21.15.174:3300/0 conn(0x7fa6c006bc60 0x7fa6c006c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.014+0000 7fa6bffff640 1 -- 172.21.15.172:0/3417972935 >> v1:172.21.15.174:6789/0 conn(0x7fa6c0154b90 legacy=0x7fa6c00cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.014+0000 7fa6bffff640 1 -- 172.21.15.172:0/3417972935 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c015a950 con 0x7fa6c00cb560 2024-03-16T19:31:15.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.015+0000 7fa6bffff640 1 --2- 172.21.15.172:0/3417972935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c00cb560 0x7fa6c015e8a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa6c00b2b80 tx=0x7fa6ac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6659f1291c696215 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:15.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.015+0000 7fa6bf7fe640 1 -- 172.21.15.172:0/3417972935 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac030860 con 0x7fa6c00cb560 2024-03-16T19:31:15.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.015+0000 7fa6bf7fe640 1 -- 172.21.15.172:0/3417972935 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac0309c0 con 0x7fa6c00cb560 2024-03-16T19:31:15.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.015+0000 7fa6bf7fe640 1 -- 172.21.15.172:0/3417972935 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac030d50 con 0x7fa6c00cb560 2024-03-16T19:31:15.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.016+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3417972935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c00cb560 msgr2=0x7fa6c015e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:15.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.016+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3417972935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c00cb560 0x7fa6c015e8a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa6c00b2b80 tx=0x7fa6ac02b9a0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.016+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3417972935 shutdown_connections 2024-03-16T19:31:15.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.016+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3417972935 >> v2:172.21.15.174:3300/0 conn(0x7fa6c006bc60 0x7fa6c006c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.016+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3417972935 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c00cb560 0x7fa6c015e8a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:15.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.016+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3417972935 >> 172.21.15.172:0/3417972935 conn(0x7fa6c0066e00 msgr2=0x7fa6c0067210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:15.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.016+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3417972935 shutdown_connections 2024-03-16T19:31:15.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.016+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3417972935 wait complete. 2024-03-16T19:31:15.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.018+0000 7fa6c6ddd640 1 Processor -- start 2024-03-16T19:31:15.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.018+0000 7fa6c6ddd640 1 -- start start 2024-03-16T19:31:16.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.019+0000 7fa6c6ddd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c006bc60 0x7fa6c0159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c4b79640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c006bc60 0x7fa6c0159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c4b79640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c006bc60 0x7fa6c0159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42690/0 (socket says 172.21.15.172:42690) 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c4b79640 1 -- 172.21.15.172:0/3206020906 learned_addr learned my addr 172.21.15.172:0/3206020906 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6c00cb560 0x7fa6c01eff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c0154b90 0x7fa6c01f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c014a4f0 con 0x7fa6c006bc60 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c015c730 con 0x7fa6c0154b90 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c015c2a0 con 0x7fa6c00cb560 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c537a640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c0154b90 0x7fa6c01f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6bffff640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6c00cb560 0x7fa6c01eff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c537a640 1 -- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6c00cb560 msgr2=0x7fa6c01eff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c537a640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6c00cb560 0x7fa6c01eff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c537a640 1 -- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c006bc60 msgr2=0x7fa6c0159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:16.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c537a640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c006bc60 0x7fa6c0159a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:16.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c537a640 1 -- 172.21.15.172:0/3206020906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0094d0 con 0x7fa6c0154b90 2024-03-16T19:31:16.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.020+0000 7fa6c4b79640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c006bc60 0x7fa6c0159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:16.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.021+0000 7fa6c537a640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c0154b90 0x7fa6c01f04d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800efc0 tx=0x7fa6b800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:16.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.021+0000 7fa6bd7fa640 1 -- 172.21.15.172:0/3206020906 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8009c00 con 0x7fa6c0154b90 2024-03-16T19:31:16.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.021+0000 7fa6bd7fa640 1 -- 172.21.15.172:0/3206020906 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b800f040 con 0x7fa6c0154b90 2024-03-16T19:31:16.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.021+0000 7fa6bd7fa640 1 -- 172.21.15.172:0/3206020906 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b80146c0 con 0x7fa6c0154b90 2024-03-16T19:31:16.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.021+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c0149f50 con 0x7fa6c0154b90 2024-03-16T19:31:16.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.021+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c015bc50 con 0x7fa6c0154b90 2024-03-16T19:31:16.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.022+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c015c410 con 0x7fa6c0154b90 2024-03-16T19:31:16.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.024+0000 7fa6bd7fa640 1 -- 172.21.15.172:0/3206020906 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa6b8014820 con 0x7fa6c0154b90 2024-03-16T19:31:16.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.024+0000 7fa6bd7fa640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa6a406b3e0 0x7fa6a406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:16.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.024+0000 7fa6bd7fa640 1 -- 172.21.15.172:0/3206020906 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6002+0+0 (secure 0 0 0) 0x7fa6b808b310 con 0x7fa6c0154b90 2024-03-16T19:31:16.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.024+0000 7fa6c4b79640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa6a406b3e0 0x7fa6a406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:16.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.025+0000 7fa6c4b79640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa6a406b3e0 0x7fa6a406d8a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40011e0 tx=0x7fa6b4007df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:16.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.027+0000 7fa6bd7fa640 1 -- 172.21.15.172:0/3206020906 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6b8055eb0 con 0x7fa6c0154b90 2024-03-16T19:31:16.015 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:16.034+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:16.020 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:16.020 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:16.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:16.020 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:16.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.020 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:16.020 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.68201364700001, 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:16.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:31:16.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.681996533, 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.025 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:31:16.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:16.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:31:16.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:31:16.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.681989253, 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:31:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:31:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.681975386, 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:31:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:31:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.681302551, 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:31:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:31:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.67900430100001, 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.049 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.67895919999999, 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:16.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:31:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:31:16.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:31:16.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.67893114500001, 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:31:16.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.678909731, 2024-03-16T19:31:16.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:31:16.062 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.67888738000001, 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.678866981, 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.678760989, 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.071 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.676969278, 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:31:16.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.676921777, 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:16.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:31:16.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:31:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 104.67668252599999, 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:16.082 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:31:16.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 99.668184381000003, 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:31:16.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 99.666844505, 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:16.091 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:31:16.094 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.663853523, 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.663222720999997, 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.100 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:31:16.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:47 2.6 e40/18 ondisk, result = 0)", 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.661740311999999, 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017677, 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377611+0000", 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9599999999999998e-07 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377618+0000", 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9990000000000002e-06 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377626+0000", 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4249999999999998e-06 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377632+0000", 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377707+0000", 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4506999999999997e-05 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377714+0000", 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5819999999999998e-06 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377787+0000", 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3232000000000006e-05 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.109 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.109 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:16.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:16.116 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:16.135+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:16.217 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:16.236+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:16.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.271+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa6c0075a70 con 0x7fa6c0154b90 2024-03-16T19:31:16.317 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:16.337+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:16.323 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:16.323 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:16.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:16.418 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:16.437+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:16.485 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:16.485 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:16.485 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:16.485 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:16.485 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:16.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:16.538 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:16.557+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.26688815200001, 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:16.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.266873679, 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:16.609 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:31:16.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:31:16.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.26636919000001, 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.613 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:31:16.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:31:16.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:31:16.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.266353794, 2024-03-16T19:31:16.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:16.621 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.26589511100001, 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:16.625 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.265348941, 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:31:16.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.629 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.26479184999999, 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:16.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:31:16.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.26359010199999, 2024-03-16T19:31:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:31:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.263011393, 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.262961866, 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.262001223, 2024-03-16T19:31:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.258287025, 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:31:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:31:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 96.817886113, 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 95.252333109999995, 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.657 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:31:16.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:31:16.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:31:16.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 93.744734385000001, 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.661 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:31:16.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPull(2.e e67/66 cost 1048576)", 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.840060476, 2024-03-16T19:31:16.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026662799999999998, 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0500000000000003e-07 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784292+0000", 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5139e-05 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784304+0000", 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1501000000000001e-05 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784313+0000", 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8189999999999998e-06 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784384+0000", 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1192999999999994e-05 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784543+0000", 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159371 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:16.669 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:16.669 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:16.670 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:16.671 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:16.689+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:16.770 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:16.790+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:16.777 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:16.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:16.778 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:16.778 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:16.786 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:16.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.836+0000 7fa6bd7fa640 1 -- 172.21.15.172:0/3206020906 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v71) v1 ==== 101+0+0 (secure 0 0 0) 0x7fa6b8018020 con 0x7fa6c0154b90 2024-03-16T19:31:16.817 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:31:16.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa6a406b3e0 msgr2=0x7fa6a406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:16.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa6a406b3e0 0x7fa6a406d8a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40011e0 tx=0x7fa6b4007df0 comp rx=0 tx=0).stop 2024-03-16T19:31:16.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c0154b90 msgr2=0x7fa6c01f04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:16.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c0154b90 0x7fa6c01f04d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800efc0 tx=0x7fa6b800c490 comp rx=0 tx=0).stop 2024-03-16T19:31:16.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 shutdown_connections 2024-03-16T19:31:16.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c0154b90 0x7fa6c01f04d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:16.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa6a406b3e0 0x7fa6a406d8a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:16.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6c00cb560 0x7fa6c01eff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:16.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 --2- 172.21.15.172:0/3206020906 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa6c006bc60 0x7fa6c0159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:16.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 >> 172.21.15.172:0/3206020906 conn(0x7fa6c0066e00 msgr2=0x7fa6c014cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:16.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 shutdown_connections 2024-03-16T19:31:16.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:16.839+0000 7fa6c6ddd640 1 -- 172.21.15.172:0/3206020906 wait complete. 2024-03-16T19:31:16.871 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:16.890+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:16.932 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:16.932 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:16.933 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:16.933 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:16.933 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:16.941 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:16.986 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:17.005+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:17.048 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.71127821, 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.710050788, 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.057 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.709648082, 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.709039331, 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:31:17.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:31:17.065 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.70859546200001, 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:31:17.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.708452686, 2024-03-16T19:31:17.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.073 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.708438145, 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:17.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:31:17.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:31:17.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.70556044999999, 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:31:17.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:31:17.082 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.082 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.082 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.082 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.082 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.082 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:17.083 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:17.084 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:17.102+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:17.184 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:17.203+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:17.231 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:17.231 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:17.240 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:17.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:31:17.315 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:17.334+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:17.387 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:17.406+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:17.391 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:17.391 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:17.391 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:17.391 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:17.391 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:17.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:17.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.442+0000 7fd8f873a640 1 Processor -- start 2024-03-16T19:31:17.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.443+0000 7fd8f873a640 1 -- start start 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f873a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd8f00c9cc0 0x7fd8f0070880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f873a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd8f0070dc0 0x7fd8f015b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f873a640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00c9380 con 0x7fd8f0154de0 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f873a640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f006d9e0 con 0x7fd8f0070dc0 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f873a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f00cab00 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f6cd7640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd8f0070dc0 0x7fd8f015b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f6cd7640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fd8f0070dc0 0x7fd8f015b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42696/0 (socket says 172.21.15.172:42696) 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f5cd5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd8f00c9cc0 0x7fd8f0070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f6cd7640 1 -- 172.21.15.172:0/1882866669 learned_addr learned my addr 172.21.15.172:0/1882866669 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:17.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f6cd7640 1 -- 172.21.15.172:0/1882866669 >> v2:172.21.15.174:3300/0 conn(0x7fd8f00c9cc0 msgr2=0x7fd8f0070880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:17.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f6cd7640 1 --2- 172.21.15.172:0/1882866669 >> v2:172.21.15.174:3300/0 conn(0x7fd8f00c9cc0 0x7fd8f0070880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:17.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.444+0000 7fd8f6cd7640 1 -- 172.21.15.172:0/1882866669 >> v1:172.21.15.172:6789/0 conn(0x7fd8f0154de0 legacy=0x7fd8f015ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:17.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.445+0000 7fd8f6cd7640 1 -- 172.21.15.172:0/1882866669 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00c90a0 con 0x7fd8f0070dc0 2024-03-16T19:31:17.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.445+0000 7fd8f5cd5640 1 --2- 172.21.15.172:0/1882866669 >> v2:172.21.15.174:3300/0 conn(0x7fd8f00c9cc0 0x7fd8f0070880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:17.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.445+0000 7fd8f6cd7640 1 --2- 172.21.15.172:0/1882866669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8f0070dc0 0x7fd8f015b1e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00713d0 tx=0x7fd8e802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac161f69108c808 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f54d4640 1 -- 172.21.15.172:0/1882866669 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e802c9f0 con 0x7fd8f0070dc0 2024-03-16T19:31:17.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f54d4640 1 -- 172.21.15.172:0/1882866669 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e802cb50 con 0x7fd8f0070dc0 2024-03-16T19:31:17.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f54d4640 1 -- 172.21.15.172:0/1882866669 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80343f0 con 0x7fd8f0070dc0 2024-03-16T19:31:17.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f873a640 1 -- 172.21.15.172:0/1882866669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8f0070dc0 msgr2=0x7fd8f015b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:17.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f873a640 1 --2- 172.21.15.172:0/1882866669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8f0070dc0 0x7fd8f015b1e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00713d0 tx=0x7fd8e802be50 comp rx=0 tx=0).stop 2024-03-16T19:31:17.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f873a640 1 -- 172.21.15.172:0/1882866669 shutdown_connections 2024-03-16T19:31:17.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f873a640 1 --2- 172.21.15.172:0/1882866669 >> v2:172.21.15.174:3300/0 conn(0x7fd8f00c9cc0 0x7fd8f0070880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:17.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f873a640 1 --2- 172.21.15.172:0/1882866669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8f0070dc0 0x7fd8f015b1e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:17.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f873a640 1 -- 172.21.15.172:0/1882866669 >> 172.21.15.172:0/1882866669 conn(0x7fd8f006e880 msgr2=0x7fd8f006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:17.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.446+0000 7fd8f873a640 1 -- 172.21.15.172:0/1882866669 shutdown_connections 2024-03-16T19:31:17.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.447+0000 7fd8f873a640 1 -- 172.21.15.172:0/1882866669 wait complete. 2024-03-16T19:31:17.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.449+0000 7fd8f873a640 1 Processor -- start 2024-03-16T19:31:17.430 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.449+0000 7fd8f873a640 1 -- start start 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.450+0000 7fd8f873a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f0070dc0 0x7fd8f01f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.450+0000 7fd8f873a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f00c9cc0 0x7fd8f01f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.450+0000 7fd8f873a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8f0154de0 0x7fd8f014e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.450+0000 7fd8f64d6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f0070dc0 0x7fd8f01f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.450+0000 7fd8f64d6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f0070dc0 0x7fd8f01f36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:41034/0 (socket says 172.21.15.172:41034) 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.450+0000 7fd8f64d6640 1 -- 172.21.15.172:0/3320873125 learned_addr learned my addr 172.21.15.172:0/3320873125 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.450+0000 7fd8f5cd5640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f00c9cc0 0x7fd8f01f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.450+0000 7fd8f6cd7640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8f0154de0 0x7fd8f014e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00c8bd0 con 0x7fd8f0154de0 2024-03-16T19:31:17.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00c6650 con 0x7fd8f0070dc0 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00cc300 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f5cd5640 1 -- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f0070dc0 msgr2=0x7fd8f01f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f5cd5640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f0070dc0 0x7fd8f01f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f5cd5640 1 -- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8f0154de0 msgr2=0x7fd8f014e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f5cd5640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8f0154de0 0x7fd8f014e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f5cd5640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e0009400 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f64d6640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f0070dc0 0x7fd8f01f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.451+0000 7fd8f5cd5640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f00c9cc0 0x7fd8f01f3be0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e900 tx=0x7fd8e000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.452+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000ca00 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.452+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e000cb60 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.452+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0013470 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.452+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e8009b30 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.452+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f00cc8c0 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.435 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.453+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd8f0068d20 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.454+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd8e0005990 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.454+0000 7fd8e6ffd640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd8d406b3e0 0x7fd8d406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.455+0000 7fd8f64d6640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd8d406b3e0 0x7fd8d406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.455+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6002+0+0 (secure 0 0 0) 0x7fd8e008a430 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.455+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v71) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd8e008a7a0 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.455+0000 7fd8f873a640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd8f015b010 0x7fd8f015b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.455+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8f01551c0 con 0x7fd8f015b010 2024-03-16T19:31:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.455+0000 7fd8f6cd7640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd8f015b010 0x7fd8f015b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.456+0000 7fd8f64d6640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd8d406b3e0 0x7fd8d406d8a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009d80 tx=0x7fd8ec009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.456+0000 7fd8f6cd7640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fd8f015b010 0x7fd8f015b3f0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.456+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd8e802c7f0 con 0x7fd8f015b010 2024-03-16T19:31:17.452 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.471+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd8f00681c0 con 0x7fd8f015b010 2024-03-16T19:31:17.454 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.473+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd8f00681c0 con 0x7fd8f015b010 2024-03-16T19:31:17.454 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:17.455 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:17.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.473+0000 7fd8f873a640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd8f014f2a0 0x7fd8f0061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.473+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8f014f680 con 0x7fd8f014f2a0 2024-03-16T19:31:17.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.473+0000 7fd8f64d6640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd8f014f2a0 0x7fd8f0061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.455 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.474+0000 7fd8f64d6640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fd8f014f2a0 0x7fd8f0061930 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.475+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd8f014f680 con 0x7fd8f014f2a0 2024-03-16T19:31:17.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.490+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd8f0154580 con 0x7fd8f014f2a0 2024-03-16T19:31:17.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.492+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd8ec0099d0 con 0x7fd8f014f2a0 2024-03-16T19:31:17.474 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:17.475 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:17.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.493+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=72}) v3 -- 0x7fd8f0062120 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.493+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8f0062460 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.493+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8e0009280 con 0x7fd8f00c9cc0 2024-03-16T19:31:17.475 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:17.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.493+0000 7fd8f873a640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd8f00aedb0 0x7fd8f00af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.493+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8f00af700 con 0x7fd8f00aedb0 2024-03-16T19:31:17.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.493+0000 7fd8f5cd5640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd8f00aedb0 0x7fd8f00af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.494+0000 7fd8f5cd5640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fd8f00aedb0 0x7fd8f00af1c0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.495+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd8f00af700 con 0x7fd8f00aedb0 2024-03-16T19:31:17.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:17.506+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:17.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.509+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd8f002c0b0 con 0x7fd8f00aedb0 2024-03-16T19:31:17.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.510+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd8f002c0b0 con 0x7fd8f00aedb0 2024-03-16T19:31:17.492 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:17.492 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:17.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.511+0000 7fd8f873a640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd8f002caf0 0x7fd8f002cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.511+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8f00ceab0 con 0x7fd8f002caf0 2024-03-16T19:31:17.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.511+0000 7fd8f6cd7640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd8f002caf0 0x7fd8f002cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.511+0000 7fd8f6cd7640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fd8f002caf0 0x7fd8f002cf30 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.512+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd8f00ceab0 con 0x7fd8f002caf0 2024-03-16T19:31:17.508 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:17.508 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 106.17066637800001, 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:31:17.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:17.513 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 106.17058726, 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:31:17.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.517 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 106.169973136, 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:31:17.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 106.169533067, 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.522 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:31:17.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:31:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:17.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 106.169291477, 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:31:17.526 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:31:17.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:31:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:31:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:17.530 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:17.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.540+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd8f00cef80 con 0x7fd8f002caf0 2024-03-16T19:31:17.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.542+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd8f00cef80 con 0x7fd8f002caf0 2024-03-16T19:31:17.531 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:17.532 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:17.533 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:17.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.542+0000 7fd8f873a640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd8f00cd8b0 0x7fd8f00cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.542+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8f00ce230 con 0x7fd8f00cd8b0 2024-03-16T19:31:17.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.543+0000 7fd8f64d6640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd8f00cd8b0 0x7fd8f00cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.543+0000 7fd8f64d6640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fd8f00cd8b0 0x7fd8f00cdcf0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.544+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd8f00ce230 con 0x7fd8f00cd8b0 2024-03-16T19:31:17.553 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.572+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd8f00ce230 con 0x7fd8f00cd8b0 2024-03-16T19:31:17.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.574+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd8f00ce230 con 0x7fd8f00cd8b0 2024-03-16T19:31:17.555 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:17.556 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:17.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.574+0000 7fd8f873a640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd8f02049c0 0x7fd8f0206db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.574+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8f00ce910 con 0x7fd8f02049c0 2024-03-16T19:31:17.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.574+0000 7fd8f5cd5640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd8f02049c0 0x7fd8f0206db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.574+0000 7fd8f5cd5640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fd8f02049c0 0x7fd8f0206db0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.575+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd8f00ce910 con 0x7fd8f02049c0 2024-03-16T19:31:17.579 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.598+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd8f00ceb10 con 0x7fd8f02049c0 2024-03-16T19:31:17.581 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.601+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd8f00ceb10 con 0x7fd8f02049c0 2024-03-16T19:31:17.582 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:17.582 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:17.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.601+0000 7fd8f873a640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd8f0207800 0x7fd8f0209bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:17.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.601+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8f00cea80 con 0x7fd8f0207800 2024-03-16T19:31:17.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.601+0000 7fd8f6cd7640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd8f0207800 0x7fd8f0209bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:17.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.601+0000 7fd8f6cd7640 1 --2- 172.21.15.172:0/3320873125 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fd8f0207800 0x7fd8f0209bf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:17.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.602+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd8f00cea80 con 0x7fd8f0207800 2024-03-16T19:31:17.587 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:17.606+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:17.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.630+0000 7fd8f873a640 1 -- 172.21.15.172:0/3320873125 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd8f00cea80 con 0x7fd8f0207800 2024-03-16T19:31:17.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:17.632+0000 7fd8e6ffd640 1 -- 172.21.15.172:0/3320873125 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd8f00cea80 con 0x7fd8f0207800 2024-03-16T19:31:17.613 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:17.613 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:17.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:17.679 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:17.679 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:17.688 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:17.688 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:17.707+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:17.789 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:17.808+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:17.857 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:17.857 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:17.865 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:17.909 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:17.928+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:17.972 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:17.972 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:17.980 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:18.022 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:18.041+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:18.092 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:18.111+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:18.131 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:18.131 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:18.140 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:18.193 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:18.212+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:18.293 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:18.293 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:18.294 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:18.313+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:18.302 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:18.395 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:18.414+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:18.453 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:18.453 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:18.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:18.507 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:18.526+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:18.597 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:18.616+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:18.635 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:18.635 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:18.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:18.698 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:18.717+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:18.800 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:18.819+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:18.802 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:18.802 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:18.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:18.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:31:18.900 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:18.919+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:18.970 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:18.970 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:18.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:19.001 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:19.020+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:19.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.021+0000 7efd201d7640 1 Processor -- start 2024-03-16T19:31:19.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.022+0000 7efd201d7640 1 -- start start 2024-03-16T19:31:19.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd201d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 0x7efd1806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd201d7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efd180c79d0 0x7efd1806b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd201d7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd18072610 con 0x7efd18152a20 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd201d7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd180c9890 con 0x7efd180c79d0 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd201d7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd180729c0 con 0x7efd180c93a0 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd1df73640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7efd18152a20 0x7efd180ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51120/0 (socket says 172.21.15.172:51120) 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd1df73640 1 -- 172.21.15.172:0/4099859649 learned_addr learned my addr 172.21.15.172:0/4099859649 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd1d772640 1 --2- 172.21.15.172:0/4099859649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 0x7efd1806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd1e774640 1 --2- 172.21.15.172:0/4099859649 >> v2:172.21.15.174:3300/0 conn(0x7efd180c79d0 0x7efd1806b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd1e774640 1 -- 172.21.15.172:0/4099859649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 msgr2=0x7efd1806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd1e774640 1 --2- 172.21.15.172:0/4099859649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 0x7efd1806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd1e774640 1 -- 172.21.15.172:0/4099859649 >> v1:172.21.15.172:6789/0 conn(0x7efd18152a20 legacy=0x7efd180ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.023+0000 7efd1e774640 1 -- 172.21.15.172:0/4099859649 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd18072ca0 con 0x7efd180c79d0 2024-03-16T19:31:19.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.025+0000 7efd1e774640 1 --2- 172.21.15.172:0/4099859649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd180c79d0 0x7efd1806b2b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efd10009d90 tx=0x7efd1002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=629f2d7276466801 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.025+0000 7efd1cf71640 1 -- 172.21.15.172:0/4099859649 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1002cd00 con 0x7efd180c79d0 2024-03-16T19:31:19.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.025+0000 7efd1cf71640 1 -- 172.21.15.172:0/4099859649 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1002ce60 con 0x7efd180c79d0 2024-03-16T19:31:19.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.025+0000 7efd1cf71640 1 -- 172.21.15.172:0/4099859649 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10034660 con 0x7efd180c79d0 2024-03-16T19:31:19.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.026+0000 7efd201d7640 1 -- 172.21.15.172:0/4099859649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd180c79d0 msgr2=0x7efd1806b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.026+0000 7efd201d7640 1 --2- 172.21.15.172:0/4099859649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd180c79d0 0x7efd1806b2b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efd10009d90 tx=0x7efd1002bf50 comp rx=0 tx=0).stop 2024-03-16T19:31:19.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.026+0000 7efd201d7640 1 -- 172.21.15.172:0/4099859649 shutdown_connections 2024-03-16T19:31:19.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.026+0000 7efd201d7640 1 --2- 172.21.15.172:0/4099859649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd180c79d0 0x7efd1806b2b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.026+0000 7efd201d7640 1 --2- 172.21.15.172:0/4099859649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 0x7efd1806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.026+0000 7efd201d7640 1 -- 172.21.15.172:0/4099859649 >> 172.21.15.172:0/4099859649 conn(0x7efd1806db70 msgr2=0x7efd1806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:19.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.026+0000 7efd201d7640 1 -- 172.21.15.172:0/4099859649 shutdown_connections 2024-03-16T19:31:19.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.026+0000 7efd201d7640 1 -- 172.21.15.172:0/4099859649 wait complete. 2024-03-16T19:31:19.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.028+0000 7efd201d7640 1 Processor -- start 2024-03-16T19:31:19.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.028+0000 7efd201d7640 1 -- start start 2024-03-16T19:31:19.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.029+0000 7efd201d7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd180c79d0 0x7efd18142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.029+0000 7efd201d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 0x7efd180c2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.030+0000 7efd1df73640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd180c79d0 0x7efd18142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.030+0000 7efd1df73640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd180c79d0 0x7efd18142fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:44878/0 (socket says 172.21.15.172:44878) 2024-03-16T19:31:19.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.030+0000 7efd1df73640 1 -- 172.21.15.172:0/3705273215 learned_addr learned my addr 172.21.15.172:0/3705273215 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:19.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.030+0000 7efd1d772640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 0x7efd180c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.030+0000 7efd201d7640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd18152a20 0x7efd180c2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.030+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd181f37d0 con 0x7efd180c79d0 2024-03-16T19:31:19.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.030+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd181f45f0 con 0x7efd18152a20 2024-03-16T19:31:19.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.030+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1814d590 con 0x7efd180c93a0 2024-03-16T19:31:19.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.030+0000 7efd1e774640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd18152a20 0x7efd180c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.031+0000 7efd1df73640 1 -- 172.21.15.172:0/3705273215 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 msgr2=0x7efd180c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.031+0000 7efd1df73640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 0x7efd180c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.031+0000 7efd1df73640 1 -- 172.21.15.172:0/3705273215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd18152a20 msgr2=0x7efd180c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.031+0000 7efd1df73640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd18152a20 0x7efd180c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.031+0000 7efd1df73640 1 -- 172.21.15.172:0/3705273215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10009a70 con 0x7efd180c79d0 2024-03-16T19:31:19.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.031+0000 7efd1df73640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd180c79d0 0x7efd18142fb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c3c0 tx=0x7efd1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.031+0000 7efd0e7fc640 1 -- 172.21.15.172:0/3705273215 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14005ce0 con 0x7efd180c79d0 2024-03-16T19:31:19.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.031+0000 7efd0e7fc640 1 -- 172.21.15.172:0/3705273215 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd14005e40 con 0x7efd180c79d0 2024-03-16T19:31:19.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.032+0000 7efd0e7fc640 1 -- 172.21.15.172:0/3705273215 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd140072f0 con 0x7efd180c79d0 2024-03-16T19:31:19.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.032+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd18152f80 con 0x7efd180c79d0 2024-03-16T19:31:19.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.032+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd181f3a00 con 0x7efd180c79d0 2024-03-16T19:31:19.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.034+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd18075a70 con 0x7efd180c79d0 2024-03-16T19:31:19.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.034+0000 7efd0e7fc640 1 -- 172.21.15.172:0/3705273215 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7efd14007510 con 0x7efd180c79d0 2024-03-16T19:31:19.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.034+0000 7efd0e7fc640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcfc06b400 0x7efcfc06d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.034+0000 7efd0e7fc640 1 -- 172.21.15.172:0/3705273215 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6002+0+0 (secure 0 0 0) 0x7efd14089140 con 0x7efd180c79d0 2024-03-16T19:31:19.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.034+0000 7efd1d772640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcfc06b400 0x7efcfc06d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.035+0000 7efd1d772640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcfc06b400 0x7efcfc06d8c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efd180c45d0 tx=0x7efd08009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.043+0000 7efd0e7fc640 1 -- 172.21.15.172:0/3705273215 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd14053e00 con 0x7efd180c79d0 2024-03-16T19:31:19.102 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:19.121+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:19.146 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:19.146 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:19.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:19.204 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:19.222+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:19.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.235+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7efd1806cd40 con 0x7efd180c79d0 2024-03-16T19:31:19.303 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:19.322+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:19.315 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:19.315 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:19.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:19.404 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:19.423+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:19.505 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:19.524+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:19.538 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:19.538 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:19.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:19.605 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:19.625+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:19.619 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:31:19.707 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:19.726+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:19.726 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:19.726 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:19.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:19.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.792+0000 7fda8fe07640 1 Processor -- start 2024-03-16T19:31:19.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.792+0000 7fda8fe07640 1 -- start start 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.793+0000 7fda8fe07640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fda88154ed0 0x7fda8815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.793+0000 7fda8fe07640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b4700 0x7fda880b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.793+0000 7fda8fe07640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda880b4270 con 0x7fda880b3630 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.793+0000 7fda8fe07640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda881552b0 con 0x7fda88154ed0 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.793+0000 7fda8fe07640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8815b4b0 con 0x7fda880b4700 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8d3a2640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fda880b3630 0x7fda880cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51128/0 (socket says 172.21.15.172:51128) 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8d3a2640 1 -- 172.21.15.172:0/2948509761 learned_addr learned my addr 172.21.15.172:0/2948509761 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8dba3640 1 --2- 172.21.15.172:0/2948509761 >> v2:172.21.15.174:3300/0 conn(0x7fda88154ed0 0x7fda8815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2948509761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b4700 0x7fda880b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8cba1640 1 -- 172.21.15.172:0/2948509761 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793208127 0 0) 0x7fda880b4270 con 0x7fda880b3630 2024-03-16T19:31:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8cba1640 1 -- 172.21.15.172:0/2948509761 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda74003520 con 0x7fda880b3630 2024-03-16T19:31:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8e3a4640 1 -- 172.21.15.172:0/2948509761 >> v2:172.21.15.174:3300/0 conn(0x7fda88154ed0 msgr2=0x7fda8815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2948509761 >> v2:172.21.15.174:3300/0 conn(0x7fda88154ed0 0x7fda8815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8e3a4640 1 -- 172.21.15.172:0/2948509761 >> v1:172.21.15.172:6789/0 conn(0x7fda880b3630 legacy=0x7fda880cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.794+0000 7fda8e3a4640 1 -- 172.21.15.172:0/2948509761 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8815b790 con 0x7fda880b4700 2024-03-16T19:31:19.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.795+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2948509761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b4700 0x7fda880b4b50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fda80009d90 tx=0x7fda8002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e56fa208724fa61 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.795+0000 7fda8cba1640 1 -- 172.21.15.172:0/2948509761 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8002cc60 con 0x7fda880b4700 2024-03-16T19:31:19.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.796+0000 7fda8cba1640 1 -- 172.21.15.172:0/2948509761 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8002cdc0 con 0x7fda880b4700 2024-03-16T19:31:19.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.796+0000 7fda8cba1640 1 -- 172.21.15.172:0/2948509761 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda80034660 con 0x7fda880b4700 2024-03-16T19:31:19.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.796+0000 7fda8fe07640 1 -- 172.21.15.172:0/2948509761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b4700 msgr2=0x7fda880b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.796+0000 7fda8fe07640 1 --2- 172.21.15.172:0/2948509761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b4700 0x7fda880b4b50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fda80009d90 tx=0x7fda8002beb0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.797+0000 7fda8fe07640 1 -- 172.21.15.172:0/2948509761 shutdown_connections 2024-03-16T19:31:19.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.797+0000 7fda8fe07640 1 --2- 172.21.15.172:0/2948509761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b4700 0x7fda880b4b50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.797+0000 7fda8fe07640 1 --2- 172.21.15.172:0/2948509761 >> v2:172.21.15.174:3300/0 conn(0x7fda88154ed0 0x7fda8815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.797+0000 7fda8fe07640 1 -- 172.21.15.172:0/2948509761 >> 172.21.15.172:0/2948509761 conn(0x7fda8806bf60 msgr2=0x7fda8806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:19.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.797+0000 7fda8fe07640 1 -- 172.21.15.172:0/2948509761 shutdown_connections 2024-03-16T19:31:19.779 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.797+0000 7fda8fe07640 1 -- 172.21.15.172:0/2948509761 wait complete. 2024-03-16T19:31:19.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.799+0000 7fda8fe07640 1 Processor -- start 2024-03-16T19:31:19.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.799+0000 7fda8fe07640 1 -- start start 2024-03-16T19:31:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.800+0000 7fda8fe07640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b3630 0x7fda880c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.800+0000 7fda8fe07640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda880b4700 0x7fda880c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.800+0000 7fda8dba3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b3630 0x7fda880c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8dba3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b3630 0x7fda880c3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:50992/0 (socket says 172.21.15.172:50992) 2024-03-16T19:31:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8fe07640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda88154ed0 0x7fda8815e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8fe07640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880d0a50 con 0x7fda880b4700 2024-03-16T19:31:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8fe07640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880d0cf0 con 0x7fda88154ed0 2024-03-16T19:31:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8fe07640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda880d0fd0 con 0x7fda880b3630 2024-03-16T19:31:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8dba3640 1 -- 172.21.15.172:0/2957366115 learned_addr learned my addr 172.21.15.172:0/2957366115 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:19.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8d3a2640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda880b4700 0x7fda880c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda88154ed0 0x7fda8815e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8dba3640 1 -- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda88154ed0 msgr2=0x7fda8815e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8dba3640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda88154ed0 0x7fda8815e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8dba3640 1 -- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda880b4700 msgr2=0x7fda880c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8dba3640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda880b4700 0x7fda880c51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8dba3640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda80009a70 con 0x7fda880b3630 2024-03-16T19:31:19.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8d3a2640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda880b4700 0x7fda880c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:19.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.801+0000 7fda8dba3640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fda880b3630 0x7fda880c3de0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fda8400e870 tx=0x7fda8400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.802+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda840094f0 con 0x7fda880b3630 2024-03-16T19:31:19.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.802+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8400c940 con 0x7fda880b3630 2024-03-16T19:31:19.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.802+0000 7fda8fe07640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda880ce1b0 con 0x7fda880b3630 2024-03-16T19:31:19.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.803+0000 7fda8fe07640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda880c4320 con 0x7fda880b3630 2024-03-16T19:31:19.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.803+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8400ccf0 con 0x7fda880b3630 2024-03-16T19:31:19.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.804+0000 7fda8fe07640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fda88068d20 con 0x7fda880b3630 2024-03-16T19:31:19.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.804+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fda84013470 con 0x7fda880b3630 2024-03-16T19:31:19.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.805+0000 7fda7e7fc640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fda6c06b3b0 0x7fda6c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.805+0000 7fda8d3a2640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fda6c06b3b0 0x7fda6c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.805+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6002+0+0 (secure 0 0 0) 0x7fda84089490 con 0x7fda880b3630 2024-03-16T19:31:19.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.805+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v71) v1 ==== 54+0+16 (secure 0 0 0) 0x7fda840896d0 con 0x7fda880b3630 2024-03-16T19:31:19.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.806+0000 7fda8d3a2640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fda6c06b3b0 0x7fda6c06d870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fda880ca820 tx=0x7fda78009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.806+0000 7fda6bfff640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fda64001650 0x7fda64003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.806+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda64006bb0 con 0x7fda64001650 2024-03-16T19:31:19.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.806+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fda64001650 0x7fda64003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.806+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fda64001650 0x7fda64003b10 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.807+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda64006bb0 con 0x7fda64001650 2024-03-16T19:31:19.807 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:19.826+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:19.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.840+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fda64005c00 con 0x7fda64001650 2024-03-16T19:31:19.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.842+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fda64005c00 con 0x7fda64001650 2024-03-16T19:31:19.823 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:19.824 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:19.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.842+0000 7fda6bfff640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fda64004190 0x7fda64004640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.842+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda64004b80 con 0x7fda64004190 2024-03-16T19:31:19.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.843+0000 7fda8dba3640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fda64004190 0x7fda64004640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.843+0000 7fda8dba3640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fda64004190 0x7fda64004640 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.845+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda64004b80 con 0x7fda64004190 2024-03-16T19:31:19.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.844+0000 7efd0e7fc640 1 -- 172.21.15.172:0/3705273215 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v72) v1 ==== 87+0+0 (secure 0 0 0) 0x7efd14017090 con 0x7efd180c79d0 2024-03-16T19:31:19.826 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:31:19.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.847+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcfc06b400 msgr2=0x7efcfc06d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.847+0000 7efd201d7640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcfc06b400 0x7efcfc06d8c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efd180c45d0 tx=0x7efd08009190 comp rx=0 tx=0).stop 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.847+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd180c79d0 msgr2=0x7efd18142fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.847+0000 7efd201d7640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd180c79d0 0x7efd18142fb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c3c0 tx=0x7efd1400c880 comp rx=0 tx=0).stop 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.848+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 shutdown_connections 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.848+0000 7efd201d7640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd18152a20 0x7efd180c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.848+0000 7efd201d7640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcfc06b400 0x7efcfc06d8c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.848+0000 7efd201d7640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd180c93a0 0x7efd180c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.848+0000 7efd201d7640 1 --2- 172.21.15.172:0/3705273215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efd180c79d0 0x7efd18142fb0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.848+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 >> 172.21.15.172:0/3705273215 conn(0x7efd1806db70 msgr2=0x7efd181512f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.848+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 shutdown_connections 2024-03-16T19:31:19.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.848+0000 7efd201d7640 1 -- 172.21.15.172:0/3705273215 wait complete. 2024-03-16T19:31:19.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.875+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fda64005050 con 0x7fda64004190 2024-03-16T19:31:19.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.878+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fda64005050 con 0x7fda64004190 2024-03-16T19:31:19.859 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:19.860 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:19.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.878+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=72}) v3 -- 0x7fda64005520 con 0x7fda880b3630 2024-03-16T19:31:19.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.878+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fda6400b620 con 0x7fda880b3630 2024-03-16T19:31:19.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.879+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== mon.2 v2:172.21.15.174:3301/0 7 ==== osd_map(72..72 src has 1..72) v4 ==== 288+0+0 (secure 0 0 0) 0x7fda84057dd0 con 0x7fda880b3630 2024-03-16T19:31:19.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.879+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== mon.2 v2:172.21.15.174:3301/0 8 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda8400ce50 con 0x7fda880b3630 2024-03-16T19:31:19.860 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:19.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.879+0000 7fda6bfff640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fda64009660 0x7fda6400ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.879+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda6400eb70 con 0x7fda64009660 2024-03-16T19:31:19.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.879+0000 7fda8d3a2640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fda64009660 0x7fda6400ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.880+0000 7fda8d3a2640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fda64009660 0x7fda6400ba90 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.881+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda6400eb70 con 0x7fda64009660 2024-03-16T19:31:19.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.910+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fda6400db60 con 0x7fda64009660 2024-03-16T19:31:19.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.912+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fda6400db60 con 0x7fda64009660 2024-03-16T19:31:19.894 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:19.894 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:19.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.913+0000 7fda6bfff640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fda6400c110 0x7fda6400c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.913+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda6400cae0 con 0x7fda6400c110 2024-03-16T19:31:19.895 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:19.895 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:19.895 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:19.895 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:19.895 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:19.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.913+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fda6400c110 0x7fda6400c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.913+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fda6400c110 0x7fda6400c5a0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.914+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda6400cae0 con 0x7fda6400c110 2024-03-16T19:31:19.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:19.908 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:19.927+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:19.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.943+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fda6400cfb0 con 0x7fda6400c110 2024-03-16T19:31:19.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.945+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fda6400cfb0 con 0x7fda6400c110 2024-03-16T19:31:19.927 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:19.927 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:19.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.945+0000 7fda6bfff640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fda6400d3b0 0x7fda64012d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.945+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda64015d00 con 0x7fda6400d3b0 2024-03-16T19:31:19.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.945+0000 7fda8dba3640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fda6400d3b0 0x7fda64012d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.946+0000 7fda8dba3640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fda6400d3b0 0x7fda64012d90 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.946+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda64015d00 con 0x7fda6400d3b0 2024-03-16T19:31:19.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.963+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fda6400cbc0 con 0x7fda6400d3b0 2024-03-16T19:31:19.946 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.965+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fda6400cbc0 con 0x7fda6400d3b0 2024-03-16T19:31:19.946 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:19.947 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:19.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.965+0000 7fda6bfff640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fda640151a0 0x7fda64015600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.965+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda64019110 con 0x7fda640151a0 2024-03-16T19:31:19.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.965+0000 7fda8d3a2640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fda640151a0 0x7fda64015600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.966+0000 7fda8d3a2640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fda640151a0 0x7fda64015600 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.966+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda64019110 con 0x7fda640151a0 2024-03-16T19:31:19.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.983+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fda64017f90 con 0x7fda640151a0 2024-03-16T19:31:19.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.985+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fda64017f90 con 0x7fda640151a0 2024-03-16T19:31:19.966 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:19.967 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:19.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.985+0000 7fda6bfff640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fda64018610 0x7fda64018a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:19.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.985+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda6401c4c0 con 0x7fda64018610 2024-03-16T19:31:19.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.988+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fda64018610 0x7fda64018a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:19.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.988+0000 7fda8e3a4640 1 --2- 172.21.15.172:0/2957366115 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fda64018610 0x7fda64018a30 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:19.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:19.989+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda6401c4c0 con 0x7fda64018610 2024-03-16T19:31:19.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:20.003+0000 7fda6bfff640 1 -- 172.21.15.172:0/2957366115 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fda64013300 con 0x7fda64018610 2024-03-16T19:31:19.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:20.005+0000 7fda7e7fc640 1 -- 172.21.15.172:0/2957366115 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fda64013300 con 0x7fda64018610 2024-03-16T19:31:19.986 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:19.986 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:19.990 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:19.990 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:20.032 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:20.051+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:20.095 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:20.095 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:20.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:20.095 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:20.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.75701429999999, 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.756999827, 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:31:20.099 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:31:20.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:31:20.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.75649533799999, 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:20.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:31:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.106 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.756479942, 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.75602125899999, 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:31:20.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:31:20.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:20.114 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.755475089, 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.754917998, 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:20.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:31:20.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:31:20.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.75371625, 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:31:20.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.753137541, 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.753088014, 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:31:20.134 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.752127371, 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:20.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:31:20.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.748413173, 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:31:20.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:31:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.308012261, 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:31:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 98.742459257999997, 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:31:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 97.234860533000003, 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:31:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPull(2.e e67/66 cost 1048576)", 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.330186624, 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026662799999999998, 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0500000000000003e-07 2024-03-16T19:31:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784292+0000", 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5139e-05 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784304+0000", 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1501000000000001e-05 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784313+0000", 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8189999999999998e-06 2024-03-16T19:31:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784384+0000", 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1192999999999994e-05 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784543+0000", 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159371 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:20.157 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:20.158 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-16T19:31:20.158 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:31:20.158 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-16T19:31:20.158 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-16T19:31:20.158 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-16T19:31:20.158 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-03-16T19:31:20.160 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:20.161 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:20.177+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:20.260 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:20.279+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:20.269 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:20.269 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:20.270 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:20.270 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:20.279 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:20.314 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:20.314 INFO:teuthology.orchestra.run.smithi174.stdout: "success": "" 2024-03-16T19:31:20.314 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:20.367 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:20.386+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:20.420 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:20.421 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:20.421 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:20.421 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:20.421 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:20.429 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:20.461 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:20.480+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:20.562 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:20.581+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:20.576 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:20.576 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:20.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:20.576 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.239150709, 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:31:20.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:31:20.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:31:20.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:31:20.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.237923287, 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:20.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:20.585 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.237520581, 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:20.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:31:20.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:20.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.589 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.23691183, 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:20.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:20.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.236467961, 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.236325185, 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:20.598 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:31:20.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.236310644, 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:20.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.233432949, 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:20.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:31:20.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:31:20.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:20.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:31:20.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:20.611 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:20.613 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:20.663 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:20.682+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:20.742 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:20.742 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:20.742 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:20.742 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:20.751 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:20.763 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:20.783+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:20.864 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:20.883+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:20.900 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:20.901 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:20.901 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:20.901 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:20.901 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:20.909 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:20.993 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:21.012+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:21.055 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:21.055 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:21.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.71752924, 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:31:21.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:31:21.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:21.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:31:21.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:31:21.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:31:21.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:31:21.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.061 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.717450122, 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:31:21.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:31:21.065 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.71683599799999, 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:31:21.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:31:21.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.716395929, 2024-03-16T19:31:21.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 109.716154339, 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:31:21.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:31:21.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.078 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.079 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.079 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.079 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:21.080 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:21.080 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:21.099+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:21.180 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:21.199+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:21.227 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:21.227 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:21.227 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:21.227 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:21.235 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:21.281 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:21.300+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:21.382 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:21.401+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:21.383 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:21.383 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:21.383 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:21.383 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:21.383 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:21.392 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:21.483 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:21.502+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:21.500 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:21.500 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:21.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:21.500 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:21.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.500 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:21.500 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:21.500 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.161914614, 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:31:21.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.505 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.161890441, 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:21.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:31:21.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:31:21.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.161884614, 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.511 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:31:21.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.161880702, 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:31:21.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.16187472, 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.523 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:31:21.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:31:21.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:31:21.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:31:21.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:21.529 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.161871054, 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:21.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.161860394, 2024-03-16T19:31:21.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:31:21.539 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.541 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.541 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.541 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:21.541 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:21.541 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.161856505, 2024-03-16T19:31:21.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:21.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.545 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.158339814, 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:31:21.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:31:21.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:31:21.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.158269379, 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.550 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:31:21.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.157887032, 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:31:21.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.156120749, 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:31:21.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:31:21.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.563 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.154307827, 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:31:21.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 110.153452078, 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.567 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:31:21.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:31:21.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:31:21.572 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.147591312, 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:31:21.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:31:21.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.577 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 101.715040104, 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:21.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:31:21.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:31:21.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 100.147546398, 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.583 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:31:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 98.639934147000005, 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:31:21.588 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:31:21.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:31:21.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 62.143070922, 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.593 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:21.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:31:21.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:31:21.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 56.142535006000003, 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:21.597 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:31:21.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:31:21.601 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.602 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.602 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.602 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:21.602 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:21.602 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:21.603 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:21.604 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:21.622+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:21.704 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:21.723+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:21.750 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:21.750 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:21.751 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:21.751 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:21.759 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:21.843 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:21.862+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:21.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:31:21.905 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:21.906 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:21.906 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:21.906 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:21.906 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:21.906 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:21.925+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:21.914 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:21.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.975+0000 7fef2fdc7640 1 Processor -- start 2024-03-16T19:31:21.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.975+0000 7fef2fdc7640 1 -- start start 2024-03-16T19:31:21.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2fdc7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fef28154e50 0x7fef280cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:21.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2fdc7640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fef280caf40 0x7fef280cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:21.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2fdc7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2806b110 con 0x7fef280b5f10 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2fdc7640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2815b8d0 con 0x7fef280caf40 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2fdc7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2815ad50 con 0x7fef28154e50 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2d362640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fef280b5f10 0x7fef2806b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51134/0 (socket says 172.21.15.172:51134) 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2d362640 1 -- 172.21.15.172:0/3162326017 learned_addr learned my addr 172.21.15.172:0/3162326017 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2db63640 1 --2- 172.21.15.172:0/3162326017 >> v2:172.21.15.174:3300/0 conn(0x7fef28154e50 0x7fef280cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2e364640 1 --2- 172.21.15.172:0/3162326017 >> v2:172.21.15.172:3300/0 conn(0x7fef280caf40 0x7fef280cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2db63640 1 -- 172.21.15.172:0/3162326017 >> v2:172.21.15.172:3300/0 conn(0x7fef280caf40 msgr2=0x7fef280cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2db63640 1 --2- 172.21.15.172:0/3162326017 >> v2:172.21.15.172:3300/0 conn(0x7fef280caf40 0x7fef280cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2db63640 1 -- 172.21.15.172:0/3162326017 >> v1:172.21.15.172:6789/0 conn(0x7fef280b5f10 legacy=0x7fef2806b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:21.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.976+0000 7fef2db63640 1 -- 172.21.15.172:0/3162326017 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2815b030 con 0x7fef28154e50 2024-03-16T19:31:21.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.977+0000 7fef2db63640 1 --2- 172.21.15.172:0/3162326017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 0x7fef280cbc30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fef2808e370 tx=0x7fef2402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f829ff6eb9dd73c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.977+0000 7fef2cb61640 1 -- 172.21.15.172:0/3162326017 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24030610 con 0x7fef28154e50 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.977+0000 7fef2cb61640 1 -- 172.21.15.172:0/3162326017 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef24030770 con 0x7fef28154e50 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.977+0000 7fef2cb61640 1 -- 172.21.15.172:0/3162326017 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24030b00 con 0x7fef28154e50 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.978+0000 7fef2fdc7640 1 -- 172.21.15.172:0/3162326017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 msgr2=0x7fef280cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.978+0000 7fef2fdc7640 1 --2- 172.21.15.172:0/3162326017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 0x7fef280cbc30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fef2808e370 tx=0x7fef2402b960 comp rx=0 tx=0).stop 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.978+0000 7fef2fdc7640 1 -- 172.21.15.172:0/3162326017 shutdown_connections 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.978+0000 7fef2fdc7640 1 --2- 172.21.15.172:0/3162326017 >> v2:172.21.15.172:3300/0 conn(0x7fef280caf40 0x7fef280cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.978+0000 7fef2fdc7640 1 --2- 172.21.15.172:0/3162326017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 0x7fef280cbc30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.978+0000 7fef2fdc7640 1 -- 172.21.15.172:0/3162326017 >> 172.21.15.172:0/3162326017 conn(0x7fef28066c20 msgr2=0x7fef28067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.978+0000 7fef2fdc7640 1 -- 172.21.15.172:0/3162326017 shutdown_connections 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.978+0000 7fef2fdc7640 1 -- 172.21.15.172:0/3162326017 wait complete. 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.979+0000 7fef2fdc7640 1 Processor -- start 2024-03-16T19:31:21.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.979+0000 7fef2fdc7640 1 -- start start 2024-03-16T19:31:21.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.980+0000 7fef2fdc7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef280b5f10 0x7fef281f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:21.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.980+0000 7fef2fdc7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef280caf40 0x7fef28156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.980+0000 7fef2fdc7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 0x7fef28159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2fdc7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef281f1430 con 0x7fef280caf40 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2fdc7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef28155440 con 0x7fef28154e50 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2fdc7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef28156950 con 0x7fef280b5f10 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2d362640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef280caf40 0x7fef28156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2db63640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef280b5f10 0x7fef281f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2db63640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef280b5f10 0x7fef281f0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:50996/0 (socket says 172.21.15.172:50996) 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2d362640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef280caf40 0x7fef28156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:44896/0 (socket says 172.21.15.172:44896) 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2db63640 1 -- 172.21.15.172:0/234462882 learned_addr learned my addr 172.21.15.172:0/234462882 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2e364640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 0x7fef28159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2d362640 1 -- 172.21.15.172:0/234462882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef280b5f10 msgr2=0x7fef281f0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2d362640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef280b5f10 0x7fef281f0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2d362640 1 -- 172.21.15.172:0/234462882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 msgr2=0x7fef28159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:21.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2d362640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 0x7fef28159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2d362640 1 -- 172.21.15.172:0/234462882 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24009400 con 0x7fef280caf40 2024-03-16T19:31:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2db63640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef280b5f10 0x7fef281f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2e364640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 0x7fef28159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef2d362640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef280caf40 0x7fef28156ef0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fef18000f60 tx=0x7fef1800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.981+0000 7fef1e7fc640 1 -- 172.21.15.172:0/234462882 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18005ce0 con 0x7fef280caf40 2024-03-16T19:31:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.982+0000 7fef1e7fc640 1 -- 172.21.15.172:0/234462882 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef18005e40 con 0x7fef280caf40 2024-03-16T19:31:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.982+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2815ab20 con 0x7fef280caf40 2024-03-16T19:31:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.982+0000 7fef1e7fc640 1 -- 172.21.15.172:0/234462882 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18007310 con 0x7fef280caf40 2024-03-16T19:31:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.982+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef281582b0 con 0x7fef280caf40 2024-03-16T19:31:21.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.983+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef280c9ab0 con 0x7fef280caf40 2024-03-16T19:31:21.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.983+0000 7fef1e7fc640 1 -- 172.21.15.172:0/234462882 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fef18007470 con 0x7fef280caf40 2024-03-16T19:31:21.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.984+0000 7fef1e7fc640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fef0c06b3b0 0x7fef0c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:21.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.984+0000 7fef2db63640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fef0c06b3b0 0x7fef0c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:21.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.984+0000 7fef1e7fc640 1 -- 172.21.15.172:0/234462882 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6002+0+0 (secure 0 0 0) 0x7fef18089280 con 0x7fef280caf40 2024-03-16T19:31:21.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.984+0000 7fef2db63640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fef0c06b3b0 0x7fef0c06d870 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fef2402b9a0 tx=0x7fef2403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:21.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:21.988+0000 7fef1e7fc640 1 -- 172.21.15.172:0/234462882 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef18053dd0 con 0x7fef280caf40 2024-03-16T19:31:21.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:31:22.006 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:22.026+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:22.063 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 125.852918974, 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 125.47622342699999, 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.700246983, 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.70016257, 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.70015521400001, 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.50761602199999, 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:31:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:31:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.507610215, 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:31:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:31:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.507492082, 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:31:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:31:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:31:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:31:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:31:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.50739688500001, 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:31:22.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:31:22.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.114 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.507392874, 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:31:22.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:31:22.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.088461, 2024-03-16T19:31:22.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:31:22.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:31:22.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:31:22.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.088411631, 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.126 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:31:22.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.131 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.088332799, 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:31:22.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:31:22.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:31:22.136 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.08832958799999, 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.137 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.088326635, 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.142 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.147 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.088211661, 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:31:22.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.088208309, 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:31:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:22.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:31:22.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:31:22.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.08820519299999, 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:31:22.158 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:31:22.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:31:22.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 124.088144724, 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.164 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.169 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.479464674, 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:22.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:31:22.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:31:22.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:31:22.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:31:22.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:31:22.175 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.175 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.175 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.175 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:22.175 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:22.175 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:22.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:22.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.112+0000 7fa3ce076640 1 Processor -- start 2024-03-16T19:31:22.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.112+0000 7fa3ce076640 1 -- start start 2024-03-16T19:31:22.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.113+0000 7fa3ce076640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa3c8154ed0 0x7fa3c815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.113+0000 7fa3ce076640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa3c80b4700 0x7fa3c80b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.113+0000 7fa3ce076640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c80b4270 con 0x7fa3c80b3630 2024-03-16T19:31:22.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.113+0000 7fa3ce076640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c80b5050 con 0x7fa3c8154ed0 2024-03-16T19:31:22.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.113+0000 7fa3ce076640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c81552b0 con 0x7fa3c80b4700 2024-03-16T19:31:22.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.113+0000 7fa3c7fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa3c80b4700 0x7fa3c80b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.113+0000 7fa3c7fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa3c80b4700 0x7fa3c80b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42294/0 (socket says 172.21.15.172:42294) 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.113+0000 7fa3c7fff640 1 -- 172.21.15.172:0/2512593337 learned_addr learned my addr 172.21.15.172:0/2512593337 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.113+0000 7fa3c77fe640 1 --2- 172.21.15.172:0/2512593337 >> v2:172.21.15.172:3300/0 conn(0x7fa3c8154ed0 0x7fa3c815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.114+0000 7fa3c7fff640 1 -- 172.21.15.172:0/2512593337 >> v2:172.21.15.172:3300/0 conn(0x7fa3c8154ed0 msgr2=0x7fa3c815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.114+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2512593337 >> v2:172.21.15.172:3300/0 conn(0x7fa3c8154ed0 0x7fa3c815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.114+0000 7fa3c7fff640 1 -- 172.21.15.172:0/2512593337 >> v1:172.21.15.172:6789/0 conn(0x7fa3c80b3630 legacy=0x7fa3c80cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.114+0000 7fa3c7fff640 1 -- 172.21.15.172:0/2512593337 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c815b4b0 con 0x7fa3c80b4700 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.114+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2512593337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80b4700 0x7fa3c80b4b10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa3c815f1d0 tx=0x7fa3c002bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed321e2935ef784a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.115+0000 7fa3c67fc640 1 -- 172.21.15.172:0/2512593337 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c002cb10 con 0x7fa3c80b4700 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.115+0000 7fa3c67fc640 1 -- 172.21.15.172:0/2512593337 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c002cc70 con 0x7fa3c80b4700 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.115+0000 7fa3c67fc640 1 -- 172.21.15.172:0/2512593337 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00343f0 con 0x7fa3c80b4700 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.115+0000 7fa3ce076640 1 -- 172.21.15.172:0/2512593337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80b4700 msgr2=0x7fa3c80b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.115+0000 7fa3ce076640 1 --2- 172.21.15.172:0/2512593337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80b4700 0x7fa3c80b4b10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa3c815f1d0 tx=0x7fa3c002bf70 comp rx=0 tx=0).stop 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.115+0000 7fa3ce076640 1 -- 172.21.15.172:0/2512593337 shutdown_connections 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.115+0000 7fa3ce076640 1 --2- 172.21.15.172:0/2512593337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80b4700 0x7fa3c80b4b10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:22.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.115+0000 7fa3ce076640 1 --2- 172.21.15.172:0/2512593337 >> v2:172.21.15.172:3300/0 conn(0x7fa3c8154ed0 0x7fa3c815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.115+0000 7fa3ce076640 1 -- 172.21.15.172:0/2512593337 >> 172.21.15.172:0/2512593337 conn(0x7fa3c806bf60 msgr2=0x7fa3c806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.118+0000 7fa3ce076640 1 -- 172.21.15.172:0/2512593337 shutdown_connections 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.118+0000 7fa3ce076640 1 -- 172.21.15.172:0/2512593337 wait complete. 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.119+0000 7fa3ce076640 1 Processor -- start 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.119+0000 7fa3ce076640 1 -- start start 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3ce076640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c80b3630 0x7fa3c8148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3ce076640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c80b4700 0x7fa3c815e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3ce076640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c8154ed0 0x7fa3c80cff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3ce076640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80d1100 con 0x7fa3c80b4700 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3ce076640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c81490b0 con 0x7fa3c8154ed0 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3ce076640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80cf4d0 con 0x7fa3c80b3630 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c6ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c80b4700 0x7fa3c815e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c6ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c80b4700 0x7fa3c815e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:44918/0 (socket says 172.21.15.172:44918) 2024-03-16T19:31:22.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c6ffd640 1 -- 172.21.15.172:0/2218938807 learned_addr learned my addr 172.21.15.172:0/2218938807 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c8154ed0 0x7fa3c80cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c77fe640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c80b3630 0x7fa3c8148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c7fff640 1 -- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c80b3630 msgr2=0x7fa3c8148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c80b3630 0x7fa3c8148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c7fff640 1 -- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c80b4700 msgr2=0x7fa3c815e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa3c80b4700 0x7fa3c815e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.120+0000 7fa3c7fff640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c0009b30 con 0x7fa3c8154ed0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.121+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c8154ed0 0x7fa3c80cff20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0032000 tx=0x7fa3c0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.121+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0009510 con 0x7fa3c8154ed0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.121+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c003e450 con 0x7fa3c8154ed0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.121+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c003e7e0 con 0x7fa3c8154ed0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.121+0000 7fa3ce076640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c80ca900 con 0x7fa3c8154ed0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.121+0000 7fa3ce076640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c80c9da0 con 0x7fa3c8154ed0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.122+0000 7fa3ce076640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa3c8068d20 con 0x7fa3c8154ed0 2024-03-16T19:31:22.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.123+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa3c003e940 con 0x7fa3c8154ed0 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.123+0000 7fa3affff640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa3b406b3b0 0x7fa3b406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.123+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6002+0+0 (secure 0 0 0) 0x7fa3c00ac7c0 con 0x7fa3c8154ed0 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.123+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v72) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa3c00b3070 con 0x7fa3c8154ed0 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.124+0000 7fa3c77fe640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa3b406b3b0 0x7fa3b406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.124+0000 7fa3adffb640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fa3c80cf610 0x7fa3c81f8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.124+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c815ba70 con 0x7fa3c80cf610 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.124+0000 7fa3c6ffd640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fa3c80cf610 0x7fa3c81f8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.124+0000 7fa3c77fe640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa3b406b3b0 0x7fa3b406d870 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8009b40 tx=0x7fa3b8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.124+0000 7fa3c6ffd640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fa3c80cf610 0x7fa3c81f8620 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.125+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3bc001440 con 0x7fa3c80cf610 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.140+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3c80681c0 con 0x7fa3c80cf610 2024-03-16T19:31:22.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.143+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3c80681c0 con 0x7fa3c80cf610 2024-03-16T19:31:22.187 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:22.188 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:22.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.143+0000 7fa3adffb640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fa3c80cf9f0 0x7fa3c81f9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.143+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c815ae70 con 0x7fa3c80cf9f0 2024-03-16T19:31:22.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.144+0000 7fa3c77fe640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fa3c80cf9f0 0x7fa3c81f9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.144+0000 7fa3c77fe640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fa3c80cf9f0 0x7fa3c81f9070 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.145+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3c815ae70 con 0x7fa3c80cf9f0 2024-03-16T19:31:22.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.160+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3c80cbbe0 con 0x7fa3c80cf9f0 2024-03-16T19:31:22.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.163+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3c80cbbe0 con 0x7fa3c80cf9f0 2024-03-16T19:31:22.188 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.163+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=73}) v3 -- 0x7fa3c8159680 con 0x7fa3c8154ed0 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.163+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa3c81593a0 con 0x7fa3c8154ed0 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.163+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3c0077430 con 0x7fa3c8154ed0 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.164+0000 7fa3adffb640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fa3c80c7ec0 0x7fa3c80c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.164+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c815bf10 con 0x7fa3c80c7ec0 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.164+0000 7fa3c6ffd640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fa3c80c7ec0 0x7fa3c80c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.164+0000 7fa3c6ffd640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fa3c80c7ec0 0x7fa3c80c82f0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.165+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3c815bf10 con 0x7fa3c80c7ec0 2024-03-16T19:31:22.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.179+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3c815c3e0 con 0x7fa3c80c7ec0 2024-03-16T19:31:22.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.181+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3c815c3e0 con 0x7fa3c80c7ec0 2024-03-16T19:31:22.191 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:22.192 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:22.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.181+0000 7fa3adffb640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fa3c8061930 0x7fa3c8061db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.181+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c815ce20 con 0x7fa3c8061930 2024-03-16T19:31:22.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.181+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fa3c8061930 0x7fa3c8061db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.182+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fa3c8061930 0x7fa3c8061db0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.182+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3c815ce20 con 0x7fa3c8061930 2024-03-16T19:31:22.192 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:22.195+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:22.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.197+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3c80625c0 con 0x7fa3c8061930 2024-03-16T19:31:22.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.198+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3c80625c0 con 0x7fa3c8061930 2024-03-16T19:31:22.193 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:22.194 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:22.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.199+0000 7fa3adffb640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fa3c80626b0 0x7fa3c8209e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.199+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c820cc10 con 0x7fa3c80626b0 2024-03-16T19:31:22.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.199+0000 7fa3c77fe640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fa3c80626b0 0x7fa3c8209e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.199+0000 7fa3c77fe640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fa3c80626b0 0x7fa3c8209e40 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.200+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3c820cc10 con 0x7fa3c80626b0 2024-03-16T19:31:22.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.216+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3c820a380 con 0x7fa3c80626b0 2024-03-16T19:31:22.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.218+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3c820a380 con 0x7fa3c80626b0 2024-03-16T19:31:22.200 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:22.200 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:22.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.219+0000 7fa3adffb640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fa3c820aa00 0x7fa3c820cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.219+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c820fda0 con 0x7fa3c820aa00 2024-03-16T19:31:22.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.219+0000 7fa3c6ffd640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fa3c820aa00 0x7fa3c820cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.219+0000 7fa3c6ffd640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fa3c820aa00 0x7fa3c820cdf0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.220+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3c820fda0 con 0x7fa3c820aa00 2024-03-16T19:31:22.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.249+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3c80623d0 con 0x7fa3c820aa00 2024-03-16T19:31:22.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.251+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3c80623d0 con 0x7fa3c820aa00 2024-03-16T19:31:22.232 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:22.232 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:22.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.251+0000 7fa3adffb640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fa3c820f1f0 0x7fa3c820f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:22.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.251+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3c8213130 con 0x7fa3c820f1f0 2024-03-16T19:31:22.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.257+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fa3c820f1f0 0x7fa3c820f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:22.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.257+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fef28075a70 con 0x7fef280caf40 2024-03-16T19:31:22.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.263+0000 7fa3c7fff640 1 --2- 172.21.15.172:0/2218938807 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fa3c820f1f0 0x7fa3c820f650 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:22.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.265+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa3c8213130 con 0x7fa3c820f1f0 2024-03-16T19:31:22.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.280+0000 7fa3adffb640 1 -- 172.21.15.172:0/2218938807 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3c8211fb0 con 0x7fa3c820f1f0 2024-03-16T19:31:22.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.281+0000 7fa3affff640 1 -- 172.21.15.172:0/2218938807 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3c8211fb0 con 0x7fa3c820f1f0 2024-03-16T19:31:22.263 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:22.263 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:22.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:22.277 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:22.296+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:22.364 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:22.364 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:22.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:22.377 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:22.396+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:22.478 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:22.498+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:22.485 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:22.485 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:22.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:22.579 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:22.599+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:22.644 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:22.644 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:22.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:22.680 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:22.700+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:22.781 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:22.800+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:22.804 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:22.804 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:22.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:22.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.850+0000 7fef1e7fc640 1 -- 172.21.15.172:0/234462882 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v73) v1 ==== 97+0+0 (secure 0 0 0) 0x7fef18017070 con 0x7fef280caf40 2024-03-16T19:31:22.831 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:31:22.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.853+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fef0c06b3b0 msgr2=0x7fef0c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:22.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.853+0000 7fef2fdc7640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fef0c06b3b0 0x7fef0c06d870 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fef2402b9a0 tx=0x7fef2403b040 comp rx=0 tx=0).stop 2024-03-16T19:31:22.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.853+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef280caf40 msgr2=0x7fef28156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:22.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.853+0000 7fef2fdc7640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef280caf40 0x7fef28156ef0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fef18000f60 tx=0x7fef1800c800 comp rx=0 tx=0).stop 2024-03-16T19:31:22.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.854+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 shutdown_connections 2024-03-16T19:31:22.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.854+0000 7fef2fdc7640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef28154e50 0x7fef28159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:22.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.854+0000 7fef2fdc7640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fef280caf40 0x7fef28156ef0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:22.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.854+0000 7fef2fdc7640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fef0c06b3b0 0x7fef0c06d870 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:22.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.854+0000 7fef2fdc7640 1 --2- 172.21.15.172:0/234462882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef280b5f10 0x7fef281f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:22.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.854+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 >> 172.21.15.172:0/234462882 conn(0x7fef28066c20 msgr2=0x7fef280ce5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:22.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.854+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 shutdown_connections 2024-03-16T19:31:22.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:22.854+0000 7fef2fdc7640 1 -- 172.21.15.172:0/234462882 wait complete. 2024-03-16T19:31:22.888 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:22.907+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:22.924 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:22.925 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:22.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:22.982 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:23.001+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:23.083 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:23.103+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:23.086 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:23.086 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:23.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:23.184 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:23.203+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:23.204 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:23.204 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:23.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:23.284 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:23.304+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:23.385 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:23.404+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:23.425 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:23.426 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:23.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:23.487 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:23.506+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:23.550 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:23.550 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:23.560 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:23.587 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:23.606+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:23.667 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:23.667 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:23.676 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:23.721 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:23.740+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:23.782 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:23.782 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:23.790 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:23.809+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:23.791 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:23.891 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:23.910+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:23.937 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:23.937 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:23.946 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:23.991 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:24.010+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:24.091 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:24.111+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:24.092 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:24.093 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:24.101 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:24.192 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:24.211+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:24.247 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:24.248 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:24.256 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:24.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:31:24.300 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:24.319+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:24.389 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:24.389 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:24.394 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:24.413+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:24.399 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:24.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.435+0000 7f3f165b5640 1 Processor -- start 2024-03-16T19:31:24.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.435+0000 7f3f165b5640 1 -- start start 2024-03-16T19:31:24.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.435+0000 7f3f165b5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 0x7f3f100cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.436+0000 7f3f165b5640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f100cc190 con 0x7f3f10154e60 2024-03-16T19:31:24.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.436+0000 7f3f165b5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f100caaa0 con 0x7f3f100cc3f0 2024-03-16T19:31:24.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.436+0000 7f3f165b5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f100c9de0 con 0x7f3f100c7520 2024-03-16T19:31:24.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.436+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 0x7f3f100cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.436+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 0x7f3f100cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:51004/0 (socket says 172.21.15.172:51004) 2024-03-16T19:31:24.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.436+0000 7f3f0f7fe640 1 -- 172.21.15.172:0/1159095598 learned_addr learned my addr 172.21.15.172:0/1159095598 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:24.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.436+0000 7f3f0f7fe640 1 -- 172.21.15.172:0/1159095598 >> v1:172.21.15.174:6789/0 conn(0x7f3f100cc3f0 legacy=0x7f3f1006afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:24.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.436+0000 7f3f0f7fe640 1 -- 172.21.15.172:0/1159095598 >> v1:172.21.15.172:6789/0 conn(0x7f3f10154e60 legacy=0x7f3f1015aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:24.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.436+0000 7f3f0f7fe640 1 -- 172.21.15.172:0/1159095598 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f100cba30 con 0x7f3f100c7520 2024-03-16T19:31:24.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.437+0000 7f3f0f7fe640 1 --2- 172.21.15.172:0/1159095598 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 0x7f3f100cb4f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3f040098a0 tx=0x7f3f0402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1e5dc094c0ddf0a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.437+0000 7f3f0effd640 1 -- 172.21.15.172:0/1159095598 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0402f830 con 0x7f3f100c7520 2024-03-16T19:31:24.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.437+0000 7f3f0effd640 1 -- 172.21.15.172:0/1159095598 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0402f990 con 0x7f3f100c7520 2024-03-16T19:31:24.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.437+0000 7f3f0effd640 1 -- 172.21.15.172:0/1159095598 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0402fd20 con 0x7f3f100c7520 2024-03-16T19:31:24.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.438+0000 7f3f165b5640 1 -- 172.21.15.172:0/1159095598 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 msgr2=0x7f3f100cb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:24.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.438+0000 7f3f165b5640 1 --2- 172.21.15.172:0/1159095598 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 0x7f3f100cb4f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3f040098a0 tx=0x7f3f0402ba10 comp rx=0 tx=0).stop 2024-03-16T19:31:24.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.438+0000 7f3f165b5640 1 -- 172.21.15.172:0/1159095598 shutdown_connections 2024-03-16T19:31:24.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.438+0000 7f3f165b5640 1 --2- 172.21.15.172:0/1159095598 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 0x7f3f100cb4f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:24.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.438+0000 7f3f165b5640 1 -- 172.21.15.172:0/1159095598 >> 172.21.15.172:0/1159095598 conn(0x7f3f100d0300 msgr2=0x7f3f100d0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:24.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.438+0000 7f3f165b5640 1 -- 172.21.15.172:0/1159095598 shutdown_connections 2024-03-16T19:31:24.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.438+0000 7f3f165b5640 1 -- 172.21.15.172:0/1159095598 wait complete. 2024-03-16T19:31:24.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.439+0000 7f3f165b5640 1 Processor -- start 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.439+0000 7f3f165b5640 1 -- start start 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f165b5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 0x7f3f10159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f165b5640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f100cc3f0 0x7f3f100c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f165b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f10154e60 0x7f3f100c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f165b5640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f100cf6a0 con 0x7f3f100cc3f0 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f165b5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f101f0ed0 con 0x7f3f10154e60 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f165b5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1014c4d0 con 0x7f3f100c7520 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f100cc3f0 0x7f3f100c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f100cc3f0 0x7f3f100c5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:44926/0 (socket says 172.21.15.172:44926) 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f0ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 0x7f3f10159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f0f7fe640 1 -- 172.21.15.172:0/580731275 learned_addr learned my addr 172.21.15.172:0/580731275 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:24.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.440+0000 7f3f14b52640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f10154e60 0x7f3f100c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0ffff640 1 -- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f10154e60 msgr2=0x7f3f100c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0ffff640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f10154e60 0x7f3f100c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0ffff640 1 -- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f100cc3f0 msgr2=0x7f3f100c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0ffff640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f100cc3f0 0x7f3f100c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0ffff640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f040094d0 con 0x7f3f100c7520 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0f7fe640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f100cc3f0 0x7f3f100c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0ffff640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f100c7520 0x7f3f10159c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000c3c0 tx=0x7f3f0000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f000052d0 con 0x7f3f100c7520 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f00005450 con 0x7f3f100c7520 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f00007260 con 0x7f3f100c7520 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f165b5640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f100c4de0 con 0x7f3f100c7520 2024-03-16T19:31:24.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.441+0000 7f3f165b5640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f100c35d0 con 0x7f3f100c7520 2024-03-16T19:31:24.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.442+0000 7f3f165b5640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3f100681c0 con 0x7f3f100c7520 2024-03-16T19:31:24.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.443+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3f000073c0 con 0x7f3f100c7520 2024-03-16T19:31:24.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.444+0000 7f3f0cff9640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3efc06b3b0 0x7f3efc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.444+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f3f00088ef0 con 0x7f3f100c7520 2024-03-16T19:31:24.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.444+0000 7f3f0f7fe640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3efc06b3b0 0x7f3efc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.444+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v73) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3f0008b530 con 0x7f3f100c7520 2024-03-16T19:31:24.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.445+0000 7f3ef67fc640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3ee8001650 0x7f3ee8003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.445+0000 7f3f0f7fe640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3efc06b3b0 0x7f3efc06d870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3f100ce7c0 tx=0x7f3f04008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.445+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ee8006bb0 con 0x7f3ee8001650 2024-03-16T19:31:24.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.446+0000 7f3f14b52640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3ee8001650 0x7f3ee8003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.446+0000 7f3f14b52640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f3ee8001650 0x7f3ee8003b10 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.447+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3ee8006bb0 con 0x7f3ee8001650 2024-03-16T19:31:24.443 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.462+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ee8005c60 con 0x7f3ee8001650 2024-03-16T19:31:24.445 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.464+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3ee8005c60 con 0x7f3ee8001650 2024-03-16T19:31:24.446 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:24.446 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:24.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.464+0000 7f3ef67fc640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3ee8004190 0x7f3ee8004640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.446 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.464+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ee8004c00 con 0x7f3ee8004190 2024-03-16T19:31:24.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.465+0000 7f3f0ffff640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3ee8004190 0x7f3ee8004640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.465+0000 7f3f0ffff640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3ee8004190 0x7f3ee8004640 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.447 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.466+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3ee8004c00 con 0x7f3ee8004190 2024-03-16T19:31:24.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.481+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ee80050d0 con 0x7f3ee8004190 2024-03-16T19:31:24.464 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.483+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3ee80050d0 con 0x7f3ee8004190 2024-03-16T19:31:24.464 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:24.465 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:24.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.483+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=74}) v3 -- 0x7f3ee8005680 con 0x7f3f100c7520 2024-03-16T19:31:24.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.483+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ee800b6b0 con 0x7f3f100c7520 2024-03-16T19:31:24.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.483+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f00057830 con 0x7f3f100c7520 2024-03-16T19:31:24.465 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:24.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.483+0000 7f3ef67fc640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3ee8009690 0x7f3ee8009ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.483+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ee800b470 con 0x7f3ee8009690 2024-03-16T19:31:24.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.483+0000 7f3f0f7fe640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3ee8009690 0x7f3ee8009ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.484+0000 7f3f0f7fe640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3ee8009690 0x7f3ee8009ae0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.485+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3ee800b470 con 0x7f3ee8009690 2024-03-16T19:31:24.480 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.499+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ee800a4c0 con 0x7f3ee8009690 2024-03-16T19:31:24.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.501+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3ee800a4c0 con 0x7f3ee8009690 2024-03-16T19:31:24.483 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:24.483 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:24.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.502+0000 7f3ef67fc640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3ee800d910 0x7f3ee800dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.502+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ee800b190 con 0x7f3ee800d910 2024-03-16T19:31:24.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.502+0000 7f3f14b52640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3ee800d910 0x7f3ee800dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.502+0000 7f3f14b52640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f3ee800d910 0x7f3ee800dd90 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.503+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3ee800b190 con 0x7f3ee800d910 2024-03-16T19:31:24.494 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:24.513+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:24.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.517+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ee800e600 con 0x7f3ee800d910 2024-03-16T19:31:24.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.519+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3ee800e600 con 0x7f3ee800d910 2024-03-16T19:31:24.500 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:24.501 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:24.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.519+0000 7f3ef67fc640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3ee8011060 0x7f3ee8013450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.519+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ee8016220 con 0x7f3ee8011060 2024-03-16T19:31:24.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.519+0000 7f3f0ffff640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3ee8011060 0x7f3ee8013450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.520+0000 7f3f0ffff640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f3ee8011060 0x7f3ee8013450 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.520+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3ee8016220 con 0x7f3ee8011060 2024-03-16T19:31:24.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.535+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ee8013990 con 0x7f3ee8011060 2024-03-16T19:31:24.517 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.536+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3ee8013990 con 0x7f3ee8011060 2024-03-16T19:31:24.517 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:24.518 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:24.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.536+0000 7f3ef67fc640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3ee80141e0 0x7f3ee8014640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.536+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ee80195d0 con 0x7f3ee80141e0 2024-03-16T19:31:24.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.537+0000 7f3f0f7fe640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3ee80141e0 0x7f3ee8014640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.519 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.537+0000 7f3f0f7fe640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f3ee80141e0 0x7f3ee8014640 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.519 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.538+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3ee80195d0 con 0x7f3ee80141e0 2024-03-16T19:31:24.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.552+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ee80183d0 con 0x7f3ee80141e0 2024-03-16T19:31:24.536 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.555+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3ee80183d0 con 0x7f3ee80141e0 2024-03-16T19:31:24.536 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:24.537 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:24.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.555+0000 7f3ef67fc640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3ee8018a50 0x7f3ee8018ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.555+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ee80195f0 con 0x7f3ee8018a50 2024-03-16T19:31:24.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.556+0000 7f3f14b52640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3ee8018a50 0x7f3ee8018ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.537 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.556+0000 7f3f14b52640 1 --2- 172.21.15.172:0/580731275 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f3ee8018a50 0x7f3ee8018ed0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.538 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.557+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3f080193f0 con 0x7f3ee8018a50 2024-03-16T19:31:24.548 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:24.548 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:24.548 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:24.548 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:24.552 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.571+0000 7f3ef67fc640 1 -- 172.21.15.172:0/580731275 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ee801b7b0 con 0x7f3ee8018a50 2024-03-16T19:31:24.555 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.574+0000 7f3f0cff9640 1 -- 172.21.15.172:0/580731275 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3ee801b7b0 con 0x7f3ee8018a50 2024-03-16T19:31:24.555 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:24.555 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:24.557 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:24.559 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:24.595 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:24.614+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:24.696 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:24.715+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:24.704 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:24.704 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:24.704 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:24.704 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:24.704 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:24.712 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:24.797 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:24.816+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:24.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.525621474, 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:31:24.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:31:24.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:31:24.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.525597301, 2024-03-16T19:31:24.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:31:24.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:24.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:31:24.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.874 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.525591474, 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:24.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:24.879 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.525587562, 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:31:24.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.884 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.52558157999999, 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.887 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.889 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.525577914, 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.890 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:31:24.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:31:24.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.894 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.52556725399999, 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.525563365, 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.898 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.899 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:24 2.7 e21/18 ondisk, result = 0)", 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.52204667399999, 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000227703, 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.902 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360794+0000", 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360795+0000", 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-06 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.903 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360802+0000", 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8889999999999996e-06 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360815+0000", 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3441000000000001e-05 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360826+0000", 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1396000000000001e-05 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.904 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360889+0000", 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2184e-05 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360896+0000", 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5800000000000003e-06 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361021+0000", 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012520399999999999 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.905 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.521976239, 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.906 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:31:24.907 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:31:24.908 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.909 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.521593892, 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:24.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.912 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:1 2.14 e21/18 ondisk, result = 0)", 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.519827609, 2024-03-16T19:31:24.913 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026650300000000002, 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.914 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363013+0000", 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363018+0000", 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4089999999999996e-06 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363026+0000", 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9959999999999993e-06 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363031+0000", 2024-03-16T19:31:24.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3199999999999999e-06 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363175+0000", 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014376499999999999 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363183+0000", 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9629999999999998e-06 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363279+0000", 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6681999999999994e-05 2024-03-16T19:31:24.916 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing_reply(2.14 epoch 21/18 rep_tid 7 lcod 21'5)", 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.518014687, 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000259533, 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.917 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364826+0000", 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9999999999999999e-07 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364828+0000", 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3120000000000001e-06 2024-03-16T19:31:24.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364835+0000", 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9709999999999998e-06 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364845+0000", 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0061000000000001e-05 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364925+0000", 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9906000000000004e-05 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365085+0000", 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015998299999999999 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 113.51715893799999, 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.920 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.921 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.922 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.923 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 108.511298172, 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.925 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.927 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.928 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 105.078746964, 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.930 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:31:24.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:31:24.933 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 103.511253258, 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.934 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.936 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.938 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 102.003641007, 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:24.939 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:31:24.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:31:24.941 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:31:24.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.943 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 65.506777782, 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.944 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.947 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 59.506241866000003, 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.948 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:31:24.949 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:31:24.950 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.951 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.952 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.952 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:24.952 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:24.952 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:24.953 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:24.953 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:24.972+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:24.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.974+0000 7f3609f56640 1 Processor -- start 2024-03-16T19:31:24.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.974+0000 7f3609f56640 1 -- start start 2024-03-16T19:31:24.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.975+0000 7f3609f56640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f36040cbc30 0x7f36040cc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.975+0000 7f3609f56640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f36040caf40 0x7f36040cb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.975+0000 7f3609f56640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360415b8f0 con 0x7f360406b610 2024-03-16T19:31:24.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.975+0000 7f3609f56640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36040cc550 con 0x7f36040cbc30 2024-03-16T19:31:24.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.975+0000 7f3609f56640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36040cb8a0 con 0x7f36040caf40 2024-03-16T19:31:24.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.975+0000 7f36037fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f36040cbc30 0x7f36040cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.975+0000 7f3602ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f36040caf40 0x7f36040cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.975+0000 7f3602ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f36040caf40 0x7f36040cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42320/0 (socket says 172.21.15.172:42320) 2024-03-16T19:31:24.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.975+0000 7f3602ffd640 1 -- 172.21.15.172:0/2751387746 learned_addr learned my addr 172.21.15.172:0/2751387746 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.976+0000 7f3602ffd640 1 -- 172.21.15.172:0/2751387746 >> v2:172.21.15.172:3300/0 conn(0x7f36040cbc30 msgr2=0x7f36040cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.976+0000 7f3602ffd640 1 --2- 172.21.15.172:0/2751387746 >> v2:172.21.15.172:3300/0 conn(0x7f36040cbc30 0x7f36040cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.976+0000 7f3602ffd640 1 -- 172.21.15.172:0/2751387746 >> v1:172.21.15.172:6789/0 conn(0x7f360406b610 legacy=0x7f360415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.976+0000 7f3602ffd640 1 -- 172.21.15.172:0/2751387746 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360415ad70 con 0x7f36040caf40 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.976+0000 7f36027fc640 1 -- 172.21.15.172:0/2751387746 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851012527 0 0) 0x7f360415b8f0 con 0x7f360406b610 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.976+0000 7f36037fe640 1 --2- 172.21.15.172:0/2751387746 >> v2:172.21.15.172:3300/0 conn(0x7f36040cbc30 0x7f36040cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.976+0000 7f3602ffd640 1 --2- 172.21.15.172:0/2751387746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36040caf40 0x7f36040cb360 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f36040ce000 tx=0x7f35f802b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c194c299949823ed server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f36027fc640 1 -- 172.21.15.172:0/2751387746 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8030870 con 0x7f36040caf40 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f36027fc640 1 -- 172.21.15.172:0/2751387746 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f80309d0 con 0x7f36040caf40 2024-03-16T19:31:24.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f36027fc640 1 -- 172.21.15.172:0/2751387746 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8030d60 con 0x7f36040caf40 2024-03-16T19:31:24.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f3609f56640 1 -- 172.21.15.172:0/2751387746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36040caf40 msgr2=0x7f36040cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:24.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f3609f56640 1 --2- 172.21.15.172:0/2751387746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36040caf40 0x7f36040cb360 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f36040ce000 tx=0x7f35f802b9b0 comp rx=0 tx=0).stop 2024-03-16T19:31:24.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f3609f56640 1 -- 172.21.15.172:0/2751387746 shutdown_connections 2024-03-16T19:31:24.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f3609f56640 1 --2- 172.21.15.172:0/2751387746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36040caf40 0x7f36040cb360 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:24.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f3609f56640 1 --2- 172.21.15.172:0/2751387746 >> v2:172.21.15.172:3300/0 conn(0x7f36040cbc30 0x7f36040cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:24.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f3609f56640 1 -- 172.21.15.172:0/2751387746 >> 172.21.15.172:0/2751387746 conn(0x7f3604066c20 msgr2=0x7f3604067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:24.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f3609f56640 1 -- 172.21.15.172:0/2751387746 shutdown_connections 2024-03-16T19:31:24.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.977+0000 7f3609f56640 1 -- 172.21.15.172:0/2751387746 wait complete. 2024-03-16T19:31:24.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.978+0000 7f3609f56640 1 Processor -- start 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.979+0000 7f3609f56640 1 -- start start 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.979+0000 7f3609f56640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f360406b610 0x7f36041f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.979+0000 7f3609f56640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36040caf40 0x7f360415a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.979+0000 7f3609f56640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36040cbc30 0x7f360414e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f3609f56640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3604157710 con 0x7f360406b610 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f3609f56640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36041f4280 con 0x7f36040cbc30 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f3609f56640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36041f0260 con 0x7f36040caf40 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f36037fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f360406b610 0x7f36041f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f36037fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f360406b610 0x7f36041f16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:44944/0 (socket says 172.21.15.172:44944) 2024-03-16T19:31:24.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f36037fe640 1 -- 172.21.15.172:0/2073860205 learned_addr learned my addr 172.21.15.172:0/2073860205 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f3603fff640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36040cbc30 0x7f360414e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f3602ffd640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36040caf40 0x7f360415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f36037fe640 1 -- 172.21.15.172:0/2073860205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36040caf40 msgr2=0x7f360415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f36037fe640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36040caf40 0x7f360415a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f36037fe640 1 -- 172.21.15.172:0/2073860205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36040cbc30 msgr2=0x7f360414e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f36037fe640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36040cbc30 0x7f360414e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f36037fe640 1 -- 172.21.15.172:0/2073860205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f4009400 con 0x7f360406b610 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f36037fe640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f360406b610 0x7f36041f16d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f35f400e960 tx=0x7f35f400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f35ebfff640 1 -- 172.21.15.172:0/2073860205 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f400ca00 con 0x7f360406b610 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f35ebfff640 1 -- 172.21.15.172:0/2073860205 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f400cb60 con 0x7f360406b610 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f80094d0 con 0x7f360406b610 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f35ebfff640 1 -- 172.21.15.172:0/2073860205 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4013470 con 0x7f360406b610 2024-03-16T19:31:24.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.980+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36041f44f0 con 0x7f360406b610 2024-03-16T19:31:24.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.981+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3604154650 con 0x7f360406b610 2024-03-16T19:31:24.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.982+0000 7f35ebfff640 1 -- 172.21.15.172:0/2073860205 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f35f40135d0 con 0x7f360406b610 2024-03-16T19:31:24.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.982+0000 7f35ebfff640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f35f006b3b0 0x7f35f006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:24.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.983+0000 7f35ebfff640 1 -- 172.21.15.172:0/2073860205 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f35f4089fe0 con 0x7f360406b610 2024-03-16T19:31:24.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.983+0000 7f3602ffd640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f35f006b3b0 0x7f35f006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:24.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.983+0000 7f3602ffd640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f35f006b3b0 0x7f35f006d870 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f35f8036000 tx=0x7f35f8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:24.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:24.987+0000 7f35ebfff640 1 -- 172.21.15.172:0/2073860205 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35f4054ac0 con 0x7f360406b610 2024-03-16T19:31:25.053 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:25.072+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:25.100 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:25.101 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:25.109 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:25.154 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:25.174+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:25.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.178+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f3604152120 con 0x7f360406b610 2024-03-16T19:31:25.255 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:25.274+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:25.256 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:25.257 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:25.257 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:25.257 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:25.257 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:25.266 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:25.326 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-16T19:31:25.326 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:31:25.326 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-16T19:31:25.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-03-16T19:31:25.356 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:25.375+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:25.415 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:25.415 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:25.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.20467219899999, 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.416 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:31:25.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:31:25.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.82797665199999, 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.425 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.05200020800001, 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:31:25.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.05191579500001, 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:31:25.431 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:25.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:31:25.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:31:25.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:31:25.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.436 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.05190843899999, 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:31:25.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:31:25.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.441 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.859369247, 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.442 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.85936344, 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.447 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:31:25.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:31:25.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:31:25.452 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.85924530699999, 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.453 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:31:25.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.85915011, 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.458 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:31:25.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.463 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.859146099, 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.464 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:31:25.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:31:25.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.44021422500001, 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:31:25.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:31:25.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.440164856, 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.475 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:25.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:31:25.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:31:25.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:25.480 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.440086024, 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.440082813, 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.44007986, 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.491 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.439964886, 2024-03-16T19:31:25.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:31:25.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:31:25.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.43996153400001, 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:25.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:31:25.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:31:25.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:31:25.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:31:25.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.439958418, 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.507 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:31:25.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:31:25.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:31:25.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 127.439897949, 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:25.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:31:25.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:31:25.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 126.83121789899999, 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:25.518 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:25.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:31:25.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:31:25.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:31:25.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:25.523 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:25.525 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:25.543+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:25.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:25.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.451+0000 7f1f8c10d640 1 Processor -- start 2024-03-16T19:31:25.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.452+0000 7f1f8c10d640 1 -- start start 2024-03-16T19:31:25.529 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.452+0000 7f1f8c10d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f84151910 0x7f1f84072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.452+0000 7f1f8c10d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1f840c3930 0x7f1f840c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.452+0000 7f1f8c10d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f84072b10 con 0x7f1f840c3ea0 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.452+0000 7f1f8c10d640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f840c2330 con 0x7f1f84151910 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.452+0000 7f1f8c10d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f840c7da0 con 0x7f1f840c3930 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.453+0000 7f1f89ea9640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f84151910 0x7f1f84072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.453+0000 7f1f896a8640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1f840c3ea0 0x7f1f840ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51160/0 (socket says 172.21.15.172:51160) 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.453+0000 7f1f896a8640 1 -- 172.21.15.172:0/4175965873 learned_addr learned my addr 172.21.15.172:0/4175965873 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.453+0000 7f1f8a6aa640 1 --2- 172.21.15.172:0/4175965873 >> v2:172.21.15.174:3300/0 conn(0x7f1f840c3930 0x7f1f840c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.453+0000 7f1f89ea9640 1 -- 172.21.15.172:0/4175965873 >> v2:172.21.15.174:3300/0 conn(0x7f1f840c3930 msgr2=0x7f1f840c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.453+0000 7f1f89ea9640 1 --2- 172.21.15.172:0/4175965873 >> v2:172.21.15.174:3300/0 conn(0x7f1f840c3930 0x7f1f840c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.453+0000 7f1f89ea9640 1 -- 172.21.15.172:0/4175965873 >> v1:172.21.15.172:6789/0 conn(0x7f1f840c3ea0 legacy=0x7f1f840ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.453+0000 7f1f89ea9640 1 -- 172.21.15.172:0/4175965873 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84071b10 con 0x7f1f84151910 2024-03-16T19:31:25.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.454+0000 7f1f89ea9640 1 --2- 172.21.15.172:0/4175965873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f84151910 0x7f1f84072fa0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1f840b2b30 tx=0x7f1f8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb8225746ce7bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.454+0000 7f1f88ea7640 1 -- 172.21.15.172:0/4175965873 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80031830 con 0x7f1f84151910 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.454+0000 7f1f88ea7640 1 -- 172.21.15.172:0/4175965873 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f80031990 con 0x7f1f84151910 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.454+0000 7f1f88ea7640 1 -- 172.21.15.172:0/4175965873 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80031d20 con 0x7f1f84151910 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.455+0000 7f1f8c10d640 1 -- 172.21.15.172:0/4175965873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f84151910 msgr2=0x7f1f84072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.455+0000 7f1f8c10d640 1 --2- 172.21.15.172:0/4175965873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f84151910 0x7f1f84072fa0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1f840b2b30 tx=0x7f1f8002b8d0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.455+0000 7f1f8c10d640 1 -- 172.21.15.172:0/4175965873 shutdown_connections 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.455+0000 7f1f8c10d640 1 --2- 172.21.15.172:0/4175965873 >> v2:172.21.15.174:3300/0 conn(0x7f1f840c3930 0x7f1f840c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.455+0000 7f1f8c10d640 1 --2- 172.21.15.172:0/4175965873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f84151910 0x7f1f84072fa0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.455+0000 7f1f8c10d640 1 -- 172.21.15.172:0/4175965873 >> 172.21.15.172:0/4175965873 conn(0x7f1f8406db70 msgr2=0x7f1f8406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.455+0000 7f1f8c10d640 1 -- 172.21.15.172:0/4175965873 shutdown_connections 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.455+0000 7f1f8c10d640 1 -- 172.21.15.172:0/4175965873 wait complete. 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.456+0000 7f1f8c10d640 1 Processor -- start 2024-03-16T19:31:25.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.456+0000 7f1f8c10d640 1 -- start start 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f8c10d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f840c3930 0x7f1f84156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f8c10d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f840c3ea0 0x7f1f841580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f8c10d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f84151910 0x7f1f8414a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f8c10d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840c92e0 con 0x7f1f840c3930 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f8c10d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840c9000 con 0x7f1f840c3ea0 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f8c10d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8414af10 con 0x7f1f84151910 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f89ea9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f840c3930 0x7f1f84156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f89ea9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f840c3930 0x7f1f84156740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:44962/0 (socket says 172.21.15.172:44962) 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f89ea9640 1 -- 172.21.15.172:0/1458018264 learned_addr learned my addr 172.21.15.172:0/1458018264 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f896a8640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f840c3ea0 0x7f1f841580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f8a6aa640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f84151910 0x7f1f8414a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f89ea9640 1 -- 172.21.15.172:0/1458018264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f84151910 msgr2=0x7f1f8414a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:25.532 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f89ea9640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f84151910 0x7f1f8414a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f89ea9640 1 -- 172.21.15.172:0/1458018264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f840c3ea0 msgr2=0x7f1f841580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f89ea9640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f840c3ea0 0x7f1f841580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f89ea9640 1 -- 172.21.15.172:0/1458018264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f80009400 con 0x7f1f840c3930 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f896a8640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f840c3ea0 0x7f1f841580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f89ea9640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f840c3930 0x7f1f84156740 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009890 tx=0x7f1f800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f7a7fc640 1 -- 172.21.15.172:0/1458018264 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80040070 con 0x7f1f840c3930 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f7a7fc640 1 -- 172.21.15.172:0/1458018264 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8003b890 con 0x7f1f840c3930 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f7a7fc640 1 -- 172.21.15.172:0/1458018264 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8003bc20 con 0x7f1f840c3930 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8414b1f0 con 0x7f1f840c3930 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.457+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f841ed010 con 0x7f1f840c3930 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.458+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f84075a70 con 0x7f1f840c3930 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.459+0000 7f1f7a7fc640 1 -- 172.21.15.172:0/1458018264 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1f8004d400 con 0x7f1f840c3930 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.459+0000 7f1f7a7fc640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1f6806b3b0 0x7f1f6806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:25.533 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.459+0000 7f1f7a7fc640 1 -- 172.21.15.172:0/1458018264 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f1f800ad8e0 con 0x7f1f840c3930 2024-03-16T19:31:25.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.459+0000 7f1f896a8640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1f6806b3b0 0x7f1f6806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:25.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.460+0000 7f1f896a8640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1f6806b3b0 0x7f1f6806d870 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1f8414d690 tx=0x7f1f74007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:25.534 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.464+0000 7f1f7a7fc640 1 -- 172.21.15.172:0/1458018264 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f80078550 con 0x7f1f840c3930 2024-03-16T19:31:25.626 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:25.645+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:25.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.651+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd in", "ids": ["3"]} v 0) v1 -- 0x7f1f8414b760 con 0x7f1f840c3930 2024-03-16T19:31:25.632 INFO:tasks.ceph.mon.a.smithi172.stderr:2024-03-16T19:31:25.651+0000 7f2459211640 -1 mon.a@0(leader).osd e73 definitely_dead 0 2024-03-16T19:31:25.643 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:25.644 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:25.644 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:25.644 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:25.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:25.727 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:25.747+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:25.762 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:25.762 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:25.762 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:25.762 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:25.762 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:25.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:25.828 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:25.848+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:25.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.857+0000 7f35ebfff640 1 -- 172.21.15.172:0/2073860205 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v74) v1 ==== 91+0+0 (secure 0 0 0) 0x7f35f4058180 con 0x7f360406b610 2024-03-16T19:31:25.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.857+0000 7f1f7a7fc640 1 -- 172.21.15.172:0/1458018264 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["3"]}]=0 marked in osd.3. v74) v1 ==== 85+0+0 (secure 0 0 0) 0x7f1f80048080 con 0x7f1f840c3930 2024-03-16T19:31:25.839 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:31:25.839 INFO:teuthology.orchestra.run.smithi172.stderr:marked in osd.3. 2024-03-16T19:31:25.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f35f006b3b0 msgr2=0x7f35f006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:25.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f35f006b3b0 0x7f35f006d870 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f35f8036000 tx=0x7f35f8008030 comp rx=0 tx=0).stop 2024-03-16T19:31:25.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f360406b610 msgr2=0x7f36041f16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f360406b610 0x7f36041f16d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f35f400e960 tx=0x7f35f400ee20 comp rx=0 tx=0).stop 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 shutdown_connections 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36040cbc30 0x7f360414e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f35f006b3b0 0x7f35f006d870 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36040caf40 0x7f360415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 --2- 172.21.15.172:0/2073860205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f360406b610 0x7f36041f16d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.861+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 >> 172.21.15.172:0/2073860205 conn(0x7f3604066c20 msgr2=0x7f360414f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.862+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1f6806b3b0 msgr2=0x7f1f6806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.862+0000 7f1f8c10d640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1f6806b3b0 0x7f1f6806d870 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1f8414d690 tx=0x7f1f74007e80 comp rx=0 tx=0).stop 2024-03-16T19:31:25.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.862+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f840c3930 msgr2=0x7f1f84156740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:25.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.862+0000 7f1f8c10d640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f840c3930 0x7f1f84156740 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009890 tx=0x7f1f800046c0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.862+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 shutdown_connections 2024-03-16T19:31:25.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.863+0000 7f3609f56640 1 -- 172.21.15.172:0/2073860205 wait complete. 2024-03-16T19:31:25.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.865+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 shutdown_connections 2024-03-16T19:31:25.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.865+0000 7f1f8c10d640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1f6806b3b0 0x7f1f6806d870 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.865+0000 7f1f8c10d640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f84151910 0x7f1f8414a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.865+0000 7f1f8c10d640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f840c3ea0 0x7f1f841580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.865+0000 7f1f8c10d640 1 --2- 172.21.15.172:0/1458018264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f840c3930 0x7f1f84156740 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:25.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.865+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 >> 172.21.15.172:0/1458018264 conn(0x7f1f8406db70 msgr2=0x7f1f8414e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:25.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.868+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 shutdown_connections 2024-03-16T19:31:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:25.870+0000 7f1f8c10d640 1 -- 172.21.15.172:0/1458018264 wait complete. 2024-03-16T19:31:25.864 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 129.314405322, 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.889 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:31:25.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:31:25.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.893 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.524179151, 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:31:25.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.897 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.523418916, 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.52260496100001, 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.52196503600001, 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.906 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:31:25.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:31:25.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:25.910 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.521306939, 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:31:25.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.520774364, 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:31:25.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:31:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.46260246099999, 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:25.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:31:25.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:31:25.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.46200026700001, 2024-03-16T19:31:25.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.461357322, 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:25.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.46085336100001, 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.932 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:31:25.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:31:25.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:31:25.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.302707095, 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.936 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:31:25.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:31:25.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.30217695899999, 2024-03-16T19:31:25.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:31:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.5502077, 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:31:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.550183164, 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:31:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.55017863400001, 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:31:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.550168822, 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.961 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:25.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:31:25.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:31:25.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:31:25.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.55016496899999, 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.966 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:25.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:31:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:31:25.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.550161653, 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:31:25.971 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:25.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:31:25.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:31:25.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.54896607000001, 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:31:25.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.978 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:25.979 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:25.979 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:25.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:25.981 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:25.999+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:26.080 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:26.099+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:26.130 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:26.130 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:26.131 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:26.131 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:26.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:26.184 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:26.203+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:26.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:31:26.249 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:31:26.263 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:26.264 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:26.264 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:26.264 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:26.264 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:26.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:26.314 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:26.333+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:26.383 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:26.402+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:26.423 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.084899389, 2024-03-16T19:31:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.084882275, 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:31:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:31:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.08487499500001, 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:31:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:31:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:31:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.084861128, 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:26.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:26.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:31:26.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:31:26.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.084188293, 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.445 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.081890043, 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:31:26.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:31:26.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.081844942, 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:26.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:31:26.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:31:26.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.081816887, 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.457 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.081795473, 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:31:26.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:31:26.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:31:26.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.081773122, 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:26.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:31:26.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.08175272299999, 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:31:26.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.08164673100001, 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:31:26.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.07985502, 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:26.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:31:26.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.079807519, 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:31:26.480 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:26.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:31:26.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.483 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.079568268, 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:26.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:31:26.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:31:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 110.071070123, 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.492 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:39 2.6 e21/18 ondisk, result = 0)", 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 110.069730247, 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151209, 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372506+0000", 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3200000000000005e-07 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372510+0000", 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1439999999999999e-06 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372518+0000", 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9129999999999992e-06 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372523+0000", 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.933e-06 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372578+0000", 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4122000000000001e-05 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372584+0000", 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372657+0000", 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3114000000000005e-05 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.066739264999995, 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:31:26.496 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:31:26.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:31:26.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.066108462999999, 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:26.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:31:26.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:47 2.6 e40/18 ondisk, result = 0)", 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.064626054000001, 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017677, 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:26.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377610+0000", 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377611+0000", 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9599999999999998e-07 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377618+0000", 2024-03-16T19:31:26.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9990000000000002e-06 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377626+0000", 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4249999999999998e-06 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377632+0000", 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377707+0000", 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4506999999999997e-05 2024-03-16T19:31:26.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377714+0000", 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5819999999999998e-06 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377787+0000", 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3232000000000006e-05 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:26.509 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:26.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:26.511 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:26.530+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:26.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:31:26.638 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:26.657+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:26.661 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:26.661 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:26.661 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:26.661 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:26.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:26.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.715+0000 7f1566fef640 1 Processor -- start 2024-03-16T19:31:26.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f1566fef640 1 -- start start 2024-03-16T19:31:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f1566fef640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1560152ba0 0x7f1560159490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f1566fef640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f15600cb920 0x7f156006e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f1566fef640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15600b1140 con 0x7f156006edc0 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f1566fef640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15600caae0 con 0x7f1560152ba0 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f1566fef640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156015c000 con 0x7f15600cb920 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f1564d8b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1560152ba0 0x7f1560159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f1564d8b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1560152ba0 0x7f1560159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:44964/0 (socket says 172.21.15.172:44964) 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f156558c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f156006edc0 0x7f15600c91f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51174/0 (socket says 172.21.15.172:51174) 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.716+0000 7f1564d8b640 1 -- 172.21.15.172:0/1914802123 learned_addr learned my addr 172.21.15.172:0/1914802123 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.717+0000 7f155ffff640 1 --2- 172.21.15.172:0/1914802123 >> v2:172.21.15.174:3300/0 conn(0x7f15600cb920 0x7f156006e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.717+0000 7f1564d8b640 1 -- 172.21.15.172:0/1914802123 >> v2:172.21.15.174:3300/0 conn(0x7f15600cb920 msgr2=0x7f156006e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.717+0000 7f1564d8b640 1 --2- 172.21.15.172:0/1914802123 >> v2:172.21.15.174:3300/0 conn(0x7f15600cb920 0x7f156006e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.717+0000 7f1564d8b640 1 -- 172.21.15.172:0/1914802123 >> v1:172.21.15.172:6789/0 conn(0x7f156006edc0 legacy=0x7f15600c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:26.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.717+0000 7f1564d8b640 1 -- 172.21.15.172:0/1914802123 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156015c2c0 con 0x7f1560152ba0 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f1564d8b640 1 --2- 172.21.15.172:0/1914802123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1560152ba0 0x7f1560159490 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f156006f8b0 tx=0x7f155402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89df01f2f8784235 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f155f7fe640 1 -- 172.21.15.172:0/1914802123 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155402f850 con 0x7f1560152ba0 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f155f7fe640 1 -- 172.21.15.172:0/1914802123 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155402f9b0 con 0x7f1560152ba0 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f155f7fe640 1 -- 172.21.15.172:0/1914802123 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155402fd40 con 0x7f1560152ba0 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f1566fef640 1 -- 172.21.15.172:0/1914802123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1560152ba0 msgr2=0x7f1560159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f1566fef640 1 --2- 172.21.15.172:0/1914802123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1560152ba0 0x7f1560159490 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f156006f8b0 tx=0x7f155402ba30 comp rx=0 tx=0).stop 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f1566fef640 1 -- 172.21.15.172:0/1914802123 shutdown_connections 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f1566fef640 1 --2- 172.21.15.172:0/1914802123 >> v2:172.21.15.174:3300/0 conn(0x7f15600cb920 0x7f156006e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f1566fef640 1 --2- 172.21.15.172:0/1914802123 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1560152ba0 0x7f1560159490 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f1566fef640 1 -- 172.21.15.172:0/1914802123 >> 172.21.15.172:0/1914802123 conn(0x7f1560070880 msgr2=0x7f1560070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:26.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f1566fef640 1 -- 172.21.15.172:0/1914802123 shutdown_connections 2024-03-16T19:31:26.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.718+0000 7f1566fef640 1 -- 172.21.15.172:0/1914802123 wait complete. 2024-03-16T19:31:26.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.720+0000 7f1566fef640 1 Processor -- start 2024-03-16T19:31:26.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.720+0000 7f1566fef640 1 -- start start 2024-03-16T19:31:26.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.720+0000 7f1566fef640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156006edc0 0x7f156014d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.720+0000 7f1566fef640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f15600cb920 0x7f1560147dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.720+0000 7f1566fef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560152ba0 0x7f1560148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.720+0000 7f1566fef640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15601f20a0 con 0x7f156006edc0 2024-03-16T19:31:26.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.720+0000 7f1566fef640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156014b3e0 con 0x7f1560152ba0 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.720+0000 7f1566fef640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15600cddd0 con 0x7f15600cb920 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1564d8b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156006edc0 0x7f156014d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1564d8b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156006edc0 0x7f156014d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:44980/0 (socket says 172.21.15.172:44980) 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1564d8b640 1 -- 172.21.15.172:0/3817130932 learned_addr learned my addr 172.21.15.172:0/3817130932 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f156558c640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560152ba0 0x7f1560148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f155ffff640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f15600cb920 0x7f1560147dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1564d8b640 1 -- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f15600cb920 msgr2=0x7f1560147dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1564d8b640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f15600cb920 0x7f1560147dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1564d8b640 1 -- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560152ba0 msgr2=0x7f1560148310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1564d8b640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560152ba0 0x7f1560148310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1564d8b640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15540094d0 con 0x7f156006edc0 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f156558c640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560152ba0 0x7f1560148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1564d8b640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f156006edc0 0x7f156014d140 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f155402ba70 tx=0x7f1554004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554037c40 con 0x7f156006edc0 2024-03-16T19:31:26.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1554037dc0 con 0x7f156006edc0 2024-03-16T19:31:26.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554041640 con 0x7f156006edc0 2024-03-16T19:31:26.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15601f1460 con 0x7f156006edc0 2024-03-16T19:31:26.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.721+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15601f1700 con 0x7f156006edc0 2024-03-16T19:31:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.722+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f15600688d0 con 0x7f156006edc0 2024-03-16T19:31:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.723+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f15540417a0 con 0x7f156006edc0 2024-03-16T19:31:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.724+0000 7f155d7fa640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f154406b3b0 0x7f154406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.724+0000 7f155ffff640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f154406b3b0 0x7f154406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.724+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f155403c070 con 0x7f156006edc0 2024-03-16T19:31:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.724+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v74) v1 ==== 54+0+16 (secure 0 0 0) 0x7f15540b3070 con 0x7f156006edc0 2024-03-16T19:31:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.724+0000 7f155ffff640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f154406b3b0 0x7f154406d870 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f15600ceec0 tx=0x7f154c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.724+0000 7f1566fef640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f156014b6a0 0x7f15601f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.724+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15601f1e30 con 0x7f156014b6a0 2024-03-16T19:31:26.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.724+0000 7f156558c640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f156014b6a0 0x7f15601f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.725+0000 7f156558c640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f156014b6a0 0x7f15601f18f0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.726+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f15601f1e30 con 0x7f156014b6a0 2024-03-16T19:31:26.712 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:26.731+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:26.721 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.740+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f156015c880 con 0x7f156014b6a0 2024-03-16T19:31:26.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.742+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1558004e20 con 0x7f156014b6a0 2024-03-16T19:31:26.723 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:26.724 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:26.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.742+0000 7f1566fef640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f15601490f0 0x7f1560149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.742+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15600ca520 con 0x7f15601490f0 2024-03-16T19:31:26.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.743+0000 7f1564d8b640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f15601490f0 0x7f1560149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.743+0000 7f1564d8b640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f15601490f0 0x7f1560149500 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.744+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f15600ca520 con 0x7f15601490f0 2024-03-16T19:31:26.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.759+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f156015bba0 con 0x7f15601490f0 2024-03-16T19:31:26.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.760+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f156015bba0 con 0x7f15601490f0 2024-03-16T19:31:26.742 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.761+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=75}) v3 -- 0x7f156014ade0 con 0x7f156006edc0 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.761+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15600cc0d0 con 0x7f156006edc0 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.761+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7f155407abc0 con 0x7f156006edc0 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.761+0000 7f1566fef640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f15600ccb90 0x7f1560061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.761+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1560061e70 con 0x7f15600ccb90 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.761+0000 7f155ffff640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f15600ccb90 0x7f1560061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.762+0000 7f155ffff640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f15600ccb90 0x7f1560061930 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.743 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.762+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1560061e70 con 0x7f15600ccb90 2024-03-16T19:31:26.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.791+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1560062340 con 0x7f15600ccb90 2024-03-16T19:31:26.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.793+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1560062340 con 0x7f15600ccb90 2024-03-16T19:31:26.774 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:26.775 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:26.775 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.793+0000 7f1566fef640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f156015a2f0 0x7f156015a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.775 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.793+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1560062870 con 0x7f156015a2f0 2024-03-16T19:31:26.775 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.794+0000 7f156558c640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f156015a2f0 0x7f156015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.775 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.794+0000 7f156558c640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f156015a2f0 0x7f156015a730 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.795+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1560062870 con 0x7f156015a2f0 2024-03-16T19:31:26.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.795+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== mon.0 v2:172.21.15.172:3300/0 8 ==== osd_map(75..75 src has 1..75) v4 ==== 444+0+0 (secure 0 0 0) 0x7f155407b260 con 0x7f156006edc0 2024-03-16T19:31:26.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.820+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f156002bde0 con 0x7f156015a2f0 2024-03-16T19:31:26.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.822+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f156002bde0 con 0x7f156015a2f0 2024-03-16T19:31:26.803 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:26.804 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:26.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.822+0000 7f1566fef640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f156002c4c0 0x7f156002c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.822+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f156002ce40 con 0x7f156002c4c0 2024-03-16T19:31:26.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.823+0000 7f1564d8b640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f156002c4c0 0x7f156002c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.823+0000 7f1564d8b640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f156002c4c0 0x7f156002c900 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.824+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f156002ce40 con 0x7f156002c4c0 2024-03-16T19:31:26.813 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:26.832+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:26.829 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:26.829 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:26.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.853+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f156002ce40 con 0x7f156002c4c0 2024-03-16T19:31:26.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.855+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f156002ce40 con 0x7f156002c4c0 2024-03-16T19:31:26.836 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:26.837 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:26.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.855+0000 7f1566fef640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f15602027e0 0x7f1560204bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.855+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15602079a0 con 0x7f15602027e0 2024-03-16T19:31:26.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.856+0000 7f155ffff640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f15602027e0 0x7f1560204bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.856+0000 7f155ffff640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f15602027e0 0x7f1560204bd0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.857+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f15602079a0 con 0x7f15602027e0 2024-03-16T19:31:26.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:26.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.872+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1560205110 con 0x7f15602027e0 2024-03-16T19:31:26.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.874+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1560205110 con 0x7f15602027e0 2024-03-16T19:31:26.855 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:26.856 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:26.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.874+0000 7f1566fef640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f15602057f0 0x7f1560207be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:26.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.874+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f156020aa30 con 0x7f15602057f0 2024-03-16T19:31:26.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.875+0000 7f156558c640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f15602057f0 0x7f1560207be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:26.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.875+0000 7f156558c640 1 --2- 172.21.15.172:0/3817130932 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f15602057f0 0x7f1560207be0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:26.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.876+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f156020aa30 con 0x7f15602057f0 2024-03-16T19:31:26.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.900+0000 7f1566fef640 1 -- 172.21.15.172:0/3817130932 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f15602081a0 con 0x7f15602057f0 2024-03-16T19:31:26.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:26.902+0000 7f155d7fa640 1 -- 172.21.15.172:0/3817130932 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f15602081a0 con 0x7f15602057f0 2024-03-16T19:31:26.883 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:26.883 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:26.886 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:26.913 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:26.933+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:26.961 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:26.961 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:26.971 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:27.015 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:27.034+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:27.079 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:27.079 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:27.088 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:27.115 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:27.134+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:27.215 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:27.234+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:27.235 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:27.235 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:27.244 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:27.289 INFO:tasks.workunit.client.0.smithi172.stderr:Updating files: 93% (11487/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-16T19:31:27.315 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:27.335+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:27.350 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:27.350 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:27.359 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:27.417 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:27.436+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:27.507 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:27.507 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:27.516 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:27.517 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:27.536+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:27.618 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:27.637+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:27.623 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:27.623 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:27.631 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr:Note: switching to 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'. 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr:state without impacting any branches by switching back to a branch. 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr: git switch -c 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr:Or undo this operation with: 2024-03-16T19:31:27.646 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-03-16T19:31:27.647 INFO:tasks.workunit.client.0.smithi172.stderr: git switch - 2024-03-16T19:31:27.647 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-03-16T19:31:27.647 INFO:tasks.workunit.client.0.smithi172.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-16T19:31:27.647 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-03-16T19:31:27.647 INFO:tasks.workunit.client.0.smithi172.stderr:HEAD is now at aba3c1d240c mds/quiesce-db: keep the db thread alive until shutdown 2024-03-16T19:31:27.649 DEBUG:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-16T19:31:27.712 INFO:tasks.workunit.client.0.smithi172.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-16T19:31:27.715 INFO:tasks.workunit.client.0.smithi172.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T19:31:27.715 INFO:tasks.workunit.client.0.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-16T19:31:27.719 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:27.738+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:27.781 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:27.781 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:27.791 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:27.820 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:27.839+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:27.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:31:27.920 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:27.939+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:27.938 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:27.938 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:27.947 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:27.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.985+0000 7fc9a4363640 1 Processor -- start 2024-03-16T19:31:27.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.985+0000 7fc9a4363640 1 -- start start 2024-03-16T19:31:27.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a4363640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc99c06aed0 0x7fc99c06b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:27.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a4363640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c0c1eb0 con 0x7fc99c1580b0 2024-03-16T19:31:27.967 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a4363640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c0c1350 con 0x7fc99c06baf0 2024-03-16T19:31:27.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a4363640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99c0c1150 con 0x7fc99c06aed0 2024-03-16T19:31:27.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a20ff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc99c06aed0 0x7fc99c06b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:27.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a20ff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc99c06aed0 0x7fc99c06b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:44996/0 (socket says 172.21.15.172:44996) 2024-03-16T19:31:27.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a2900640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc99c1580b0 0x7fc99c161fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51186/0 (socket says 172.21.15.172:51186) 2024-03-16T19:31:27.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a20ff640 1 -- 172.21.15.172:0/2362563730 learned_addr learned my addr 172.21.15.172:0/2362563730 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:27.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a20ff640 1 -- 172.21.15.172:0/2362563730 >> v1:172.21.15.174:6789/0 conn(0x7fc99c06baf0 legacy=0x7fc99c1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:27.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a20ff640 1 -- 172.21.15.172:0/2362563730 >> v1:172.21.15.172:6789/0 conn(0x7fc99c1580b0 legacy=0x7fc99c161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:27.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a20ff640 1 -- 172.21.15.172:0/2362563730 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c0c1610 con 0x7fc99c06aed0 2024-03-16T19:31:27.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.986+0000 7fc9a10fd640 1 -- 172.21.15.172:0/2362563730 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017793359 0 0) 0x7fc99c0c1eb0 con 0x7fc99c1580b0 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.987+0000 7fc9a20ff640 1 --2- 172.21.15.172:0/2362563730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c06aed0 0x7fc99c06b2b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc994009870 tx=0x7fc99402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926869bb404be46c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.987+0000 7fc9a10fd640 1 -- 172.21.15.172:0/2362563730 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99402f840 con 0x7fc99c06aed0 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.987+0000 7fc9a10fd640 1 -- 172.21.15.172:0/2362563730 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99402f9a0 con 0x7fc99c06aed0 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.987+0000 7fc9a10fd640 1 -- 172.21.15.172:0/2362563730 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99402fd30 con 0x7fc99c06aed0 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.987+0000 7fc9a4363640 1 -- 172.21.15.172:0/2362563730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c06aed0 msgr2=0x7fc99c06b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.987+0000 7fc9a4363640 1 --2- 172.21.15.172:0/2362563730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c06aed0 0x7fc99c06b2b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc994009870 tx=0x7fc99402ba20 comp rx=0 tx=0).stop 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.987+0000 7fc9a4363640 1 -- 172.21.15.172:0/2362563730 shutdown_connections 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.987+0000 7fc9a4363640 1 --2- 172.21.15.172:0/2362563730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c06aed0 0x7fc99c06b2b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.988+0000 7fc9a4363640 1 -- 172.21.15.172:0/2362563730 >> 172.21.15.172:0/2362563730 conn(0x7fc99c0cc030 msgr2=0x7fc99c0cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.988+0000 7fc9a4363640 1 -- 172.21.15.172:0/2362563730 shutdown_connections 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.988+0000 7fc9a4363640 1 -- 172.21.15.172:0/2362563730 wait complete. 2024-03-16T19:31:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a4363640 1 Processor -- start 2024-03-16T19:31:27.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a4363640 1 -- start start 2024-03-16T19:31:27.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a4363640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc99c06aed0 0x7fc99c159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:27.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a4363640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc99c06baf0 0x7fc99c14ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:27.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a4363640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c1580b0 0x7fc99c1ef470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:27.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a4363640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c0c6170 con 0x7fc99c1580b0 2024-03-16T19:31:27.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a4363640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c0c52b0 con 0x7fc99c06aed0 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a4363640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c0c5680 con 0x7fc99c06baf0 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a2900640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c1580b0 0x7fc99c1ef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a2900640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c1580b0 0x7fc99c1ef470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45002/0 (socket says 172.21.15.172:45002) 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a2900640 1 -- 172.21.15.172:0/1888320085 learned_addr learned my addr 172.21.15.172:0/1888320085 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a18fe640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc99c06baf0 0x7fc99c14ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.989+0000 7fc9a20ff640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc99c06aed0 0x7fc99c159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc9a2900640 1 -- 172.21.15.172:0/1888320085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc99c06baf0 msgr2=0x7fc99c14ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc9a2900640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc99c06baf0 0x7fc99c14ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc9a2900640 1 -- 172.21.15.172:0/1888320085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc99c06aed0 msgr2=0x7fc99c159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc9a2900640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc99c06aed0 0x7fc99c159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc9a2900640 1 -- 172.21.15.172:0/1888320085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9940094d0 con 0x7fc99c1580b0 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc9a2900640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c1580b0 0x7fc99c1ef470 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00c7b0 tx=0x7fc98c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:27.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc99affd640 1 -- 172.21.15.172:0/1888320085 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00ce60 con 0x7fc99c1580b0 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc99affd640 1 -- 172.21.15.172:0/1888320085 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98c00a640 con 0x7fc99c1580b0 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c159000 con 0x7fc99c1580b0 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc99affd640 1 -- 172.21.15.172:0/1888320085 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00a9f0 con 0x7fc99c1580b0 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.990+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c0c5940 con 0x7fc99c1580b0 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.991+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c159190 con 0x7fc99c1580b0 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.991+0000 7fc99affd640 1 -- 172.21.15.172:0/1888320085 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc98c00ab50 con 0x7fc99c1580b0 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.992+0000 7fc99affd640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc98806b3e0 0x7fc98806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.992+0000 7fc99affd640 1 -- 172.21.15.172:0/1888320085 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fc98c0893b0 con 0x7fc99c1580b0 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.992+0000 7fc9a20ff640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc98806b3e0 0x7fc98806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:27.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.992+0000 7fc9a20ff640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc98806b3e0 0x7fc98806d8a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc994036000 tx=0x7fc994008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:27.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:27.996+0000 7fc99affd640 1 -- 172.21.15.172:0/1888320085 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc98c053f90 con 0x7fc99c1580b0 2024-03-16T19:31:28.033 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:28.052+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:28.096 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:28.096 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:28.105 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:28.146 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:28.165+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:28.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.183+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc99c0c5ef0 con 0x7fc99c1580b0 2024-03-16T19:31:28.223 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:28.243+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:28.252 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:28.252 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:28.260 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:28.325 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:28.344+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:28.367 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:28.367 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:28.376 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:28.425 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:28.445+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:28.483 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:28.483 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:28.485 INFO:tasks.workunit.client.0.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-16T19:31:28.492 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:28.526 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:28.545+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:28.549 INFO:tasks.workunit.client.0.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-16T19:31:28.588 INFO:tasks.workunit.client.0.smithi172.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T19:31:28.590 INFO:tasks.workunit.client.0.smithi172.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T19:31:28.590 INFO:tasks.workunit.client.0.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-16T19:31:28.627 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:28.646+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:28.628 INFO:tasks.workunit.client.0.smithi172.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T19:31:28.632 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:31:28.632 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-16T19:31:28.642 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:28.642 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:28.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:28.687 INFO:tasks.workunit:Running workunits matching fs/misc/trivial_sync.sh on client.0... 2024-03-16T19:31:28.689 INFO:tasks.workunit:Running workunit fs/misc/trivial_sync.sh... 2024-03-16T19:31:28.689 DEBUG:teuthology.orchestra.run.smithi172:workunit test fs/misc/trivial_sync.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/trivial_sync.sh 2024-03-16T19:31:28.728 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:28.747+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:28.782 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:28.782 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:28.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:28.834 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:28.854+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:28.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.874+0000 7fc99affd640 1 -- 172.21.15.172:0/1888320085 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v77) v1 ==== 101+0+0 (secure 0 0 0) 0x7fc98c018070 con 0x7fc99c1580b0 2024-03-16T19:31:28.855 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:31:28.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc98806b3e0 msgr2=0x7fc98806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:28.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc98806b3e0 0x7fc98806d8a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc994036000 tx=0x7fc994008030 comp rx=0 tx=0).stop 2024-03-16T19:31:28.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c1580b0 msgr2=0x7fc99c1ef470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:28.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c1580b0 0x7fc99c1ef470 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00c7b0 tx=0x7fc98c00cc70 comp rx=0 tx=0).stop 2024-03-16T19:31:28.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 shutdown_connections 2024-03-16T19:31:28.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc99c1580b0 0x7fc99c1ef470 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:28.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc98806b3e0 0x7fc98806d8a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:28.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc99c06baf0 0x7fc99c14ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:28.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 --2- 172.21.15.172:0/1888320085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc99c06aed0 0x7fc99c159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:28.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 >> 172.21.15.172:0/1888320085 conn(0x7fc99c0cc030 msgr2=0x7fc99c0c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:28.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 shutdown_connections 2024-03-16T19:31:28.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:28.878+0000 7fc9a4363640 1 -- 172.21.15.172:0/1888320085 wait complete. 2024-03-16T19:31:28.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:31:28.901 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:28.901 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:28.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:28.929 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:28.949+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:29.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.020+0000 7f6304db6640 1 Processor -- start 2024-03-16T19:31:29.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.020+0000 7f6304db6640 1 -- start start 2024-03-16T19:31:29.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.021+0000 7f6304db6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f63000b3630 0x7f63000cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.021+0000 7f6304db6640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f63000cb5e0 0x7f63000b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.021+0000 7f6304db6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63000b4270 con 0x7f6300154ed0 2024-03-16T19:31:29.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.021+0000 7f6304db6640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63000cb9f0 con 0x7f63000cb5e0 2024-03-16T19:31:29.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.021+0000 7f6304db6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630015b250 con 0x7f63000b3630 2024-03-16T19:31:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.022+0000 7f62fed9d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f63000cb5e0 0x7f63000b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.022+0000 7f62fed9d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f63000cb5e0 0x7f63000b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34282/0 (socket says 172.21.15.172:34282) 2024-03-16T19:31:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.022+0000 7f62fed9d640 1 -- 172.21.15.172:0/513369658 learned_addr learned my addr 172.21.15.172:0/513369658 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.022+0000 7f62fdd9b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f63000b3630 0x7f63000cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.022+0000 7f62fed9d640 1 -- 172.21.15.172:0/513369658 >> v2:172.21.15.174:3300/0 conn(0x7f63000b3630 msgr2=0x7f63000cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.022+0000 7f62fed9d640 1 --2- 172.21.15.172:0/513369658 >> v2:172.21.15.174:3300/0 conn(0x7f63000b3630 0x7f63000cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.022+0000 7f62fed9d640 1 -- 172.21.15.172:0/513369658 >> v1:172.21.15.174:6789/0 conn(0x7f6300154ed0 legacy=0x7f630015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.022+0000 7f62fed9d640 1 -- 172.21.15.172:0/513369658 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63001552b0 con 0x7f63000cb5e0 2024-03-16T19:31:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.023+0000 7f62fed9d640 1 --2- 172.21.15.172:0/513369658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63000cb5e0 0x7f63000b4700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f630006e150 tx=0x7f62f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3eba6bcf89a164 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.023+0000 7f62fd59a640 1 -- 172.21.15.172:0/513369658 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0030860 con 0x7f63000cb5e0 2024-03-16T19:31:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.023+0000 7f62fd59a640 1 -- 172.21.15.172:0/513369658 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f00309c0 con 0x7f63000cb5e0 2024-03-16T19:31:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.023+0000 7f62fd59a640 1 -- 172.21.15.172:0/513369658 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0030d50 con 0x7f63000cb5e0 2024-03-16T19:31:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.023+0000 7f6304db6640 1 -- 172.21.15.172:0/513369658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63000cb5e0 msgr2=0x7f63000b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.023+0000 7f6304db6640 1 --2- 172.21.15.172:0/513369658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63000cb5e0 0x7f63000b4700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f630006e150 tx=0x7f62f002b9a0 comp rx=0 tx=0).stop 2024-03-16T19:31:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.024+0000 7f6304db6640 1 -- 172.21.15.172:0/513369658 shutdown_connections 2024-03-16T19:31:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.024+0000 7f6304db6640 1 --2- 172.21.15.172:0/513369658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63000cb5e0 0x7f63000b4700 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:29.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.024+0000 7f6304db6640 1 --2- 172.21.15.172:0/513369658 >> v2:172.21.15.174:3300/0 conn(0x7f63000b3630 0x7f63000cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:29.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.024+0000 7f6304db6640 1 -- 172.21.15.172:0/513369658 >> 172.21.15.172:0/513369658 conn(0x7f630006bf60 msgr2=0x7f630006d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:29.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.024+0000 7f6304db6640 1 -- 172.21.15.172:0/513369658 shutdown_connections 2024-03-16T19:31:29.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.024+0000 7f6304db6640 1 -- 172.21.15.172:0/513369658 wait complete. 2024-03-16T19:31:29.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.025+0000 7f6304db6640 1 Processor -- start 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.025+0000 7f6304db6640 1 -- start start 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f6304db6640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63000b3630 0x7f630015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f6304db6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63000cb5e0 0x7f6300150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f6304db6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6300154ed0 0x7f6300153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f6304db6640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630015c940 con 0x7f63000b3630 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f6304db6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63001f1370 con 0x7f6300154ed0 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f6304db6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63001511f0 con 0x7f63000cb5e0 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fe59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63000b3630 0x7f630015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fe59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63000b3630 0x7f630015bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34294/0 (socket says 172.21.15.172:34294) 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fe59c640 1 -- 172.21.15.172:0/1436634314 learned_addr learned my addr 172.21.15.172:0/1436634314 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fdd9b640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63000cb5e0 0x7f6300150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fed9d640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6300154ed0 0x7f6300153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fe59c640 1 -- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63000cb5e0 msgr2=0x7f6300150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fe59c640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63000cb5e0 0x7f6300150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fe59c640 1 -- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6300154ed0 msgr2=0x7f6300153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fe59c640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6300154ed0 0x7f6300153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fe59c640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f00094d0 con 0x7f63000b3630 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62fe59c640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f63000b3630 0x7f630015bd00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f62f800c420 tx=0x7f62f800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80052d0 con 0x7f63000b3630 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f8005450 con 0x7f63000b3630 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8007260 con 0x7f63000b3630 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630015c390 con 0x7f63000b3630 2024-03-16T19:31:29.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.026+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63001f0a40 con 0x7f63000b3630 2024-03-16T19:31:29.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.027+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6300068d20 con 0x7f63000b3630 2024-03-16T19:31:29.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.028+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f62f80073c0 con 0x7f63000b3630 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.028+0000 7f62eeffd640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f62e006b3b0 0x7f62e006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.028+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f62f8088f60 con 0x7f63000b3630 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.028+0000 7f62fdd9b640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f62e006b3b0 0x7f62e006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.028+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v77) v1 ==== 54+0+16 (secure 0 0 0) 0x7f62f808b5a0 con 0x7f63000b3630 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.029+0000 7f6304db6640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6300151420 0x7f63000b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.029+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63001f1170 con 0x7f6300151420 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.029+0000 7f62fdd9b640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f62e006b3b0 0x7f62e006d870 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f63000ce860 tx=0x7f62f400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.029+0000 7f62fed9d640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6300151420 0x7f63000b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.029+0000 7f62fed9d640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f6300151420 0x7f63000b4700 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.030+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f63001f1170 con 0x7f6300151420 2024-03-16T19:31:29.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.044+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63000b3f00 con 0x7f6300151420 2024-03-16T19:31:29.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.046+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62f0030b30 con 0x7f6300151420 2024-03-16T19:31:29.027 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:29.028 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:29.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.046+0000 7f6304db6640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f630014f660 0x7f63001f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.046+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63000b3c20 con 0x7f630014f660 2024-03-16T19:31:29.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.047+0000 7f62fe59c640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f630014f660 0x7f63001f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.047+0000 7f62fe59c640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f630014f660 0x7f63001f3cf0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.047+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f62f808b580 con 0x7f630014f660 2024-03-16T19:31:29.030 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:29.050+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:29.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.063+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63001f4480 con 0x7f630014f660 2024-03-16T19:31:29.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.064+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f63001f4480 con 0x7f630014f660 2024-03-16T19:31:29.045 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:29.046 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:29.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.064+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=78}) v3 -- 0x7f630014fa40 con 0x7f63000b3630 2024-03-16T19:31:29.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.064+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63000681c0 con 0x7f63000b3630 2024-03-16T19:31:29.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.065+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62f8007670 con 0x7f63000b3630 2024-03-16T19:31:29.046 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:29.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.065+0000 7f6304db6640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f63001474f0 0x7f6300147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.065+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6300147e50 con 0x7f63001474f0 2024-03-16T19:31:29.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.065+0000 7f62fdd9b640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f63001474f0 0x7f6300147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.065+0000 7f62fdd9b640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f63001474f0 0x7f6300147910 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.047 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.066+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6300147e50 con 0x7f63001474f0 2024-03-16T19:31:29.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.080+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6300061c00 con 0x7f63001474f0 2024-03-16T19:31:29.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.082+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6300061c00 con 0x7f63001474f0 2024-03-16T19:31:29.063 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:29.064 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:29.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.082+0000 7f6304db6640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f63000626a0 0x7f6300152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.082+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63000cbbe0 con 0x7f63000626a0 2024-03-16T19:31:29.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.082+0000 7f62fed9d640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f63000626a0 0x7f6300152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.064 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:29.064 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:29.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:18 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 132.490112716, 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035258519999999999, 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593389+0000", 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593390+0000", 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2699999999999999e-07 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593395+0000", 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6489999999999999e-06 2024-03-16T19:31:29.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593403+0000", 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0609999999999996e-06 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593407+0000", 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5870000000000003e-06 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593446+0000", 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9048999999999998e-05 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.593460+0000", 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4143e-05 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596877+0000", 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003417046 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:16.596915+0000", 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7790000000000002e-05 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.699886545, 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.69912631, 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:29.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:31:29.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:29.076 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.69831235500001, 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:29.080 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.69767243000001, 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.697014333, 2024-03-16T19:31:29.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:31:29.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.696481758, 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.088 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:31:29.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:31:29.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.63830985499999, 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:31:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.63770766100001, 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:29.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:31:29.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.637064716, 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:31:29.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.636560755, 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:29.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:31:29.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=17'106)", 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.47841448899999, 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036395640000000001, 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605088+0000", 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5699999999999998e-07 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605142+0000", 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3477000000000003e-05 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605148+0000", 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4520000000000002e-06 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605152+0000", 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190000000000001e-06 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605183+0000", 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0726999999999999e-05 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605192+0000", 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5100000000000004e-06 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608704+0000", 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003512331 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.608727+0000", 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2691e-05 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.47788435300001, 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:31:29.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.083+0000 7f62fed9d640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f63000626a0 0x7f6300152630 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:31:29.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:31:29.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:31:29.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.116 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.725915094, 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.725890558, 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:31:29.121 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:29.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:31:29.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.72588602800001, 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.126 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:31:29.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.725876216, 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:29.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.135 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.72587236299999, 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.725869047, 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:31:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:31:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.72467346400001, 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:31:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:31:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:31:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:31:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:31:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:31:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:31:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:31:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:31:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:31:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.151 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:29.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.084+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f63000cbbe0 con 0x7f63000626a0 2024-03-16T19:31:29.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:29.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.098+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6300152e40 con 0x7f63000626a0 2024-03-16T19:31:29.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.101+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6300152e40 con 0x7f63000626a0 2024-03-16T19:31:29.154 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.101+0000 7f6304db6640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f63001532d0 0x7f630002bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.101+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63000cd5c0 con 0x7f63001532d0 2024-03-16T19:31:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.101+0000 7f62fe59c640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f63001532d0 0x7f630002bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.101+0000 7f62fe59c640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f63001532d0 0x7f630002bde0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.102+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f63000cd5c0 con 0x7f63001532d0 2024-03-16T19:31:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.116+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6300151db0 con 0x7f63001532d0 2024-03-16T19:31:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.118+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6300151db0 con 0x7f63001532d0 2024-03-16T19:31:29.155 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:29.156 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:29.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.118+0000 7f6304db6640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f630002c830 0x7f630002cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.118+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f630002d190 con 0x7f630002c830 2024-03-16T19:31:29.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.118+0000 7f62fdd9b640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f630002c830 0x7f630002cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.118+0000 7f62fdd9b640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f630002c830 0x7f630002cc50 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.156 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:29.173+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:29.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.190+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f630002d190 con 0x7f630002c830 2024-03-16T19:31:29.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.205+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630002d190 con 0x7f630002c830 2024-03-16T19:31:29.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.207+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f630002d190 con 0x7f630002c830 2024-03-16T19:31:29.188 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:29.188 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:29.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.207+0000 7f6304db6640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f63002077a0 0x7f6300209b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:29.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.207+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f630020cae0 con 0x7f63002077a0 2024-03-16T19:31:29.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.207+0000 7f62fed9d640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f63002077a0 0x7f6300209b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:29.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.208+0000 7f62fed9d640 1 --2- 172.21.15.172:0/1436634314 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f63002077a0 0x7f6300209b90 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:29.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.208+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f630020cae0 con 0x7f63002077a0 2024-03-16T19:31:29.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.222+0000 7f6304db6640 1 -- 172.21.15.172:0/1436634314 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630020cae0 con 0x7f63002077a0 2024-03-16T19:31:29.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:29.224+0000 7f62eeffd640 1 -- 172.21.15.172:0/1436634314 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f630020cae0 con 0x7f63002077a0 2024-03-16T19:31:29.206 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:29.206 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:29.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:29.255 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:29.274+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:29.279 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:29.280 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:29.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:29.356 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:29.375+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:29.395 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:29.395 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:29.395 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:29.395 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:29.395 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:29.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:29.488 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:29.507+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.21442815, 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.553 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:31:29.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:31:29.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:31:29.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.214411036, 2024-03-16T19:31:29.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.214403756, 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:31:29.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:31:29.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:31:29.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:31:29.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:29.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:31:29.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:31:29.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:31:29.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:31:29.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:31:29.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:31:29.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:29.571 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.214389889, 2024-03-16T19:31:29.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:31:29.571 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.571 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.571 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:29.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:29.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:29.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:31:29.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:31:29.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:31:29.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:31:29.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:31:29.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:29.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:31:29.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:31:29.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.579 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.213717054, 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:31:29.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.211418804, 2024-03-16T19:31:29.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.588 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.21137370300001, 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.21134564800001, 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:31:29.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.211324234, 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.211301883, 2024-03-16T19:31:29.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:31:29.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.211281484, 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.211175492, 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:31:29.610 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.209383781, 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.618 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.20933628, 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:29.622 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.20909702900001, 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:31:29.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:31:29.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 113.200598884, 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:31:29.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.631 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.196268025999998, 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:31:29.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:31:29.636 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 64.195637223999995, 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:31:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:48 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e77)", 2024-03-16T19:31:29.641 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:29.381986+0000", 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.18977901899999999, 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001303532, 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.381986+0000", 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.381986+0000", 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.381988+0000", 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.869e-06 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.381997+0000", 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6730000000000004e-06 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382025+0000", 2024-03-16T19:31:29.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8092000000000001e-05 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382038+0000", 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2333e-05 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382109+0000", 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1104000000000002e-05 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382176+0000", 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7533999999999994e-05 2024-03-16T19:31:29.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382274+0000", 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7132000000000004e-05 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382979+0000", 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070523000000000005 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383190+0000", 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021099899999999999 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383243+0000", 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3579999999999999e-05 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383289+0000", 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5986000000000003e-05 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:48 2.6 e76/18 ondisk, result = 0)", 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:29.383116+0000", 2024-03-16T19:31:29.646 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.188648749, 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017887099999999999, 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383116+0000", 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383116+0000", 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383117+0000", 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5700000000000002e-07 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383120+0000", 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5410000000000001e-06 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383131+0000", 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0866999999999999e-05 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:29.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383136+0000", 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0699999999999997e-06 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383174+0000", 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8293000000000001e-05 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383182+0000", 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5089999999999996e-06 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383295+0000", 2024-03-16T19:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112934 2024-03-16T19:31:29.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.650 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:29.650 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:29.650 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:29.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:29.652 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:29.670+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:29.752 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:29.770+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:29.798 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:29.799 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:29.807 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:29.853 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:29.872+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:29.914 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:29.915 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:29.915 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:29.915 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:29.915 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:29.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:29.965 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:29.984+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:30.054 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:30.073+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:30.071 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:30.071 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.733962404, 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:30.075 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.733947931, 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:31:30.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.733443442, 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.079 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.733428046, 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:31:30.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.732969363, 2024-03-16T19:31:30.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:31:30.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:31:30.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.732423193, 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:31:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:31:30.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.731866102, 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.096 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:31:30.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.730664354, 2024-03-16T19:31:30.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:31:30.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.730085645, 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:30.104 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:30.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.730036118, 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:31:30.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.729075475, 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:30.111 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.725361277, 2024-03-16T19:31:30.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:31:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:31:30.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 110.284960365, 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:31:30.118 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:31:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:31:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.719407362, 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:31:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:31:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 107.211808637, 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:31:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:31:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:31:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPull(2.e e67/66 cost 1048576)", 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.307134728000001, 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026662799999999998, 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0500000000000003e-07 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784292+0000", 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5139e-05 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784304+0000", 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1501000000000001e-05 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784313+0000", 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8189999999999998e-06 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784384+0000", 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1192999999999994e-05 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784543+0000", 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159371 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:30.133 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:30.134 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:30.155 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:30.175+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:30.240 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:30.240 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:30.241 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:30.241 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:30.249 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:30.293 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:30.312+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:30.356 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:30.356 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:30.356 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:30.356 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:30.356 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:30.357 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:30.377+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:30.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:30.457 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:30.477+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.178672738, 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:30.516 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:31:30.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:30.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:30.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.177445316, 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:30.525 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.17704261, 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:30.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.176433859, 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:30.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:31:30.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:31:30.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:30.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.17598999000001, 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.175847214, 2024-03-16T19:31:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.175832673, 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:30.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.17295497800001, 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:30.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:30.549 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:30.550 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:30.592 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:30.611+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:30.659 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:30.678+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:30.698 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:30.698 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:30.699 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:30.699 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:30.708 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:30.791 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:30.810+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:30.853 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:30.853 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:30.853 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:30.853 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:30.853 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:30.860 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:30.880+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:30.862 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:30.864 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-16T19:31:30.865 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:31:30.865 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-16T19:31:30.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2024-03-16T19:31:30.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-16T19:31:30.961 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:30.981+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:30.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.995+0000 7f1c15cff640 1 Processor -- start 2024-03-16T19:31:30.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.995+0000 7f1c15cff640 1 -- start start 2024-03-16T19:31:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.995+0000 7f1c15cff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1c100b5f10 0x7f1c100caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.995+0000 7f1c15cff640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1015b990 con 0x7f1c10154e00 2024-03-16T19:31:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.995+0000 7f1c15cff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c100cc090 con 0x7f1c100c7800 2024-03-16T19:31:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.995+0000 7f1c15cff640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1015b570 con 0x7f1c100b5f10 2024-03-16T19:31:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.996+0000 7f1c0f7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f1c10154e00 0x7f1c1006afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56024/0 (socket says 172.21.15.172:56024) 2024-03-16T19:31:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.996+0000 7f1c0f7fe640 1 -- 172.21.15.172:0/4190542185 learned_addr learned my addr 172.21.15.172:0/4190542185 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.996+0000 7f1c0effd640 1 --2- 172.21.15.172:0/4190542185 >> v2:172.21.15.174:3300/0 conn(0x7f1c100b5f10 0x7f1c100caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.996+0000 7f1c0e7fc640 1 -- 172.21.15.172:0/4190542185 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976569583 0 0) 0x7f1c1015b990 con 0x7f1c10154e00 2024-03-16T19:31:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.996+0000 7f1c0e7fc640 1 -- 172.21.15.172:0/4190542185 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf8003520 con 0x7f1c10154e00 2024-03-16T19:31:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.996+0000 7f1c0effd640 1 -- 172.21.15.172:0/4190542185 >> v1:172.21.15.174:6789/0 conn(0x7f1c100c7800 legacy=0x7f1c10165260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.996+0000 7f1c0effd640 1 -- 172.21.15.172:0/4190542185 >> v1:172.21.15.172:6789/0 conn(0x7f1c10154e00 legacy=0x7f1c1006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.996+0000 7f1c0effd640 1 -- 172.21.15.172:0/4190542185 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1015abd0 con 0x7f1c100b5f10 2024-03-16T19:31:30.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.997+0000 7f1c0effd640 1 --2- 172.21.15.172:0/4190542185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 0x7f1c100caed0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1c040097a0 tx=0x7f1c0402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1af6740b689fcb22 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:30.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.997+0000 7f1c0e7fc640 1 -- 172.21.15.172:0/4190542185 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0402f810 con 0x7f1c100b5f10 2024-03-16T19:31:30.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.997+0000 7f1c0e7fc640 1 -- 172.21.15.172:0/4190542185 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0402f970 con 0x7f1c100b5f10 2024-03-16T19:31:30.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.997+0000 7f1c0e7fc640 1 -- 172.21.15.172:0/4190542185 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0402fce0 con 0x7f1c100b5f10 2024-03-16T19:31:30.979 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.998+0000 7f1c15cff640 1 -- 172.21.15.172:0/4190542185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 msgr2=0x7f1c100caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:30.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.998+0000 7f1c15cff640 1 --2- 172.21.15.172:0/4190542185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 0x7f1c100caed0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1c040097a0 tx=0x7f1c0402b9f0 comp rx=0 tx=0).stop 2024-03-16T19:31:30.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.998+0000 7f1c15cff640 1 -- 172.21.15.172:0/4190542185 shutdown_connections 2024-03-16T19:31:30.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.998+0000 7f1c15cff640 1 --2- 172.21.15.172:0/4190542185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 0x7f1c100caed0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:30.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.998+0000 7f1c15cff640 1 -- 172.21.15.172:0/4190542185 >> 172.21.15.172:0/4190542185 conn(0x7f1c10066c20 msgr2=0x7f1c10067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:30.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.998+0000 7f1c15cff640 1 -- 172.21.15.172:0/4190542185 shutdown_connections 2024-03-16T19:31:30.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.998+0000 7f1c15cff640 1 -- 172.21.15.172:0/4190542185 wait complete. 2024-03-16T19:31:30.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.999+0000 7f1c15cff640 1 Processor -- start 2024-03-16T19:31:30.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:30.999+0000 7f1c15cff640 1 -- start start 2024-03-16T19:31:30.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c15cff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 0x7f1c10158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:30.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c15cff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c100c7800 0x7f1c10155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:30.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c15cff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c10154e00 0x7f1c101518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:30.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c15cff640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c101554b0 con 0x7f1c10154e00 2024-03-16T19:31:30.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c15cff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1015a670 con 0x7f1c100b5f10 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c15cff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c101f0bb0 con 0x7f1c100c7800 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c10154e00 0x7f1c101518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c10154e00 0x7f1c101518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34308/0 (socket says 172.21.15.172:34308) 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 0x7f1c10158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0ffff640 1 -- 172.21.15.172:0/1769442699 learned_addr learned my addr 172.21.15.172:0/1769442699 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0effd640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c100c7800 0x7f1c10155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0ffff640 1 -- 172.21.15.172:0/1769442699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c100c7800 msgr2=0x7f1c10155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0ffff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c100c7800 0x7f1c10155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0ffff640 1 -- 172.21.15.172:0/1769442699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 msgr2=0x7f1c10158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0ffff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 0x7f1c10158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0ffff640 1 -- 172.21.15.172:0/1769442699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c04009400 con 0x7f1c10154e00 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0f7fe640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 0x7f1c10158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.000+0000 7f1c0ffff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c10154e00 0x7f1c101518d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800c7a0 tx=0x7f1c0800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:30.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.001+0000 7f1bf7fff640 1 -- 172.21.15.172:0/1769442699 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08003b70 con 0x7f1c10154e00 2024-03-16T19:31:30.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.001+0000 7f1bf7fff640 1 -- 172.21.15.172:0/1769442699 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c08003cf0 con 0x7f1c10154e00 2024-03-16T19:31:30.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.001+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1006b850 con 0x7f1c10154e00 2024-03-16T19:31:30.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.001+0000 7f1bf7fff640 1 -- 172.21.15.172:0/1769442699 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08003ea0 con 0x7f1c10154e00 2024-03-16T19:31:30.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.001+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c10155770 con 0x7f1c10154e00 2024-03-16T19:31:30.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.002+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c101f10d0 con 0x7f1c10154e00 2024-03-16T19:31:30.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.002+0000 7f1bf7fff640 1 -- 172.21.15.172:0/1769442699 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1c0800aa90 con 0x7f1c10154e00 2024-03-16T19:31:30.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.002+0000 7f1bf7fff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1bfc06b3e0 0x7f1bfc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:30.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.003+0000 7f1bf7fff640 1 -- 172.21.15.172:0/1769442699 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f1c08089990 con 0x7f1c10154e00 2024-03-16T19:31:30.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.003+0000 7f1c0f7fe640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1bfc06b3e0 0x7f1bfc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:30.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.003+0000 7f1c0f7fe640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1bfc06b3e0 0x7f1bfc06d8a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1c000011e0 tx=0x7f1c00007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:30.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.007+0000 7f1bf7fff640 1 -- 172.21.15.172:0/1769442699 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c08054600 con 0x7f1c10154e00 2024-03-16T19:31:31.009 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:31.009 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:31.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.67165318000001, 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:31:31.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:31:31.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:31:31.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.671574062, 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:31.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:31.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:31.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:31.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.670959938, 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:31.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:31.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:31:31.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.670519869, 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:31.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:31.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:31.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.670278279, 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:31.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:31.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:31:31.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:31:31.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:31:31.030 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.030 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:31.030 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.030 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:31.030 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:31.030 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:31.031 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.033+0000 7fdd9c3f4640 1 Processor -- start 2024-03-16T19:31:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.033+0000 7fdd9c3f4640 1 -- start start 2024-03-16T19:31:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9c3f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd94157ec0 0x7fdd9415b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9c3f4640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdd9406dc20 0x7fdd9415b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9c3f4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdd940c9080 0x7fdd940c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9c3f4640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd940cba70 con 0x7fdd9406dc20 2024-03-16T19:31:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9c3f4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd9406bfe0 con 0x7fdd940c9080 2024-03-16T19:31:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9c3f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd940700f0 con 0x7fdd94157ec0 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9998f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdd9406dc20 0x7fdd9415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9998f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdd9406dc20 0x7fdd9415b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34316/0 (socket says 172.21.15.172:34316) 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9998f640 1 -- 172.21.15.172:0/3311923944 learned_addr learned my addr 172.21.15.172:0/3311923944 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9a190640 1 --2- 172.21.15.172:0/3311923944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd94157ec0 0x7fdd9415b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.034+0000 7fdd9a991640 1 --2- 172.21.15.172:0/3311923944 >> v2:172.21.15.174:3300/0 conn(0x7fdd940c9080 0x7fdd940c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.035+0000 7fdd9998f640 1 -- 172.21.15.172:0/3311923944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd94157ec0 msgr2=0x7fdd9415b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.035+0000 7fdd9998f640 1 --2- 172.21.15.172:0/3311923944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd94157ec0 0x7fdd9415b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.035+0000 7fdd9998f640 1 -- 172.21.15.172:0/3311923944 >> v2:172.21.15.174:3300/0 conn(0x7fdd940c9080 msgr2=0x7fdd940c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.035+0000 7fdd9998f640 1 --2- 172.21.15.172:0/3311923944 >> v2:172.21.15.174:3300/0 conn(0x7fdd940c9080 0x7fdd940c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.035+0000 7fdd9998f640 1 -- 172.21.15.172:0/3311923944 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd940c8ac0 con 0x7fdd9406dc20 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.035+0000 7fdd9998f640 1 --2- 172.21.15.172:0/3311923944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 0x7fdd9415b540 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdd940cf160 tx=0x7fdd8402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385fe7addf47bf6c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.035+0000 7fdd9918e640 1 -- 172.21.15.172:0/3311923944 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd84030860 con 0x7fdd9406dc20 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9918e640 1 -- 172.21.15.172:0/3311923944 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd840309c0 con 0x7fdd9406dc20 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9918e640 1 -- 172.21.15.172:0/3311923944 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd84030d50 con 0x7fdd9406dc20 2024-03-16T19:31:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/3311923944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 msgr2=0x7fdd9415b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/3311923944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 0x7fdd9415b540 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdd940cf160 tx=0x7fdd8402b9a0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/3311923944 shutdown_connections 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/3311923944 >> v2:172.21.15.174:3300/0 conn(0x7fdd940c9080 0x7fdd940c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/3311923944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 0x7fdd9415b540 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/3311923944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd94157ec0 0x7fdd9415b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/3311923944 >> 172.21.15.172:0/3311923944 conn(0x7fdd94070810 msgr2=0x7fdd94070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/3311923944 shutdown_connections 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.036+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/3311923944 wait complete. 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.037+0000 7fdd9c3f4640 1 Processor -- start 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.037+0000 7fdd9c3f4640 1 -- start start 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9c3f4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 0x7fdd940c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9c3f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd940c9080 0x7fdd940cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9c3f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd94157ec0 0x7fdd94158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9c3f4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd940c53a0 con 0x7fdd9406dc20 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9c3f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd94150870 con 0x7fdd94157ec0 2024-03-16T19:31:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9c3f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd941f0cf0 con 0x7fdd940c9080 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9a190640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 0x7fdd940c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9a190640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 0x7fdd940c4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34320/0 (socket says 172.21.15.172:34320) 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9a190640 1 -- 172.21.15.172:0/1126890366 learned_addr learned my addr 172.21.15.172:0/1126890366 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9998f640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd940c9080 0x7fdd940cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9a991640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd94157ec0 0x7fdd94158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9a190640 1 -- 172.21.15.172:0/1126890366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd940c9080 msgr2=0x7fdd940cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9a190640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd940c9080 0x7fdd940cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9a190640 1 -- 172.21.15.172:0/1126890366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd94157ec0 msgr2=0x7fdd94158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9a190640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd94157ec0 0x7fdd94158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.038+0000 7fdd9a190640 1 -- 172.21.15.172:0/1126890366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd840094d0 con 0x7fdd9406dc20 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.039+0000 7fdd9a190640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 0x7fdd940c4e60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00e9b0 tx=0x7fdd8c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.039+0000 7fdd8affd640 1 -- 172.21.15.172:0/1126890366 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c009670 con 0x7fdd9406dc20 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.039+0000 7fdd8affd640 1 -- 172.21.15.172:0/1126890366 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c00c9a0 con 0x7fdd9406dc20 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.039+0000 7fdd8affd640 1 -- 172.21.15.172:0/1126890366 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00cd50 con 0x7fdd9406dc20 2024-03-16T19:31:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.039+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd941f1220 con 0x7fdd9406dc20 2024-03-16T19:31:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.039+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd941582a0 con 0x7fdd9406dc20 2024-03-16T19:31:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.040+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd940688d0 con 0x7fdd9406dc20 2024-03-16T19:31:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.040+0000 7fdd8affd640 1 -- 172.21.15.172:0/1126890366 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fdd8c013470 con 0x7fdd9406dc20 2024-03-16T19:31:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.041+0000 7fdd8affd640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdd7806b3b0 0x7fdd7806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.041+0000 7fdd8affd640 1 -- 172.21.15.172:0/1126890366 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fdd8c089df0 con 0x7fdd9406dc20 2024-03-16T19:31:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.041+0000 7fdd9998f640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdd7806b3b0 0x7fdd7806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.041+0000 7fdd9998f640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdd7806b3b0 0x7fdd7806d870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd84036000 tx=0x7fdd84008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.049+0000 7fdd8affd640 1 -- 172.21.15.172:0/1126890366 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd8c054a60 con 0x7fdd9406dc20 2024-03-16T19:31:31.062 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:31.081+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:31.163 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:31.182+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:31.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.188+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) v1 -- 0x7f1c1006b6d0 con 0x7f1bfc06b3e0 2024-03-16T19:31:31.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.189+0000 7f1bf7fff640 1 -- 172.21.15.172:0/1769442699 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+294 (secure 0 0 0) 0x7f1c00006b60 con 0x7f1bfc06b3e0 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:moved 0 / 194 (0%) 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:avg 27.7143 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:stddev 4.02543 -> 4.02543 (expected baseline 4.87392) 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:min osd.1 with 20 -> 20 pgs (0.72165 -> 0.72165 * mean) 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:max osd.7 with 30 -> 30 pgs (1.08247 -> 1.08247 * mean) 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:oload 110 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:max_change 3 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:max_change_osds 4 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:average_utilization 0.0003 2024-03-16T19:31:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:overload_utilization 0.0003 2024-03-16T19:31:31.171 INFO:teuthology.orchestra.run.smithi172.stderr:no change 2024-03-16T19:31:31.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.190+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1bfc06b3e0 msgr2=0x7f1bfc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.190+0000 7f1c15cff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1bfc06b3e0 0x7f1bfc06d8a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1c000011e0 tx=0x7f1c00007d30 comp rx=0 tx=0).stop 2024-03-16T19:31:31.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c10154e00 msgr2=0x7f1c101518d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c10154e00 0x7f1c101518d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800c7a0 tx=0x7f1c0800cc60 comp rx=0 tx=0).stop 2024-03-16T19:31:31.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 shutdown_connections 2024-03-16T19:31:31.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c10154e00 0x7f1c101518d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1bfc06b3e0 0x7f1bfc06d8a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c100c7800 0x7f1c10155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 --2- 172.21.15.172:0/1769442699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c100b5f10 0x7f1c10158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 >> 172.21.15.172:0/1769442699 conn(0x7f1c10066c20 msgr2=0x7f1c100ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:31.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 shutdown_connections 2024-03-16T19:31:31.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.191+0000 7f1c15cff640 1 -- 172.21.15.172:0/1769442699 wait complete. 2024-03-16T19:31:31.182 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:31.182 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:31.182 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:31.182 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:31.183 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-16T19:31:31.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-03-16T19:31:31.191 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:31.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:31:31.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.235+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fdd94075a70 con 0x7fdd9406dc20 2024-03-16T19:31:31.264 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:31.283+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:31.297 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:31.297 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:31.297 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:31.297 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:31.297 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:31.308 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:31.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.336+0000 7f770dad0640 1 Processor -- start 2024-03-16T19:31:31.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.336+0000 7f770dad0640 1 -- start start 2024-03-16T19:31:31.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.337+0000 7f770dad0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f77080b4700 0x7f77080b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.337+0000 7f770dad0640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080b4270 con 0x7f77080b3630 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.337+0000 7f770dad0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770815b250 con 0x7f7708154ed0 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.337+0000 7f770dad0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77080b5050 con 0x7f77080b4700 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.337+0000 7f76fffff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f77080b3630 0x7f77080cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56034/0 (socket says 172.21.15.172:56034) 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.337+0000 7f76fffff640 1 -- 172.21.15.172:0/1010234827 learned_addr learned my addr 172.21.15.172:0/1010234827 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.337+0000 7f7707fff640 1 --2- 172.21.15.172:0/1010234827 >> v2:172.21.15.174:3300/0 conn(0x7f77080b4700 0x7f77080b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.337+0000 7f7706ffd640 1 -- 172.21.15.172:0/1010234827 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749334749 0 0) 0x7f77080b4270 con 0x7f77080b3630 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.337+0000 7f7706ffd640 1 -- 172.21.15.172:0/1010234827 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f4003520 con 0x7f77080b3630 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.338+0000 7f7707fff640 1 -- 172.21.15.172:0/1010234827 >> v1:172.21.15.174:6789/0 conn(0x7f7708154ed0 legacy=0x7f770815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.338+0000 7f7706ffd640 1 -- 172.21.15.172:0/1010234827 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452263676 0 0) 0x7f770815b250 con 0x7f7708154ed0 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.338+0000 7f7707fff640 1 -- 172.21.15.172:0/1010234827 >> v1:172.21.15.172:6789/0 conn(0x7f77080b3630 legacy=0x7f77080cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.338+0000 7f7707fff640 1 -- 172.21.15.172:0/1010234827 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77081552b0 con 0x7f77080b4700 2024-03-16T19:31:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.338+0000 7f7707fff640 1 --2- 172.21.15.172:0/1010234827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77080b4700 0x7f77080b4b10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7700009d90 tx=0x7f770002bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e18151705c244c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.338+0000 7f7706ffd640 1 -- 172.21.15.172:0/1010234827 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0001670 con 0x7f77080b4700 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.339+0000 7f7706ffd640 1 -- 172.21.15.172:0/1010234827 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770002c860 con 0x7f77080b4700 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.339+0000 7f7706ffd640 1 -- 172.21.15.172:0/1010234827 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770002cbf0 con 0x7f77080b4700 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.339+0000 7f770dad0640 1 -- 172.21.15.172:0/1010234827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77080b4700 msgr2=0x7f77080b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.339+0000 7f770dad0640 1 --2- 172.21.15.172:0/1010234827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77080b4700 0x7f77080b4b10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7700009d90 tx=0x7f770002bd30 comp rx=0 tx=0).stop 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.339+0000 7f770dad0640 1 -- 172.21.15.172:0/1010234827 shutdown_connections 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.339+0000 7f770dad0640 1 --2- 172.21.15.172:0/1010234827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77080b4700 0x7f77080b4b10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.339+0000 7f770dad0640 1 -- 172.21.15.172:0/1010234827 >> 172.21.15.172:0/1010234827 conn(0x7f770806bf60 msgr2=0x7f770806d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.339+0000 7f770dad0640 1 -- 172.21.15.172:0/1010234827 shutdown_connections 2024-03-16T19:31:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.339+0000 7f770dad0640 1 -- 172.21.15.172:0/1010234827 wait complete. 2024-03-16T19:31:31.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.340+0000 7fc6bc38e640 1 Processor -- start 2024-03-16T19:31:31.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.340+0000 7f770dad0640 1 Processor -- start 2024-03-16T19:31:31.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.340+0000 7fc6bc38e640 1 -- start start 2024-03-16T19:31:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.340+0000 7f770dad0640 1 -- start start 2024-03-16T19:31:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f770dad0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77080b3630 0x7f7708151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f770dad0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77080b4700 0x7f77080cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f770dad0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7708154ed0 0x7f77080cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f770dad0640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770815c8f0 con 0x7f77080b4700 2024-03-16T19:31:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f770dad0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770814edb0 con 0x7f77080b3630 2024-03-16T19:31:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f770dad0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770815bec0 con 0x7f7708154ed0 2024-03-16T19:31:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f76fffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77080b4700 0x7f77080cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7fc6bc38e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc6b4152af0 0x7fc6b40b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f76fffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77080b4700 0x7f77080cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34326/0 (socket says 172.21.15.172:34326) 2024-03-16T19:31:31.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f7707fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7708154ed0 0x7f77080cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f77077fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77080b3630 0x7f7708151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7f76fffff640 1 -- 172.21.15.172:0/2940987751 learned_addr learned my addr 172.21.15.172:0/2940987751 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7fc6bc38e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc6b40c33d0 0x7fc6b40c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7fc6bc38e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b4152ed0 con 0x7fc6b40c97f0 2024-03-16T19:31:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7fc6bc38e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b40c55b0 con 0x7fc6b4152af0 2024-03-16T19:31:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7fc6bc38e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b40cec50 con 0x7fc6b40c33d0 2024-03-16T19:31:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7fc6b9929640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc6b40c33d0 0x7fc6b40c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7fc6ba12a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc6b4152af0 0x7fc6b40b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7fc6ba92b640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc6b40c97f0 0x7fc6b40ca250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:59070/0 (socket says 172.21.15.172:59070) 2024-03-16T19:31:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.341+0000 7fc6ba92b640 1 -- 172.21.15.172:0/983276276 learned_addr learned my addr 172.21.15.172:0/983276276 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f77077fe640 1 -- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7708154ed0 msgr2=0x7f77080cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7708154ed0 0x7f77080cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f77077fe640 1 -- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77080b4700 msgr2=0x7f77080cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f77080b4700 0x7f77080cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f77077fe640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7700009a70 con 0x7f77080b3630 2024-03-16T19:31:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f7707fff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7708154ed0 0x7f77080cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:31.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7fc6b9929640 1 -- 172.21.15.172:0/983276276 >> v2:172.21.15.172:3300/0 conn(0x7fc6b4152af0 msgr2=0x7fc6b40b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:31.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7fc6b9929640 1 --2- 172.21.15.172:0/983276276 >> v2:172.21.15.172:3300/0 conn(0x7fc6b4152af0 0x7fc6b40b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7fc6b9929640 1 -- 172.21.15.172:0/983276276 >> v1:172.21.15.174:6789/0 conn(0x7fc6b40c97f0 legacy=0x7fc6b40ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.327 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7fc6b9929640 1 -- 172.21.15.172:0/983276276 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b40c4ff0 con 0x7fc6b40c33d0 2024-03-16T19:31:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77080b3630 0x7f7708151820 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f76f800c410 tx=0x7f76f800c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80035f0 con 0x7f77080b3630 2024-03-16T19:31:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f8003750 con 0x7f77080b3630 2024-03-16T19:31:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77080c6500 con 0x7f77080b3630 2024-03-16T19:31:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8003b00 con 0x7f77080b3630 2024-03-16T19:31:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77080c6770 con 0x7f77080b3630 2024-03-16T19:31:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.342+0000 7fc6b9929640 1 --2- 172.21.15.172:0/983276276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b40c33d0 0x7fc6b40c92b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc6b40c8a70 tx=0x7fc6a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=763eb1b6431183ca server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.343+0000 7fc6b9128640 1 -- 172.21.15.172:0/983276276 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4031830 con 0x7fc6b40c33d0 2024-03-16T19:31:31.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.343+0000 7fc6b9128640 1 -- 172.21.15.172:0/983276276 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a4031990 con 0x7fc6b40c33d0 2024-03-16T19:31:31.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.343+0000 7fc6b9128640 1 -- 172.21.15.172:0/983276276 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4031d20 con 0x7fc6b40c33d0 2024-03-16T19:31:31.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.343+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7708068d20 con 0x7f77080b3630 2024-03-16T19:31:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.343+0000 7fc6bc38e640 1 -- 172.21.15.172:0/983276276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b40c33d0 msgr2=0x7fc6b40c92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.343+0000 7fc6bc38e640 1 --2- 172.21.15.172:0/983276276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b40c33d0 0x7fc6b40c92b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc6b40c8a70 tx=0x7fc6a402b8d0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.343+0000 7fc6bc38e640 1 -- 172.21.15.172:0/983276276 shutdown_connections 2024-03-16T19:31:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.343+0000 7fc6bc38e640 1 --2- 172.21.15.172:0/983276276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b40c33d0 0x7fc6b40c92b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7fc6bc38e640 1 --2- 172.21.15.172:0/983276276 >> v2:172.21.15.172:3300/0 conn(0x7fc6b4152af0 0x7fc6b40b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7fc6bc38e640 1 -- 172.21.15.172:0/983276276 >> 172.21.15.172:0/983276276 conn(0x7fc6b4066c20 msgr2=0x7fc6b4067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7fc6bc38e640 1 -- 172.21.15.172:0/983276276 shutdown_connections 2024-03-16T19:31:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7fc6bc38e640 1 -- 172.21.15.172:0/983276276 wait complete. 2024-03-16T19:31:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f76f8003c60 con 0x7f77080b3630 2024-03-16T19:31:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7f7704ff9640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f76e806b3b0 0x7f76e806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f76f80898d0 con 0x7f77080b3630 2024-03-16T19:31:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7f76fffff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f76e806b3b0 0x7f76e806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v77) v1 ==== 54+0+16 (secure 0 0 0) 0x7f76f808a0c0 con 0x7f77080b3630 2024-03-16T19:31:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.344+0000 7f770dad0640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f770814e870 0x7f77081f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.345+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77080b3f00 con 0x7f770814e870 2024-03-16T19:31:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.345+0000 7f76fffff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f76e806b3b0 0x7f76e806d870 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f76f0001df0 tx=0x7f76f0001d80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.345+0000 7f7707fff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f770814e870 0x7f77081f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.345+0000 7fc6bc38e640 1 Processor -- start 2024-03-16T19:31:31.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.345+0000 7fc6bc38e640 1 -- start start 2024-03-16T19:31:31.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.345+0000 7f7707fff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f770814e870 0x7f77081f38b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6bc38e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6b40c33d0 0x7fc6b4151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6bc38e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6b40c97f0 0x7fc6b40b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6bc38e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b4152af0 0x7fc6b41ede60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.333 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7700003a90 con 0x7f770814e870 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6bc38e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b4159380 con 0x7fc6b40c97f0 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6bc38e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b41595b0 con 0x7fc6b4152af0 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6bc38e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b41f13d0 con 0x7fc6b40c33d0 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6b9929640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6b40c97f0 0x7fc6b40b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6b9929640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6b40c97f0 0x7fc6b40b2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34336/0 (socket says 172.21.15.172:34336) 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6b9929640 1 -- 172.21.15.172:0/2242993341 learned_addr learned my addr 172.21.15.172:0/2242993341 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6ba12a640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6b40c33d0 0x7fc6b4151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6ba92b640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b4152af0 0x7fc6b41ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6b9929640 1 -- 172.21.15.172:0/2242993341 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6b40c33d0 msgr2=0x7fc6b4151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6b9929640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6b40c33d0 0x7fc6b4151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6b9929640 1 -- 172.21.15.172:0/2242993341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b4152af0 msgr2=0x7fc6b41ede60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6b9929640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b4152af0 0x7fc6b41ede60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6b9929640 1 -- 172.21.15.172:0/2242993341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a4009400 con 0x7fc6b40c97f0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.346+0000 7fc6ba12a640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6b40c33d0 0x7fc6b4151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.347+0000 7fc6b9929640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6b40c97f0 0x7fc6b40b2dc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009890 tx=0x7fc6a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.347+0000 7fc6aaffd640 1 -- 172.21.15.172:0/2242993341 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a403ba70 con 0x7fc6b40c97f0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.347+0000 7fc6aaffd640 1 -- 172.21.15.172:0/2242993341 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a403bbf0 con 0x7fc6b40c97f0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.347+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b41f16b0 con 0x7fc6b40c97f0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.347+0000 7fc6aaffd640 1 -- 172.21.15.172:0/2242993341 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4045400 con 0x7fc6b40c97f0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.347+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b41f1c70 con 0x7fc6b40c97f0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.348+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b40688d0 con 0x7fc6b40c97f0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.348+0000 7fc6aaffd640 1 -- 172.21.15.172:0/2242993341 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc6a404e440 con 0x7fc6b40c97f0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.349+0000 7fc6aaffd640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc69806b3b0 0x7fc69806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.349+0000 7fc6aaffd640 1 -- 172.21.15.172:0/2242993341 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fc6a40abe30 con 0x7fc6b40c97f0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.349+0000 7fc6ba12a640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc69806b3b0 0x7fc69806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.349+0000 7fc6ba12a640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc69806b3b0 0x7fc69806d870 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0007d90 tx=0x7fc6b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.353+0000 7fc6aaffd640 1 -- 172.21.15.172:0/2242993341 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6a4076aa0 con 0x7fc6b40c97f0 2024-03-16T19:31:31.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.361+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f770815c000 con 0x7f770814e870 2024-03-16T19:31:31.345 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.364+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7700003c00 con 0x7f770814e870 2024-03-16T19:31:31.345 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:31.345 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:31.346 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.364+0000 7f770dad0640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f77080c5fb0 0x7f77081f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.346 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.364+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77080cabb0 con 0x7f77080c5fb0 2024-03-16T19:31:31.346 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.364+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f77080c5fb0 0x7f77081f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.346 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.365+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f77080c5fb0 0x7f77081f4300 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.346 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.365+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77080cabb0 con 0x7f77080c5fb0 2024-03-16T19:31:31.362 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.381+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f77080681c0 con 0x7f77080c5fb0 2024-03-16T19:31:31.363 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.382+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77080681c0 con 0x7f77080c5fb0 2024-03-16T19:31:31.364 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:31.364 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.383+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=78}) v3 -- 0x7f77080ca8c0 con 0x7f77080b3630 2024-03-16T19:31:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.383+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f77080cbbe0 con 0x7f77080b3630 2024-03-16T19:31:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.383+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76f8058360 con 0x7f77080b3630 2024-03-16T19:31:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.383+0000 7f770dad0640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f7708147440 0x7f7708147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.383+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7708147d80 con 0x7f7708147440 2024-03-16T19:31:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.383+0000 7f76fffff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f7708147440 0x7f7708147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.383+0000 7f76fffff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f7708147440 0x7f7708147840 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.384+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7708147d80 con 0x7f7708147440 2024-03-16T19:31:31.379 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.398+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7708061930 con 0x7f7708147440 2024-03-16T19:31:31.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.400+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7708061930 con 0x7f7708147440 2024-03-16T19:31:31.381 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:31.381 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:31.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.400+0000 7f770dad0640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f7708062370 0x7f77080c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.400+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77080627d0 con 0x7f7708062370 2024-03-16T19:31:31.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.400+0000 7f7707fff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f7708062370 0x7f77080c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.401+0000 7f7707fff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f7708062370 0x7f77080c3de0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.383 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.402+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77080627d0 con 0x7f7708062370 2024-03-16T19:31:31.392 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:31.412+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:31.397 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.416+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f77080c45f0 con 0x7f7708062370 2024-03-16T19:31:31.399 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f77080c45f0 con 0x7f7708062370 2024-03-16T19:31:31.400 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f770dad0640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f770802bde0 0x7f770802c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77080c4cd0 con 0x7f770802bde0 2024-03-16T19:31:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f770802bde0 0x7f770802c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f770802bde0 0x7f770802c220 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f77077fe640 1 -- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f770802bde0 msgr2=0x7f770802c220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 25 2024-03-16T19:31:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f77077fe640 1 -- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f770802bde0 msgr2=0x7f770802c220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:31:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f770802bde0 0x7f770802c220 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:31:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f770802bde0 0x7f770802c220 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f770802bde0 msgr2=0x7f770802c220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:31:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f7704ff9640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f770802bde0 0x7f770802c220 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f7704ff9640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f76e8075680 0x7f76e8077a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.419+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77080627d0 con 0x7f76e8075680 2024-03-16T19:31:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.420+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f76e8075680 0x7f76e8077a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.420+0000 7f77077fe640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f76e8075680 0x7f76e8077a70 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.421+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77080627d0 con 0x7f76e8075680 2024-03-16T19:31:31.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.434+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f770802c1c0 con 0x7f76e8075680 2024-03-16T19:31:31.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.436+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f770802c1c0 con 0x7f76e8075680 2024-03-16T19:31:31.417 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:31.418 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:31.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.436+0000 7f770dad0640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f770802c840 0x7f770802cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.436+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f770802d1c0 con 0x7f770802c840 2024-03-16T19:31:31.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.436+0000 7f76fffff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f770802c840 0x7f770802cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.437+0000 7f76fffff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f770802c840 0x7f770802cc80 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.438+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f770802d1c0 con 0x7f770802c840 2024-03-16T19:31:31.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.452+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f770802d1c0 con 0x7f770802c840 2024-03-16T19:31:31.436 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.455+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f770802d1c0 con 0x7f770802c840 2024-03-16T19:31:31.436 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:31.436 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:31.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.455+0000 7f770dad0640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f7708205350 0x7f7708207740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.455+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f770820a510 con 0x7f7708205350 2024-03-16T19:31:31.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.455+0000 7f7707fff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f7708205350 0x7f7708207740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.455+0000 7f7707fff640 1 --2- 172.21.15.172:0/2940987751 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f7708205350 0x7f7708207740 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.437 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.456+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f770820a510 con 0x7f7708205350 2024-03-16T19:31:31.451 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.470+0000 7f770dad0640 1 -- 172.21.15.172:0/2940987751 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7708207c80 con 0x7f7708205350 2024-03-16T19:31:31.452 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.472+0000 7f7704ff9640 1 -- 172.21.15.172:0/2940987751 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7708207c80 con 0x7f7708205350 2024-03-16T19:31:31.453 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:31.453 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:31.457 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:31.459 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:31.459 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:31.467 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:31.486+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:31.468 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:31.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.534+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) v1 -- 0x7fc6b40ce730 con 0x7fc69806b3b0 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.535+0000 7fc6aaffd640 1 -- 172.21.15.172:0/2242993341 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+294 (secure 0 0 0) 0x7fc6b00079e0 con 0x7fc69806b3b0 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:moved 0 / 194 (0%) 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:avg 27.7143 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:stddev 4.02543 -> 4.02543 (expected baseline 4.87392) 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:min osd.1 with 20 -> 20 pgs (0.72165 -> 0.72165 * mean) 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:max osd.7 with 30 -> 30 pgs (1.08247 -> 1.08247 * mean) 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:oload 110 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:max_change 3 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:max_change_osds 4 2024-03-16T19:31:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:average_utilization 0.0003 2024-03-16T19:31:31.517 INFO:teuthology.orchestra.run.smithi172.stdout:overload_utilization 0.0003 2024-03-16T19:31:31.517 INFO:teuthology.orchestra.run.smithi172.stderr:no change 2024-03-16T19:31:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc69806b3b0 msgr2=0x7fc69806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc69806b3b0 0x7fc69806d870 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0007d90 tx=0x7fc6b0008040 comp rx=0 tx=0).stop 2024-03-16T19:31:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6b40c97f0 msgr2=0x7fc6b40b2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6b40c97f0 0x7fc6b40b2dc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009890 tx=0x7fc6a40046c0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 shutdown_connections 2024-03-16T19:31:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b4152af0 0x7fc6b41ede60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6b40c97f0 0x7fc6b40b2dc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc69806b3b0 0x7fc69806d870 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 --2- 172.21.15.172:0/2242993341 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6b40c33d0 0x7fc6b4151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.519 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 >> 172.21.15.172:0/2242993341 conn(0x7fc6b4066c20 msgr2=0x7fc6b40c9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:31.519 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 shutdown_connections 2024-03-16T19:31:31.519 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.537+0000 7fc6bc38e640 1 -- 172.21.15.172:0/2242993341 wait complete. 2024-03-16T19:31:31.529 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-16T19:31:31.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2024-03-16T19:31:31.573 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:31.592+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:31.618 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:31.618 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:31.627 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:31.631 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.650+0000 7f5ef5dd7640 1 Processor -- start 2024-03-16T19:31:31.631 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.650+0000 7f5ef5dd7640 1 -- start start 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.651+0000 7f5ef5dd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef00c7800 0x7f5ef0166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.651+0000 7f5ef5dd7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef015b990 con 0x7f5ef00b5f10 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.651+0000 7f5ef5dd7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef00cc090 con 0x7f5ef0154e00 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.651+0000 7f5ef5dd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef015b570 con 0x7f5ef00c7800 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.651+0000 7f5ee6ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5ef00b5f10 0x7f5ef00caed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56042/0 (socket says 172.21.15.172:56042) 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.651+0000 7f5ee6ffd640 1 -- 172.21.15.172:0/4055651586 learned_addr learned my addr 172.21.15.172:0/4055651586 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.651+0000 7f5eeffff640 1 --2- 172.21.15.172:0/4055651586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef00c7800 0x7f5ef0166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.652+0000 7f5eeeffd640 1 -- 172.21.15.172:0/4055651586 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245012567 0 0) 0x7f5ef015b990 con 0x7f5ef00b5f10 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.652+0000 7f5eeeffd640 1 -- 172.21.15.172:0/4055651586 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5edc003520 con 0x7f5ef00b5f10 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.652+0000 7f5eeeffd640 1 -- 172.21.15.172:0/4055651586 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478187899 0 0) 0x7f5ef00cc090 con 0x7f5ef0154e00 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.652+0000 7f5eeeffd640 1 -- 172.21.15.172:0/4055651586 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ef015b990 con 0x7f5ef0154e00 2024-03-16T19:31:31.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.652+0000 7f5eeffff640 1 -- 172.21.15.172:0/4055651586 >> v1:172.21.15.174:6789/0 conn(0x7f5ef0154e00 legacy=0x7f5ef006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.652+0000 7f5eeffff640 1 -- 172.21.15.172:0/4055651586 >> v1:172.21.15.172:6789/0 conn(0x7f5ef00b5f10 legacy=0x7f5ef00caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.652+0000 7f5eeffff640 1 -- 172.21.15.172:0/4055651586 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef015abd0 con 0x7f5ef00c7800 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5eeffff640 1 --2- 172.21.15.172:0/4055651586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef00c7800 0x7f5ef0166270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009dc0 tx=0x7f5ee802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f321a940815bc05 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5eeeffd640 1 -- 172.21.15.172:0/4055651586 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee802cd50 con 0x7f5ef00c7800 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5eeeffd640 1 -- 172.21.15.172:0/4055651586 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee8034440 con 0x7f5ef00c7800 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5eeeffd640 1 -- 172.21.15.172:0/4055651586 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee80347b0 con 0x7f5ef00c7800 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/4055651586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef00c7800 msgr2=0x7f5ef0166270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5ef5dd7640 1 --2- 172.21.15.172:0/4055651586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef00c7800 0x7f5ef0166270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009dc0 tx=0x7f5ee802bfa0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/4055651586 shutdown_connections 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5ef5dd7640 1 --2- 172.21.15.172:0/4055651586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef00c7800 0x7f5ef0166270 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/4055651586 >> 172.21.15.172:0/4055651586 conn(0x7f5ef0066c20 msgr2=0x7f5ef0067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/4055651586 shutdown_connections 2024-03-16T19:31:31.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.653+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/4055651586 wait complete. 2024-03-16T19:31:31.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.654+0000 7f5ef5dd7640 1 Processor -- start 2024-03-16T19:31:31.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ef5dd7640 1 -- start start 2024-03-16T19:31:31.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ef5dd7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef00b5f10 0x7f5ef0151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ef5dd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef00c7800 0x7f5ef01f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ef5dd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef0154e00 0x7f5ef01f1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ef5dd7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00c4de0 con 0x7f5ef00b5f10 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ef5dd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef01f4fa0 con 0x7f5ef00c7800 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ef5dd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef01f4b10 con 0x7f5ef0154e00 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eef7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef00b5f10 0x7f5ef0151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eef7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef00b5f10 0x7f5ef0151d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34340/0 (socket says 172.21.15.172:34340) 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eef7fe640 1 -- 172.21.15.172:0/2755358207 learned_addr learned my addr 172.21.15.172:0/2755358207 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eeffff640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef0154e00 0x7f5ef01f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ee6ffd640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef00c7800 0x7f5ef01f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eef7fe640 1 -- 172.21.15.172:0/2755358207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef0154e00 msgr2=0x7f5ef01f1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eef7fe640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef0154e00 0x7f5ef01f1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eef7fe640 1 -- 172.21.15.172:0/2755358207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef00c7800 msgr2=0x7f5ef01f0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eef7fe640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef00c7800 0x7f5ef01f0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eef7fe640 1 -- 172.21.15.172:0/2755358207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8009a70 con 0x7f5ef00b5f10 2024-03-16T19:31:31.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eef7fe640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef00b5f10 0x7f5ef0151d20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000c460 tx=0x7f5ee000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eecff9640 1 -- 172.21.15.172:0/2755358207 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00052d0 con 0x7f5ef00b5f10 2024-03-16T19:31:31.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eecff9640 1 -- 172.21.15.172:0/2755358207 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee0005450 con 0x7f5ef00b5f10 2024-03-16T19:31:31.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5eecff9640 1 -- 172.21.15.172:0/2755358207 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0003830 con 0x7f5ef00b5f10 2024-03-16T19:31:31.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef006b850 con 0x7f5ef00b5f10 2024-03-16T19:31:31.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.656+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef00c4b00 con 0x7f5ef00b5f10 2024-03-16T19:31:31.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.657+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef00cf870 con 0x7f5ef00b5f10 2024-03-16T19:31:31.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.657+0000 7f5eecff9640 1 -- 172.21.15.172:0/2755358207 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5ee0003990 con 0x7f5ef00b5f10 2024-03-16T19:31:31.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.659+0000 7f5eecff9640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5ed006b3b0 0x7f5ed006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.659+0000 7f5eecff9640 1 -- 172.21.15.172:0/2755358207 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f5ee0089220 con 0x7f5ef00b5f10 2024-03-16T19:31:31.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.659+0000 7f5ee6ffd640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5ed006b3b0 0x7f5ed006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.659+0000 7f5ee6ffd640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5ed006b3b0 0x7f5ed006d870 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0152f90 tx=0x7f5ed8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.663+0000 7f5eecff9640 1 -- 172.21.15.172:0/2755358207 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ee0053e90 con 0x7f5ef00b5f10 2024-03-16T19:31:31.668 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:31.687+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:31.733 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:31.733 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:31.742 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:31.787 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:31.806+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.834+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}) v1 -- 0x7f5ef006afc0 con 0x7f5ed006b3b0 2024-03-16T19:31:31.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.834+0000 7f5eecff9640 1 -- 172.21.15.172:0/2755358207 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+294 (secure 0 0 0) 0x7f5ef006afc0 con 0x7f5ed006b3b0 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:moved 0 / 194 (0%) 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:avg 27.7143 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:stddev 4.02543 -> 4.02543 (expected baseline 4.87392) 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:min osd.1 with 20 -> 20 pgs (0.72165 -> 0.72165 * mean) 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:max osd.7 with 30 -> 30 pgs (1.08247 -> 1.08247 * mean) 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:oload 110 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:max_change 1 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:max_change_osds 4 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:average_utilization 0.0003 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stdout:overload_utilization 0.0003 2024-03-16T19:31:31.815 INFO:teuthology.orchestra.run.smithi172.stderr:no change 2024-03-16T19:31:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5ed006b3b0 msgr2=0x7f5ed006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5ed006b3b0 0x7f5ed006d870 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0152f90 tx=0x7f5ed8009040 comp rx=0 tx=0).stop 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef00b5f10 msgr2=0x7f5ef0151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef00b5f10 0x7f5ef0151d20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000c460 tx=0x7f5ee000c920 comp rx=0 tx=0).stop 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 shutdown_connections 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5ed006b3b0 0x7f5ed006d870 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ef0154e00 0x7f5ef01f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ef00c7800 0x7f5ef01f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 --2- 172.21.15.172:0/2755358207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ef00b5f10 0x7f5ef0151d20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 >> 172.21.15.172:0/2755358207 conn(0x7f5ef0066c20 msgr2=0x7f5ef00ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 shutdown_connections 2024-03-16T19:31:31.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.836+0000 7f5ef5dd7640 1 -- 172.21.15.172:0/2755358207 wait complete. 2024-03-16T19:31:31.827 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-16T19:31:31.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-03-16T19:31:31.849 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:31.849 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:31.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.877+0000 7fdd8affd640 1 -- 172.21.15.172:0/1126890366 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v78) v1 ==== 87+0+0 (secure 0 0 0) 0x7fdd8c021090 con 0x7fdd9406dc20 2024-03-16T19:31:31.857 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is set 2024-03-16T19:31:31.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdd7806b3b0 msgr2=0x7fdd7806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdd7806b3b0 0x7fdd7806d870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd84036000 tx=0x7fdd84008030 comp rx=0 tx=0).stop 2024-03-16T19:31:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 msgr2=0x7fdd940c4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 0x7fdd940c4e60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00e9b0 tx=0x7fdd8c00ee70 comp rx=0 tx=0).stop 2024-03-16T19:31:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 shutdown_connections 2024-03-16T19:31:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd94157ec0 0x7fdd94158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdd7806b3b0 0x7fdd7806d870 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd940c9080 0x7fdd940cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 --2- 172.21.15.172:0/1126890366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdd9406dc20 0x7fdd940c4e60 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.879+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 >> 172.21.15.172:0/1126890366 conn(0x7fdd94070810 msgr2=0x7fdd940c30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.880+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 shutdown_connections 2024-03-16T19:31:31.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.880+0000 7fdd9c3f4640 1 -- 172.21.15.172:0/1126890366 wait complete. 2024-03-16T19:31:31.872 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:31.892+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:31.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.981+0000 7fa4527e2640 1 Processor -- start 2024-03-16T19:31:31.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.981+0000 7fa4527e2640 1 -- start start 2024-03-16T19:31:31.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.982+0000 7fa4527e2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa44c066f40 0x7fa44c15bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.982+0000 7fa4527e2640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa44c071f50 0x7fa44c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.982+0000 7fa4527e2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c0723e0 con 0x7fa44c154ba0 2024-03-16T19:31:31.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.982+0000 7fa4527e2640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44c15b910 con 0x7fa44c071f50 2024-03-16T19:31:31.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.982+0000 7fa4527e2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44c06df30 con 0x7fa44c066f40 2024-03-16T19:31:31.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.982+0000 7fa450d7f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa44c071f50 0x7fa44c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.982+0000 7fa450d7f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa44c071f50 0x7fa44c15e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34348/0 (socket says 172.21.15.172:34348) 2024-03-16T19:31:31.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.982+0000 7fa450d7f640 1 -- 172.21.15.172:0/2948878361 learned_addr learned my addr 172.21.15.172:0/2948878361 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:31.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.982+0000 7fa44b7fe640 1 --2- 172.21.15.172:0/2948878361 >> v2:172.21.15.174:3300/0 conn(0x7fa44c066f40 0x7fa44c15bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.983+0000 7fa44b7fe640 1 -- 172.21.15.172:0/2948878361 >> v2:172.21.15.172:3300/0 conn(0x7fa44c071f50 msgr2=0x7fa44c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:31.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.983+0000 7fa44b7fe640 1 --2- 172.21.15.172:0/2948878361 >> v2:172.21.15.172:3300/0 conn(0x7fa44c071f50 0x7fa44c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.983+0000 7fa44b7fe640 1 -- 172.21.15.172:0/2948878361 >> v1:172.21.15.174:6789/0 conn(0x7fa44c154ba0 legacy=0x7fa44c15de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.983+0000 7fa44b7fe640 1 -- 172.21.15.172:0/2948878361 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c06d9e0 con 0x7fa44c066f40 2024-03-16T19:31:31.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.983+0000 7fa44affd640 1 -- 172.21.15.172:0/2948878361 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056516313 0 0) 0x7fa44c0723e0 con 0x7fa44c154ba0 2024-03-16T19:31:31.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.983+0000 7fa44b7fe640 1 --2- 172.21.15.172:0/2948878361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa44c066f40 0x7fa44c15bd90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa44c06bf20 tx=0x7fa43c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e5174c2124bb8e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa44affd640 1 -- 172.21.15.172:0/2948878361 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c031830 con 0x7fa44c066f40 2024-03-16T19:31:31.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa44affd640 1 -- 172.21.15.172:0/2948878361 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c031990 con 0x7fa44c066f40 2024-03-16T19:31:31.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa44affd640 1 -- 172.21.15.172:0/2948878361 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c031d20 con 0x7fa44c066f40 2024-03-16T19:31:31.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa4527e2640 1 -- 172.21.15.172:0/2948878361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa44c066f40 msgr2=0x7fa44c15bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa4527e2640 1 --2- 172.21.15.172:0/2948878361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa44c066f40 0x7fa44c15bd90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa44c06bf20 tx=0x7fa43c02b8d0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa4527e2640 1 -- 172.21.15.172:0/2948878361 shutdown_connections 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa4527e2640 1 --2- 172.21.15.172:0/2948878361 >> v2:172.21.15.172:3300/0 conn(0x7fa44c071f50 0x7fa44c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa4527e2640 1 --2- 172.21.15.172:0/2948878361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa44c066f40 0x7fa44c15bd90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa4527e2640 1 -- 172.21.15.172:0/2948878361 >> 172.21.15.172:0/2948878361 conn(0x7fa44c0c9390 msgr2=0x7fa44c0c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa4527e2640 1 -- 172.21.15.172:0/2948878361 shutdown_connections 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.984+0000 7fa4527e2640 1 -- 172.21.15.172:0/2948878361 wait complete. 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.985+0000 7fa4527e2640 1 Processor -- start 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa4527e2640 1 -- start start 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:31.966 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:31.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa4527e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa44c066f40 0x7fa44c149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa4527e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44c071f50 0x7fa44c0c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa4527e2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa44c154ba0 0x7fa44c0caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa4527e2640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0c5970 con 0x7fa44c154ba0 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa4527e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c149580 con 0x7fa44c066f40 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa4527e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0c5ca0 con 0x7fa44c071f50 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa450d7f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa44c154ba0 0x7fa44c0caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa44b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44c071f50 0x7fa44c0c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa450d7f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa44c154ba0 0x7fa44c0caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34350/0 (socket says 172.21.15.172:34350) 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa44bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa44c066f40 0x7fa44c149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.986+0000 7fa450d7f640 1 -- 172.21.15.172:0/1740644038 learned_addr learned my addr 172.21.15.172:0/1740644038 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa44b7fe640 1 -- 172.21.15.172:0/1740644038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa44c066f40 msgr2=0x7fa44c149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa44b7fe640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa44c066f40 0x7fa44c149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa44b7fe640 1 -- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa44c154ba0 msgr2=0x7fa44c0caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa44b7fe640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa44c154ba0 0x7fa44c0caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa44b7fe640 1 -- 172.21.15.172:0/1740644038 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c009400 con 0x7fa44c071f50 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa450d7f640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa44c154ba0 0x7fa44c0caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:31.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa44b7fe640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44c071f50 0x7fa44c0c43e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa44c06e070 tx=0x7fa43c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa448ff9640 1 -- 172.21.15.172:0/1740644038 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c03a8f0 con 0x7fa44c071f50 2024-03-16T19:31:31.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa448ff9640 1 -- 172.21.15.172:0/1740644038 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c03aa90 con 0x7fa44c071f50 2024-03-16T19:31:31.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44c0c5f10 con 0x7fa44c071f50 2024-03-16T19:31:31.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa448ff9640 1 -- 172.21.15.172:0/1740644038 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c03ae40 con 0x7fa44c071f50 2024-03-16T19:31:31.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.987+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44c149210 con 0x7fa44c071f50 2024-03-16T19:31:31.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.988+0000 7fa448ff9640 1 -- 172.21.15.172:0/1740644038 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa43c0443f0 con 0x7fa44c071f50 2024-03-16T19:31:31.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.988+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44c0cd7f0 con 0x7fa44c071f50 2024-03-16T19:31:31.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.989+0000 7fa448ff9640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa43806b3e0 0x7fa43806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:31.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.989+0000 7fa448ff9640 1 -- 172.21.15.172:0/1740644038 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fa43c0abc30 con 0x7fa44c071f50 2024-03-16T19:31:31.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.989+0000 7fa44bfff640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa43806b3e0 0x7fa43806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:31.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.989+0000 7fa44bfff640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa43806b3e0 0x7fa43806d8a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa440000f60 tx=0x7fa440009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:31.971 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:31.990+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:31.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:31.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:31.998+0000 7fa448ff9640 1 -- 172.21.15.172:0/1740644038 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa43c076890 con 0x7fa44c071f50 2024-03-16T19:31:32.071 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:32.091+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:32.083 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:32.083 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:32.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:32.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.176+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}) v1 -- 0x7fa44c071750 con 0x7fa43806b3e0 2024-03-16T19:31:32.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.177+0000 7fa448ff9640 1 -- 172.21.15.172:0/1740644038 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+295 (secure 0 0 0) 0x7fa44000f820 con 0x7fa43806b3e0 2024-03-16T19:31:32.157 INFO:teuthology.orchestra.run.smithi172.stdout:moved 0 / 194 (0%) 2024-03-16T19:31:32.157 INFO:teuthology.orchestra.run.smithi172.stdout:avg 27.7143 2024-03-16T19:31:32.157 INFO:teuthology.orchestra.run.smithi172.stdout:stddev 4.02543 -> 4.02543 (expected baseline 4.87392) 2024-03-16T19:31:32.157 INFO:teuthology.orchestra.run.smithi172.stdout:min osd.1 with 20 -> 20 pgs (0.72165 -> 0.72165 * mean) 2024-03-16T19:31:32.157 INFO:teuthology.orchestra.run.smithi172.stdout:max osd.7 with 30 -> 30 pgs (1.08247 -> 1.08247 * mean) 2024-03-16T19:31:32.157 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:31:32.157 INFO:teuthology.orchestra.run.smithi172.stdout:oload 1000 2024-03-16T19:31:32.157 INFO:teuthology.orchestra.run.smithi172.stdout:max_change 3 2024-03-16T19:31:32.158 INFO:teuthology.orchestra.run.smithi172.stdout:max_change_osds 4 2024-03-16T19:31:32.158 INFO:teuthology.orchestra.run.smithi172.stdout:average_utilization 0.0003 2024-03-16T19:31:32.158 INFO:teuthology.orchestra.run.smithi172.stdout:overload_utilization 0.0030 2024-03-16T19:31:32.158 INFO:teuthology.orchestra.run.smithi172.stderr:no change 2024-03-16T19:31:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa43806b3e0 msgr2=0x7fa43806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa43806b3e0 0x7fa43806d8a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa440000f60 tx=0x7fa440009040 comp rx=0 tx=0).stop 2024-03-16T19:31:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44c071f50 msgr2=0x7fa44c0c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44c071f50 0x7fa44c0c43e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa44c06e070 tx=0x7fa43c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 shutdown_connections 2024-03-16T19:31:32.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa44c154ba0 0x7fa44c0caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa43806b3e0 0x7fa43806d8a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44c071f50 0x7fa44c0c43e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 --2- 172.21.15.172:0/1740644038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa44c066f40 0x7fa44c149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 >> 172.21.15.172:0/1740644038 conn(0x7fa44c0c9390 msgr2=0x7fa44c15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 shutdown_connections 2024-03-16T19:31:32.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.179+0000 7fa4527e2640 1 -- 172.21.15.172:0/1740644038 wait complete. 2024-03-16T19:31:32.171 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-16T19:31:32.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-03-16T19:31:32.172 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:32.191+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:32.205 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:32.205 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:32.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:32.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.296+0000 7fd971bd7640 1 Processor -- start 2024-03-16T19:31:32.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.296+0000 7fd971bd7640 1 -- start start 2024-03-16T19:31:32.276 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:32.296+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:32.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd971bd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c154e70 0x7fd96c0c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:32.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd971bd7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c15df50 con 0x7fd96c154a90 2024-03-16T19:31:32.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd971bd7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c15e1f0 con 0x7fd96c0cc280 2024-03-16T19:31:32.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd971bd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c15e4d0 con 0x7fd96c154e70 2024-03-16T19:31:32.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd96b7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fd96c154a90 0x7fd96c06b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56054/0 (socket says 172.21.15.172:56054) 2024-03-16T19:31:32.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd96b7fe640 1 -- 172.21.15.172:0/3221043182 learned_addr learned my addr 172.21.15.172:0/3221043182 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:32.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd962ffd640 1 --2- 172.21.15.172:0/3221043182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c154e70 0x7fd96c0c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:32.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd96affd640 1 -- 172.21.15.172:0/3221043182 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543017293 0 0) 0x7fd96c15df50 con 0x7fd96c154a90 2024-03-16T19:31:32.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd96affd640 1 -- 172.21.15.172:0/3221043182 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd958003520 con 0x7fd96c154a90 2024-03-16T19:31:32.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd962ffd640 1 -- 172.21.15.172:0/3221043182 >> v1:172.21.15.174:6789/0 conn(0x7fd96c0cc280 legacy=0x7fd96c0cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:32.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd962ffd640 1 -- 172.21.15.172:0/3221043182 >> v1:172.21.15.172:6789/0 conn(0x7fd96c154a90 legacy=0x7fd96c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:32.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd96affd640 1 -- 172.21.15.172:0/3221043182 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405703714 0 0) 0x7fd96c15e1f0 con 0x7fd96c0cc280 2024-03-16T19:31:32.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.297+0000 7fd962ffd640 1 -- 172.21.15.172:0/3221043182 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c15e7b0 con 0x7fd96c154e70 2024-03-16T19:31:32.279 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.298+0000 7fd962ffd640 1 --2- 172.21.15.172:0/3221043182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c154e70 0x7fd96c0c50b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd9540097d0 tx=0x7fd95402b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76408e3ab7730f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:32.279 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.298+0000 7fd96affd640 1 -- 172.21.15.172:0/3221043182 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954030890 con 0x7fd96c154e70 2024-03-16T19:31:32.279 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.298+0000 7fd96affd640 1 -- 172.21.15.172:0/3221043182 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9540309f0 con 0x7fd96c154e70 2024-03-16T19:31:32.279 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.298+0000 7fd96affd640 1 -- 172.21.15.172:0/3221043182 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954030d80 con 0x7fd96c154e70 2024-03-16T19:31:32.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.299+0000 7fd971bd7640 1 -- 172.21.15.172:0/3221043182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c154e70 msgr2=0x7fd96c0c50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:32.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.299+0000 7fd971bd7640 1 --2- 172.21.15.172:0/3221043182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c154e70 0x7fd96c0c50b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd9540097d0 tx=0x7fd95402b9d0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.299+0000 7fd971bd7640 1 -- 172.21.15.172:0/3221043182 shutdown_connections 2024-03-16T19:31:32.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.299+0000 7fd971bd7640 1 --2- 172.21.15.172:0/3221043182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c154e70 0x7fd96c0c50b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.299+0000 7fd971bd7640 1 -- 172.21.15.172:0/3221043182 >> 172.21.15.172:0/3221043182 conn(0x7fd96c067250 msgr2=0x7fd96c0d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:32.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.299+0000 7fd971bd7640 1 -- 172.21.15.172:0/3221043182 shutdown_connections 2024-03-16T19:31:32.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.299+0000 7fd971bd7640 1 -- 172.21.15.172:0/3221043182 wait complete. 2024-03-16T19:31:32.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.300+0000 7fd971bd7640 1 Processor -- start 2024-03-16T19:31:32.280 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.300+0000 7fd971bd7640 1 -- start start 2024-03-16T19:31:32.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd971bd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c0cc280 0x7fd96c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:32.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd971bd7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c154a90 0x7fd96c153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:32.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd971bd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c154e70 0x7fd96c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:32.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd971bd7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c154020 con 0x7fd96c154a90 2024-03-16T19:31:32.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd971bd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c146ac0 con 0x7fd96c154e70 2024-03-16T19:31:32.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd971bd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c15c630 con 0x7fd96c0cc280 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd962ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c154a90 0x7fd96c153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd96b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c0cc280 0x7fd96c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd962ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c154a90 0x7fd96c153500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34362/0 (socket says 172.21.15.172:34362) 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd962ffd640 1 -- 172.21.15.172:0/2206682276 learned_addr learned my addr 172.21.15.172:0/2206682276 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.301+0000 7fd96bfff640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c154e70 0x7fd96c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd96b7fe640 1 -- 172.21.15.172:0/2206682276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c154e70 msgr2=0x7fd96c0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd96b7fe640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c154e70 0x7fd96c0b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd96b7fe640 1 -- 172.21.15.172:0/2206682276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c154a90 msgr2=0x7fd96c153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd96b7fe640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c154a90 0x7fd96c153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd96b7fe640 1 -- 172.21.15.172:0/2206682276 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd954009400 con 0x7fd96c0cc280 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd96b7fe640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c0cc280 0x7fd96c15d460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00c400 tx=0x7fd95c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd968ff9640 1 -- 172.21.15.172:0/2206682276 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c005ce0 con 0x7fd96c0cc280 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd968ff9640 1 -- 172.21.15.172:0/2206682276 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c005e40 con 0x7fd96c0cc280 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c15c910 con 0x7fd96c0cc280 2024-03-16T19:31:32.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd968ff9640 1 -- 172.21.15.172:0/2206682276 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0072f0 con 0x7fd96c0cc280 2024-03-16T19:31:32.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.302+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c153a40 con 0x7fd96c0cc280 2024-03-16T19:31:32.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.303+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96c0ca860 con 0x7fd96c0cc280 2024-03-16T19:31:32.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.303+0000 7fd968ff9640 1 -- 172.21.15.172:0/2206682276 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd95c007450 con 0x7fd96c0cc280 2024-03-16T19:31:32.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.304+0000 7fd968ff9640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd94c06b3b0 0x7fd94c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:32.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.304+0000 7fd968ff9640 1 -- 172.21.15.172:0/2206682276 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fd95c0891c0 con 0x7fd96c0cc280 2024-03-16T19:31:32.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.304+0000 7fd962ffd640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd94c06b3b0 0x7fd94c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:32.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.304+0000 7fd962ffd640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd94c06b3b0 0x7fd94c06d870 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd95402ba10 tx=0x7fd954008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:32.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.308+0000 7fd968ff9640 1 -- 172.21.15.172:0/2206682276 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd95c053e30 con 0x7fd96c0cc280 2024-03-16T19:31:32.364 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:32.364 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:32.373 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:32.374 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:32.392+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:32.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.486+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}) v1 -- 0x7fd96c15ed70 con 0x7fd94c06b3b0 2024-03-16T19:31:32.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.487+0000 7fd968ff9640 1 -- 172.21.15.172:0/2206682276 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+294 (secure 0 0 0) 0x7fd95403f040 con 0x7fd94c06b3b0 2024-03-16T19:31:32.467 INFO:teuthology.orchestra.run.smithi172.stdout:moved 0 / 194 (0%) 2024-03-16T19:31:32.467 INFO:teuthology.orchestra.run.smithi172.stdout:avg 27.7143 2024-03-16T19:31:32.467 INFO:teuthology.orchestra.run.smithi172.stdout:stddev 4.02543 -> 4.02543 (expected baseline 4.87392) 2024-03-16T19:31:32.467 INFO:teuthology.orchestra.run.smithi172.stdout:min osd.1 with 20 -> 20 pgs (0.72165 -> 0.72165 * mean) 2024-03-16T19:31:32.467 INFO:teuthology.orchestra.run.smithi172.stdout:max osd.7 with 30 -> 30 pgs (1.08247 -> 1.08247 * mean) 2024-03-16T19:31:32.467 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:31:32.467 INFO:teuthology.orchestra.run.smithi172.stdout:oload 110 2024-03-16T19:31:32.468 INFO:teuthology.orchestra.run.smithi172.stdout:max_change 1 2024-03-16T19:31:32.468 INFO:teuthology.orchestra.run.smithi172.stdout:max_change_osds 4 2024-03-16T19:31:32.468 INFO:teuthology.orchestra.run.smithi172.stdout:average_utilization 0.0003 2024-03-16T19:31:32.468 INFO:teuthology.orchestra.run.smithi172.stdout:overload_utilization 0.0003 2024-03-16T19:31:32.468 INFO:teuthology.orchestra.run.smithi172.stderr:no change 2024-03-16T19:31:32.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.489+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd94c06b3b0 msgr2=0x7fd94c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:32.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd94c06b3b0 0x7fd94c06d870 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd95402ba10 tx=0x7fd954008030 comp rx=0 tx=0).stop 2024-03-16T19:31:32.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c0cc280 msgr2=0x7fd96c15d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:32.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c0cc280 0x7fd96c15d460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00c400 tx=0x7fd95c00c8c0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 shutdown_connections 2024-03-16T19:31:32.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c154e70 0x7fd96c0b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd96c154a90 0x7fd96c153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fd94c06b3b0 0x7fd94c06d870 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 --2- 172.21.15.172:0/2206682276 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd96c0cc280 0x7fd96c15d460 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:32.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 >> 172.21.15.172:0/2206682276 conn(0x7fd96c067250 msgr2=0x7fd96c0cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:32.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 shutdown_connections 2024-03-16T19:31:32.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:32.490+0000 7fd971bd7640 1 -- 172.21.15.172:0/2206682276 wait complete. 2024-03-16T19:31:32.474 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:32.493+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:32.522 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:32.523 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:32.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:32.576 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:32.596+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:32.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:32.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:32.695+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:32.776 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:32.796+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:32.795 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:32.795 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:32.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:32.877 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:32.896+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:32.909 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:32.909 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:32.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:32.978 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:32.998+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:33.025 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:33.025 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:33.034 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:33.079 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:33.099+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:33.141 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:33.141 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:33.150 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:33.194 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:33.213+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:33.260 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:33.260 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:33.269 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:33.311 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:33.330+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:33.382 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:33.401+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:33.418 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:33.418 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:33.426 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:33.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:31:33.511 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:33.530+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:33.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.581+0000 7fdb1bdc7640 1 Processor -- start 2024-03-16T19:31:33.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.582+0000 7fdb1bdc7640 1 -- start start 2024-03-16T19:31:33.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.582+0000 7fdb1bdc7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdb140c1150 0x7fdb140c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.582+0000 7fdb1bdc7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 0x7fdb1406cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb1bdc7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb140c8860 con 0x7fdb1415b030 2024-03-16T19:31:33.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb1bdc7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb14151600 con 0x7fdb140c1150 2024-03-16T19:31:33.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb1bdc7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1415a360 con 0x7fdb140c8410 2024-03-16T19:31:33.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb1a364640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 0x7fdb1406cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb19362640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdb140c1150 0x7fdb140c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb1a364640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 0x7fdb1406cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:38790/0 (socket says 172.21.15.172:38790) 2024-03-16T19:31:33.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb19b63640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fdb1415b030 0x7fdb14157590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:59096/0 (socket says 172.21.15.172:59096) 2024-03-16T19:31:33.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb1a364640 1 -- 172.21.15.172:0/3263423934 learned_addr learned my addr 172.21.15.172:0/3263423934 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:33.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb19362640 1 -- 172.21.15.172:0/3263423934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 msgr2=0x7fdb1406cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:33.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb19362640 1 --2- 172.21.15.172:0/3263423934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 0x7fdb1406cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb19362640 1 -- 172.21.15.172:0/3263423934 >> v1:172.21.15.174:6789/0 conn(0x7fdb1415b030 legacy=0x7fdb14157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:33.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb19362640 1 -- 172.21.15.172:0/3263423934 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb14156df0 con 0x7fdb140c1150 2024-03-16T19:31:33.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.583+0000 7fdb1a364640 1 --2- 172.21.15.172:0/3263423934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 0x7fdb1406cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:33.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.584+0000 7fdb19362640 1 --2- 172.21.15.172:0/3263423934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb140c1150 0x7fdb140c9650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdb080097a0 tx=0x7fdb0802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72d787e47cd8f398 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.584+0000 7fdb18b61640 1 -- 172.21.15.172:0/3263423934 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08031830 con 0x7fdb140c1150 2024-03-16T19:31:33.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.584+0000 7fdb18b61640 1 -- 172.21.15.172:0/3263423934 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb08031990 con 0x7fdb140c1150 2024-03-16T19:31:33.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.584+0000 7fdb18b61640 1 -- 172.21.15.172:0/3263423934 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08031d20 con 0x7fdb140c1150 2024-03-16T19:31:33.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.584+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/3263423934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb140c1150 msgr2=0x7fdb140c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:33.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.584+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/3263423934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb140c1150 0x7fdb140c9650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdb080097a0 tx=0x7fdb0802b8d0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.585+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/3263423934 shutdown_connections 2024-03-16T19:31:33.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.585+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/3263423934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 0x7fdb1406cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.585+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/3263423934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb140c1150 0x7fdb140c9650 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.585+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/3263423934 >> 172.21.15.172:0/3263423934 conn(0x7fdb140c8b30 msgr2=0x7fdb140c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:33.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.585+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/3263423934 shutdown_connections 2024-03-16T19:31:33.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.585+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/3263423934 wait complete. 2024-03-16T19:31:33.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb1bdc7640 1 Processor -- start 2024-03-16T19:31:33.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb1bdc7640 1 -- start start 2024-03-16T19:31:33.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb1bdc7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb140c1150 0x7fdb14155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb1bdc7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 0x7fdb140ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb1bdc7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb1415b030 0x7fdb1413d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb1bdc7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb14149d90 con 0x7fdb140c1150 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb1bdc7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb140cbe00 con 0x7fdb1415b030 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb1bdc7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb14149710 con 0x7fdb140c8410 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb19b63640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb140c1150 0x7fdb14155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb19b63640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb140c1150 0x7fdb14155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34378/0 (socket says 172.21.15.172:34378) 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.586+0000 7fdb19b63640 1 -- 172.21.15.172:0/1712391241 learned_addr learned my addr 172.21.15.172:0/1712391241 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 0x7fdb140ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb1a364640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb1415b030 0x7fdb1413d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb19b63640 1 -- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 msgr2=0x7fdb140ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb19b63640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb140c8410 0x7fdb140ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb19b63640 1 -- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb1415b030 msgr2=0x7fdb1413d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb19b63640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb1415b030 0x7fdb1413d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb19b63640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb040094d0 con 0x7fdb140c1150 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb19b63640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb140c1150 0x7fdb14155c50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00c730 tx=0x7fdb0c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00ce40 con 0x7fdb140c1150 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0c00a640 con 0x7fdb140c1150 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb08009400 con 0x7fdb140c1150 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00aa70 con 0x7fdb140c1150 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.587+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb140cbb20 con 0x7fdb140c1150 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.588+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdb140681c0 con 0x7fdb140c1150 2024-03-16T19:31:33.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.589+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fdb0c00abd0 con 0x7fdb140c1150 2024-03-16T19:31:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.589+0000 7fdb127fc640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdb0006b3b0 0x7fdb0006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.589+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdb0006b3b0 0x7fdb0006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.589+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fdb0c089200 con 0x7fdb140c1150 2024-03-16T19:31:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.589+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v78) v1 ==== 54+0+16 (secure 0 0 0) 0x7fdb0c090070 con 0x7fdb140c1150 2024-03-16T19:31:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.589+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fdb140cb1f0 0x7fdb14147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.589+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb14072920 con 0x7fdb140cb1f0 2024-03-16T19:31:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.589+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdb0006b3b0 0x7fdb0006d870 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb08036000 tx=0x7fdb0803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.589+0000 7fdb1a364640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fdb140cb1f0 0x7fdb14147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.590+0000 7fdb1a364640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fdb140cb1f0 0x7fdb14147b50 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.590+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb04009e40 con 0x7fdb140cb1f0 2024-03-16T19:31:33.575 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:33.575 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:33.584 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:33.584 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:33.604+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:33.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.605+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdb14072640 con 0x7fdb140cb1f0 2024-03-16T19:31:33.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.607+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb04004040 con 0x7fdb140cb1f0 2024-03-16T19:31:33.587 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:33.587 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:33.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.607+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fdb14158730 0x7fdb141f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.607+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb140c8860 con 0x7fdb14158730 2024-03-16T19:31:33.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.607+0000 7fdb19b63640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fdb14158730 0x7fdb141f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.607+0000 7fdb19b63640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fdb14158730 0x7fdb141f1ce0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.608+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb140c8860 con 0x7fdb14158730 2024-03-16T19:31:33.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.623+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdb14149ed0 con 0x7fdb14158730 2024-03-16T19:31:33.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.624+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb14149ed0 con 0x7fdb14158730 2024-03-16T19:31:33.604 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:33.606 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:33.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.624+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=79}) v3 -- 0x7fdb1413e1d0 con 0x7fdb140c1150 2024-03-16T19:31:33.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.624+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb140c7aa0 con 0x7fdb140c1150 2024-03-16T19:31:33.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.624+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb0c0577a0 con 0x7fdb140c1150 2024-03-16T19:31:33.606 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:33.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb14149850 0x7fdb14061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb140c81f0 con 0x7fdb14149850 2024-03-16T19:31:33.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb14149850 0x7fdb14061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb14149850 0x7fdb14061930 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb19362640 1 -- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb14149850 msgr2=0x7fdb14061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 23 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb19362640 1 -- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb14149850 msgr2=0x7fdb14061930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb14149850 0x7fdb14061930 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb14149850 0x7fdb14061930 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb14149850 msgr2=0x7fdb14061930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb127fc640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb14149850 0x7fdb14061930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb127fc640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb000752c0 0x7fdb000776b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb14149ed0 con 0x7fdb000752c0 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.625+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb000752c0 0x7fdb000776b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.626+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7fdb000752c0 0x7fdb000776b0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.626+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb14149ed0 con 0x7fdb000752c0 2024-03-16T19:31:33.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.640+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdb141406a0 con 0x7fdb000752c0 2024-03-16T19:31:33.622 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.642+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb141406a0 con 0x7fdb000752c0 2024-03-16T19:31:33.622 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:33.623 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:33.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.642+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdb140c6c10 0x7fdb140c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.642+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb140c1660 con 0x7fdb140c6c10 2024-03-16T19:31:33.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.642+0000 7fdb1a364640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdb140c6c10 0x7fdb140c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.643+0000 7fdb1a364640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fdb140c6c10 0x7fdb140c6ff0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.643+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb140c1660 con 0x7fdb140c6c10 2024-03-16T19:31:33.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.657+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdb14061930 con 0x7fdb140c6c10 2024-03-16T19:31:33.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.660+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb14061930 con 0x7fdb140c6c10 2024-03-16T19:31:33.640 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:33.641 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:33.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.660+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fdb14062180 0x7fdb1402bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.660+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb140625c0 con 0x7fdb14062180 2024-03-16T19:31:33.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.660+0000 7fdb19b63640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fdb14062180 0x7fdb1402bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.661+0000 7fdb19b63640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7fdb14062180 0x7fdb1402bde0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.661+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb140625c0 con 0x7fdb14062180 2024-03-16T19:31:33.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.675+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdb140625c0 con 0x7fdb14062180 2024-03-16T19:31:33.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.678+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb140625c0 con 0x7fdb14062180 2024-03-16T19:31:33.658 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:33.658 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:33.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.678+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdb1402c830 0x7fdb1402cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.678+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb1402d250 con 0x7fdb1402c830 2024-03-16T19:31:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.678+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdb1402c830 0x7fdb1402cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.679+0000 7fdb19362640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7fdb1402c830 0x7fdb1402cc90 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.659 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.679+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb1402d250 con 0x7fdb1402c830 2024-03-16T19:31:33.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.693+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdb1402d1d0 con 0x7fdb1402c830 2024-03-16T19:31:33.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.695+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb1402d1d0 con 0x7fdb1402c830 2024-03-16T19:31:33.675 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:33.676 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:33.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.695+0000 7fdb1bdc7640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fdb14203060 0x7fdb14205450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.695+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb14208220 con 0x7fdb14203060 2024-03-16T19:31:33.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.695+0000 7fdb1a364640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fdb14203060 0x7fdb14205450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.696+0000 7fdb1a364640 1 --2- 172.21.15.172:0/1712391241 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fdb14203060 0x7fdb14205450 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.697+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb14208220 con 0x7fdb14203060 2024-03-16T19:31:33.685 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:33.704+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:33.690 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.710+0000 7fdb1bdc7640 1 -- 172.21.15.172:0/1712391241 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdb14205990 con 0x7fdb14203060 2024-03-16T19:31:33.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.713+0000 7fdb127fc640 1 -- 172.21.15.172:0/1712391241 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb14205990 con 0x7fdb14203060 2024-03-16T19:31:33.693 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:33.693 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:33.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:33.732 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:33.732 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:33.732 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:33.732 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:33.733 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:33.741 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:33.826 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:33.845+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:33.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-16T19:31:33.886 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:33.906+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 122.552928353, 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:33.891 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 122.552849235, 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:33.895 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.898 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 122.552235111, 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:33.899 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:31:33.902 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 122.55179504199999, 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:33.903 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.904 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.905 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 122.55155345199999, 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:31:33.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:31:33.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:33.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:31:33.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:31:33.911 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.911 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:33.911 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.911 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:33.911 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:33.911 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:33.912 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:33.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.993+0000 7f1f07b81640 1 Processor -- start 2024-03-16T19:31:33.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.993+0000 7f1f07b81640 1 -- start start 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f07b81640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1f00150ce0 0x7f1f000c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f07b81640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f00156dc0 0x7f1f0006dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f07b81640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f000c2230 0x7f1f000c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f07b81640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f000c1da0 con 0x7f1f00156dc0 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f07b81640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f00157370 con 0x7f1f00150ce0 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f07b81640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f00157650 con 0x7f1f000c2230 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f0511c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f00156dc0 0x7f1f0006dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f0511c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1f00156dc0 0x7f1f0006dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34390/0 (socket says 172.21.15.172:34390) 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f0511c640 1 -- 172.21.15.172:0/1411499454 learned_addr learned my addr 172.21.15.172:0/1411499454 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f0611e640 1 --2- 172.21.15.172:0/1411499454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f000c2230 0x7f1f000c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.994+0000 7f1f0591d640 1 --2- 172.21.15.172:0/1411499454 >> v2:172.21.15.174:3300/0 conn(0x7f1f00150ce0 0x7f1f000c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.995+0000 7f1f0511c640 1 -- 172.21.15.172:0/1411499454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f000c2230 msgr2=0x7f1f000c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:33.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.995+0000 7f1f0511c640 1 --2- 172.21.15.172:0/1411499454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f000c2230 0x7f1f000c26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.995+0000 7f1f0511c640 1 -- 172.21.15.172:0/1411499454 >> v2:172.21.15.174:3300/0 conn(0x7f1f00150ce0 msgr2=0x7f1f000c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:33.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.995+0000 7f1f0511c640 1 --2- 172.21.15.172:0/1411499454 >> v2:172.21.15.174:3300/0 conn(0x7f1f00150ce0 0x7f1f000c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.995+0000 7f1f0511c640 1 -- 172.21.15.172:0/1411499454 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00157930 con 0x7f1f00156dc0 2024-03-16T19:31:33.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.995+0000 7f1f0511c640 1 --2- 172.21.15.172:0/1411499454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f00156dc0 0x7f1f0006dbd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1f000c8b00 tx=0x7f1ef002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feac254db4dd89ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.995+0000 7f1f0491b640 1 -- 172.21.15.172:0/1411499454 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0030860 con 0x7f1f00156dc0 2024-03-16T19:31:33.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.995+0000 7f1f0491b640 1 -- 172.21.15.172:0/1411499454 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef00309c0 con 0x7f1f00156dc0 2024-03-16T19:31:33.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.995+0000 7f1f0491b640 1 -- 172.21.15.172:0/1411499454 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0030d50 con 0x7f1f00156dc0 2024-03-16T19:31:33.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.996+0000 7f1f07b81640 1 -- 172.21.15.172:0/1411499454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f00156dc0 msgr2=0x7f1f0006dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:33.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.996+0000 7f1f07b81640 1 --2- 172.21.15.172:0/1411499454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f00156dc0 0x7f1f0006dbd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1f000c8b00 tx=0x7f1ef002b9a0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.996+0000 7f1f07b81640 1 -- 172.21.15.172:0/1411499454 shutdown_connections 2024-03-16T19:31:33.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.996+0000 7f1f07b81640 1 --2- 172.21.15.172:0/1411499454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f000c2230 0x7f1f000c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.996+0000 7f1f07b81640 1 --2- 172.21.15.172:0/1411499454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f00156dc0 0x7f1f0006dbd0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.996+0000 7f1f07b81640 1 --2- 172.21.15.172:0/1411499454 >> v2:172.21.15.174:3300/0 conn(0x7f1f00150ce0 0x7f1f000c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.996+0000 7f1f07b81640 1 -- 172.21.15.172:0/1411499454 >> 172.21.15.172:0/1411499454 conn(0x7f1f000cc030 msgr2=0x7f1f000cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:33.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.996+0000 7f1f07b81640 1 -- 172.21.15.172:0/1411499454 shutdown_connections 2024-03-16T19:31:33.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.996+0000 7f1f07b81640 1 -- 172.21.15.172:0/1411499454 wait complete. 2024-03-16T19:31:33.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.997+0000 7f1f07b81640 1 Processor -- start 2024-03-16T19:31:33.978 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.997+0000 7f1f07b81640 1 -- start start 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f07b81640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f000c2230 0x7f1f0013e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f07b81640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f00150ce0 0x7f1f0014e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f07b81640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f00156dc0 0x7f1f001f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f07b81640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f00154e10 con 0x7f1f000c2230 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f07b81640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0014fae0 con 0x7f1f00156dc0 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f07b81640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f00155740 con 0x7f1f00150ce0 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0591d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f000c2230 0x7f1f0013e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0591d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f000c2230 0x7f1f0013e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34392/0 (socket says 172.21.15.172:34392) 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0591d640 1 -- 172.21.15.172:0/834711556 learned_addr learned my addr 172.21.15.172:0/834711556 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0511c640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f00150ce0 0x7f1f0014e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0611e640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f00156dc0 0x7f1f001f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0591d640 1 -- 172.21.15.172:0/834711556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f00150ce0 msgr2=0x7f1f0014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0591d640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f00150ce0 0x7f1f0014e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0591d640 1 -- 172.21.15.172:0/834711556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f00156dc0 msgr2=0x7f1f001f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0591d640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f00156dc0 0x7f1f001f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0591d640 1 -- 172.21.15.172:0/834711556 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef00094d0 con 0x7f1f000c2230 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.998+0000 7f1f0591d640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f000c2230 0x7f1f0013e4f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00efc0 tx=0x7f1efc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.999+0000 7f1ef67fc640 1 -- 172.21.15.172:0/834711556 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc014480 con 0x7f1f000c2230 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.999+0000 7f1ef67fc640 1 -- 172.21.15.172:0/834711556 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc00f040 con 0x7f1f000c2230 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.999+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0014b040 con 0x7f1f000c2230 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.999+0000 7f1ef67fc640 1 -- 172.21.15.172:0/834711556 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc014800 con 0x7f1f000c2230 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:33.999+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f00156b80 con 0x7f1f000c2230 2024-03-16T19:31:33.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.000+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f00075a70 con 0x7f1f000c2230 2024-03-16T19:31:33.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.000+0000 7f1ef67fc640 1 -- 172.21.15.172:0/834711556 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1efc014960 con 0x7f1f000c2230 2024-03-16T19:31:33.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.001+0000 7f1ef67fc640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1ee406b3b0 0x7f1ee406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:33.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.001+0000 7f1f0511c640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1ee406b3b0 0x7f1ee406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:33.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.001+0000 7f1ef67fc640 1 -- 172.21.15.172:0/834711556 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f1efc08a7c0 con 0x7f1f000c2230 2024-03-16T19:31:33.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.001+0000 7f1f0511c640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1ee406b3b0 0x7f1ee406d870 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0036000 tx=0x7f1ef0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:33.985 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.005+0000 7f1ef67fc640 1 -- 172.21.15.172:0/834711556 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1efc0553a0 con 0x7f1f000c2230 2024-03-16T19:31:33.995 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:34.014+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:34.072 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:34.072 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:34.072 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:34.072 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:34.083 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:34.125 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:34.144+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:34.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.185+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f1f000c8080 con 0x7f1f000c2230 2024-03-16T19:31:34.189 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:34.209+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:34.230 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:34.230 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:34.231 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:34.231 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:34.231 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:34.239 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:34.290 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:34.310+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:34.351 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:34.351 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:34.351 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:34.351 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:34.351 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.351 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:34.351 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:34.351 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.012889328, 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.352 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:31:34.353 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:31:34.354 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:31:34.355 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:34.356 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.012865155, 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.357 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.360 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:34.361 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.012859328, 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:34.362 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:31:34.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:31:34.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:31:34.365 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.01285541599999, 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:34.367 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.370 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.371 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.012849434, 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.372 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.373 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.01284576800001, 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:31:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:31:34.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:31:34.380 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.381 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.012835108, 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:34.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:31:34.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.01283121900001, 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:34.385 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:31:34.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.00924409300001, 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.389 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:31:34.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.00886174599999, 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:31:34.393 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.394 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:31:34.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:31:34.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 123.004426792, 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.398 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:31:34.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:39 2.6 e21/18 2:654134d2:::mds0_openfiles.0:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:34.402 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 117.99856602600001, 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086850899999999995, 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371542+0000", 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371543+0000", 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0750000000000001e-06 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371552+0000", 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6130000000000004e-06 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371566+0000", 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3635000000000001e-05 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371589+0000", 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3625000000000002e-05 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371671+0000", 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1316999999999994e-05 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.371691+0000", 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0729000000000001e-05 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372373+0000", 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068143299999999995 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:36.372411+0000", 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8081999999999999e-05 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 114.566014818, 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:31:34.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:31:34.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.413 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 112.99852111200001, 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:31:34.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:31:34.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:31:34.418 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 111.49090886099999, 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:31:34.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:31:34.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:31:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.423 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:46 2.6 e36/18 2:654134d2:::mds0_openfiles.0:head v 36'3, mlcod=21'1)", 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 74.994045635999996, 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059498600000000004, 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376063+0000", 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.99e-07 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376070+0000", 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1060000000000002e-06 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376081+0000", 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0312999999999999e-05 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376087+0000", 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376133+0000", 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5985000000000001e-05 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376148+0000", 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5345999999999998e-05 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376623+0000", 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047476799999999998 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:19.376658+0000", 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4688999999999999e-05 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.427 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 68.993509720000006, 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:31:34.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:48 2.6 e77/18 2:654134d2:::mds0_openfiles.0:head v 77'5, mlcod=36'3)", 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:29.382337+0000", 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.9877711189999996, 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00062656099999999998, 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382337+0000", 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382337+0000", 2024-03-16T19:31:34.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382337+0000", 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5800000000000001e-07 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382342+0000", 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382351+0000", 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1409999999999994e-06 2024-03-16T19:31:34.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382358+0000", 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7479999999999996e-06 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382402+0000", 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4070999999999999e-05 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382419+0000", 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6912e-05 2024-03-16T19:31:34.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382933+0000", 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00051417200000000002 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:29.382964+0000", 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0768000000000001e-05 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:49 2.14 2:2e2fa760:::200.00000001:head [write 5990~7422 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e77)", 2024-03-16T19:31:34.435 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:31.372820+0000", 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 2.9972881579999999, 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001781929, 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372820+0000", 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372820+0000", 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372822+0000", 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1959999999999998e-06 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372849+0000", 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6591e-05 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372885+0000", 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5865999999999998e-05 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372896+0000", 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1212e-05 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372960+0000", 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3788999999999995e-05 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373030+0000", 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9993000000000006e-05 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373130+0000", 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000100214 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374163+0000", 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010331839999999999 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374534+0000", 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037083 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374567+0000", 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2549999999999998e-05 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374602+0000", 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5503999999999998e-05 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:50 2.14 2:292cf221:::200.00000000:head [writefull 0~90 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e77)", 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:31.372854+0000", 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 2.9972542760000001, 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001920001, 2024-03-16T19:31:34.440 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372854+0000", 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372854+0000", 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372854+0000", 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5900000000000003e-07 2024-03-16T19:31:34.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372857+0000", 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3140000000000002e-06 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372925+0000", 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8816e-05 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372930+0000", 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9429999999999999e-06 2024-03-16T19:31:34.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373562+0000", 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00063115399999999998 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373649+0000", 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7378000000000003e-05 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373777+0000", 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012839500000000001 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374622+0000", 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084469299999999998 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374726+0000", 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000103853 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374753+0000", 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6825000000000002e-05 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374774+0000", 2024-03-16T19:31:34.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1271e-05 2024-03-16T19:31:34.445 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.445 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.445 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.445 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.445 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.445 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:34.446 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:34.446 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:34.465+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:34.547 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:34.566+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:34.593 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:34.593 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:34.601 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:34.687 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:34.706+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:34.748 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:34.749 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:34.749 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:34.749 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:34.749 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:34.749 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:34.768+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:34.757 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:34.850 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:34.870+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:34.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.891+0000 7f1ef67fc640 1 -- 172.21.15.172:0/834711556 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v79) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1efc058a60 con 0x7f1f000c2230 2024-03-16T19:31:34.871 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is set 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1ee406b3b0 msgr2=0x7f1ee406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1ee406b3b0 0x7f1ee406d870 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0036000 tx=0x7f1ef0008030 comp rx=0 tx=0).stop 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f000c2230 msgr2=0x7f1f0013e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f000c2230 0x7f1f0013e4f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00efc0 tx=0x7f1efc00c490 comp rx=0 tx=0).stop 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 shutdown_connections 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f00156dc0 0x7f1f001f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1ee406b3b0 0x7f1ee406d870 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f00150ce0 0x7f1f0014e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 --2- 172.21.15.172:0/834711556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f000c2230 0x7f1f0013e4f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 >> 172.21.15.172:0/834711556 conn(0x7f1f000cc030 msgr2=0x7f1f000c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 shutdown_connections 2024-03-16T19:31:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:34.894+0000 7f1f07b81640 1 -- 172.21.15.172:0/834711556 wait complete. 2024-03-16T19:31:34.907 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 138.69714014499999, 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.908 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.911 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:31:34.912 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 138.32044459799999, 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.913 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.914 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.916 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 137.54446815399999, 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.917 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.918 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.919 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.920 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:31:34.921 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 137.54438374099999, 2024-03-16T19:31:34.922 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.923 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.925 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.926 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.927 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 137.54437638499999, 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.928 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.930 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.931 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.932 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 137.35183719299999, 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.933 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.935 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.936 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.937 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.938 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 137.35183138599999, 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.939 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:31:34.940 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:31:34.941 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:31:34.942 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.943 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 137.35171325300001, 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.944 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.945 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.946 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:31:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:31:34.948 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 137.35161805600001, 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.949 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:31:34.950 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:31:34.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 137.35161404499999, 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:31:34.954 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:31:34.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:31:34.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.958 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.93268217100001, 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.959 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.960 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:31:34.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:31:34.962 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.932632802, 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.964 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.965 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.966 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:31:34.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:31:34.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.93255396999999, 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.969 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.970 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:31:34.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:31:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:34.974 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.93255075900001, 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.975 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:31:34.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:31:34.978 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.979 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.932547806, 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:31:34.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.93243283199999, 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.985 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:31:34.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.989 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.93242948, 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.990 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:31:34.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:31:34.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.994 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.93242636400001, 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.995 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:34.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:34.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.932365895, 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:35.000 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:35.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:31:35.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:31:35.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 136.323685845, 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:35.005 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:31:35.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:35.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:35.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:35.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:31:35.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:31:35.010 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:35.010 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:35.010 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:35.010 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:35.010 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:35.010 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:35.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:35.012 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:35.031+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:35.112 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:35.131+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:35.119 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:35.119 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:35.128 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:35.213 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:35.232+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:35.234 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:35.234 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:35.234 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:35.234 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:35.235 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:35.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:35.314 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:35.335+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:35.351 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:35.351 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:35.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:35.351 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:35.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.351 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:31:35.351 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.98694272099999, 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:31:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.98618248599999, 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:31:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.98536853100001, 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:31:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:31:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:31:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.984728606, 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:31:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.98407050899999, 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:35.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.983537934, 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.92536603100001, 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:31:35.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:31:35.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:31:35.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.924763837, 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:35.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:31:35.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:31:35.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:31:35.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.92412089199999, 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:31:35.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.388 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 137.923616931, 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.392 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 136.764940529, 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 124.01297126999999, 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:31:35.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 124.012946734, 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:31:35.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:31:35.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:31:35.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:35.406 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 124.012942204, 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 124.012932392, 2024-03-16T19:31:35.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:35.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:31:35.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 124.012928539, 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.416 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:31:35.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:31:35.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 124.012925223, 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:35.421 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:31:35.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 124.01172964, 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4588.0:1 3.32 3.2b4ef7b2 (undecoded) ondisk+read+known_if_redirected e21)", 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:29.176862+0000", 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1936961589999999, 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036772179999999999, 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4588", 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/2627359036", 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176862+0000", 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176862+0000", 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176864+0000", 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8640000000000001e-06 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176869+0000", 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849999999999999e-06 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176887+0000", 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8048999999999999e-05 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176897+0000", 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0441999999999999e-05 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.180528+0000", 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036310629999999999 2024-03-16T19:31:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.180539+0000", 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1015e-05 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4588.0:2 3.32 3.2b4ef7b2 (undecoded) ondisk+write+known_if_redirected e21)", 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:29.176872+0000", 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1936866430000004, 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037192620000000001, 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:35.432 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4588", 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/2627359036", 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176872+0000", 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176872+0000", 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176872+0000", 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1899999999999999e-07 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176873+0000", 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176927+0000", 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4358999999999999e-05 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176934+0000", 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5039999999999999e-06 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.180586+0000", 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036517640000000001 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.180591+0000", 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3020000000000002e-06 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:35.435 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:35.436 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:35.437 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:35.437 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:35.457+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:35.537 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:35.556+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:35.584 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:35.584 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:35.584 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:35.584 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:35.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:35.638 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:35.658+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:35.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-16T19:31:35.700 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:35.701 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:35.701 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:35.701 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:35.701 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:35.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:35.751 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:35.771+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:35.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.819+0000 7ff0679e9640 1 Processor -- start 2024-03-16T19:31:35.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.820+0000 7ff0679e9640 1 -- start start 2024-03-16T19:31:35.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.820+0000 7ff0679e9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 0x7ff06015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.820+0000 7ff0679e9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff0600b3630 0x7ff0600cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.820+0000 7ff0679e9640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0600b4270 con 0x7ff0600cb5e0 2024-03-16T19:31:35.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.820+0000 7ff0679e9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0601552b0 con 0x7ff0600b3630 2024-03-16T19:31:35.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.820+0000 7ff0679e9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06015b4b0 con 0x7ff060154ed0 2024-03-16T19:31:35.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff065f86640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff0600cb5e0 0x7ff0600b4700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56064/0 (socket says 172.21.15.172:56064) 2024-03-16T19:31:35.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff065f86640 1 -- 172.21.15.172:0/1315338859 learned_addr learned my addr 172.21.15.172:0/1315338859 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:35.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff065785640 1 --2- 172.21.15.172:0/1315338859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 0x7ff06015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff064f84640 1 --2- 172.21.15.172:0/1315338859 >> v2:172.21.15.174:3300/0 conn(0x7ff0600b3630 0x7ff0600cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff057fff640 1 -- 172.21.15.172:0/1315338859 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054455781 0 0) 0x7ff0600b4270 con 0x7ff0600cb5e0 2024-03-16T19:31:35.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff057fff640 1 -- 172.21.15.172:0/1315338859 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff04c003520 con 0x7ff0600cb5e0 2024-03-16T19:31:35.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff065785640 1 -- 172.21.15.172:0/1315338859 >> v2:172.21.15.174:3300/0 conn(0x7ff0600b3630 msgr2=0x7ff0600cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:35.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff065785640 1 --2- 172.21.15.172:0/1315338859 >> v2:172.21.15.174:3300/0 conn(0x7ff0600b3630 0x7ff0600cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:35.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff065785640 1 -- 172.21.15.172:0/1315338859 >> v1:172.21.15.172:6789/0 conn(0x7ff0600cb5e0 legacy=0x7ff0600b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:35.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.821+0000 7ff065785640 1 -- 172.21.15.172:0/1315338859 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06015b790 con 0x7ff060154ed0 2024-03-16T19:31:35.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.822+0000 7ff065785640 1 --2- 172.21.15.172:0/1315338859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 0x7ff06015ab40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff0580097a0 tx=0x7ff05802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30d19a325e7f1f79 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:35.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.822+0000 7ff057fff640 1 -- 172.21.15.172:0/1315338859 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05802d040 con 0x7ff060154ed0 2024-03-16T19:31:35.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.822+0000 7ff057fff640 1 -- 172.21.15.172:0/1315338859 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff058031810 con 0x7ff060154ed0 2024-03-16T19:31:35.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.822+0000 7ff057fff640 1 -- 172.21.15.172:0/1315338859 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058031b80 con 0x7ff060154ed0 2024-03-16T19:31:35.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.823+0000 7ff0679e9640 1 -- 172.21.15.172:0/1315338859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 msgr2=0x7ff06015ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:35.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.823+0000 7ff0679e9640 1 --2- 172.21.15.172:0/1315338859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 0x7ff06015ab40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff0580097a0 tx=0x7ff05802b8b0 comp rx=0 tx=0).stop 2024-03-16T19:31:35.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.823+0000 7ff0679e9640 1 -- 172.21.15.172:0/1315338859 shutdown_connections 2024-03-16T19:31:35.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.823+0000 7ff0679e9640 1 --2- 172.21.15.172:0/1315338859 >> v2:172.21.15.174:3300/0 conn(0x7ff0600b3630 0x7ff0600cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:35.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.823+0000 7ff0679e9640 1 --2- 172.21.15.172:0/1315338859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 0x7ff06015ab40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:35.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.823+0000 7ff0679e9640 1 -- 172.21.15.172:0/1315338859 >> 172.21.15.172:0/1315338859 conn(0x7ff06006bf60 msgr2=0x7ff06006d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:35.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.823+0000 7ff0679e9640 1 -- 172.21.15.172:0/1315338859 shutdown_connections 2024-03-16T19:31:35.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.823+0000 7ff0679e9640 1 -- 172.21.15.172:0/1315338859 wait complete. 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.824+0000 7ff0679e9640 1 Processor -- start 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.824+0000 7ff0679e9640 1 -- start start 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.824+0000 7ff0679e9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0600b3630 0x7ff0600c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.824+0000 7ff0679e9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0600cb5e0 0x7ff0600c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.824+0000 7ff0679e9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 0x7ff0600c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.824+0000 7ff0679e9640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06015ab40 con 0x7ff0600cb5e0 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.824+0000 7ff0679e9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0600c6390 con 0x7ff0600b3630 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.824+0000 7ff0679e9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0600c3fa0 con 0x7ff060154ed0 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065785640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0600b3630 0x7ff0600c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065f86640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 0x7ff0600c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff064f84640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0600cb5e0 0x7ff0600c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065785640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0600b3630 0x7ff0600c68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:37418/0 (socket says 172.21.15.172:37418) 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065785640 1 -- 172.21.15.172:0/2230556564 learned_addr learned my addr 172.21.15.172:0/2230556564 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065785640 1 -- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 msgr2=0x7ff0600c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065785640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 0x7ff0600c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065785640 1 -- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0600cb5e0 msgr2=0x7ff0600c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065785640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0600cb5e0 0x7ff0600c4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065785640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff058009400 con 0x7ff0600b3630 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff060154ed0 0x7ff0600c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff064f84640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff0600cb5e0 0x7ff0600c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff065785640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0600b3630 0x7ff0600c68c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff05802bec0 tx=0x7ff0580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058003820 con 0x7ff0600b3630 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05803f070 con 0x7ff0600b3630 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0600ca520 con 0x7ff0600b3630 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.825+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058003ba0 con 0x7ff0600b3630 2024-03-16T19:31:35.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.826+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0600c6e00 con 0x7ff0600b3630 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.826+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff060068d20 con 0x7ff0600b3630 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.827+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff058003d00 con 0x7ff0600b3630 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.827+0000 7ff055ffb640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff04406b3b0 0x7ff04406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.827+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5857+0+0 (secure 0 0 0) 0x7ff058041070 con 0x7ff0600b3630 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.828+0000 7ff064f84640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff04406b3b0 0x7ff04406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.828+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v79) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff0580b0240 con 0x7ff0600b3630 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.828+0000 7ff0679e9640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ff0600cce10 0x7ff060144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.828+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff06015ba70 con 0x7ff0600cce10 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.828+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ff0600cce10 0x7ff060144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.828+0000 7ff064f84640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff04406b3b0 0x7ff04406d870 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff06015f1d0 tx=0x7ff050009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.828+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7ff0600cce10 0x7ff060144730 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:35.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.829+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff05c006d20 con 0x7ff0600cce10 2024-03-16T19:31:35.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.844+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0600b3f00 con 0x7ff0600cce10 2024-03-16T19:31:35.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.845+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff05c004b60 con 0x7ff0600cce10 2024-03-16T19:31:35.825 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:35.826 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:35.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.845+0000 7ff0679e9640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ff0601f3c70 0x7ff0601f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.845+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0600c40e0 con 0x7ff0601f3c70 2024-03-16T19:31:35.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.846+0000 7ff065785640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ff0601f3c70 0x7ff0601f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.846+0000 7ff065785640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7ff0601f3c70 0x7ff0601f4050 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:35.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.847+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff05807a0a0 con 0x7ff0601f3c70 2024-03-16T19:31:35.841 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:35.860+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:35.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.862+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0600cd440 con 0x7ff0601f3c70 2024-03-16T19:31:35.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.864+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0600cd440 con 0x7ff0601f3c70 2024-03-16T19:31:35.844 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:35.845 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:35.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.864+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=80}) v3 -- 0x7ff060145180 con 0x7ff0600b3630 2024-03-16T19:31:35.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.864+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff0600681c0 con 0x7ff0600b3630 2024-03-16T19:31:35.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.864+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0580b0220 con 0x7ff0600b3630 2024-03-16T19:31:35.845 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:35.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.864+0000 7ff0679e9640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ff0600c82c0 0x7ff060061930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.864+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff06014ef50 con 0x7ff0600c82c0 2024-03-16T19:31:35.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.865+0000 7ff064f84640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ff0600c82c0 0x7ff060061930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.865+0000 7ff064f84640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7ff0600c82c0 0x7ff060061930 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:35.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.865+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff06014ef50 con 0x7ff0600c82c0 2024-03-16T19:31:35.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.880+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff060062140 con 0x7ff0600c82c0 2024-03-16T19:31:35.861 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:35.861 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:35.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.882+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff060062140 con 0x7ff0600c82c0 2024-03-16T19:31:35.862 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:35.862 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:35.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.882+0000 7ff0679e9640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff0600626d0 0x7ff06014ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.882+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0600cbbe0 con 0x7ff0600626d0 2024-03-16T19:31:35.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.882+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff0600626d0 0x7ff06014ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.883+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff0600626d0 0x7ff06014ff80 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:35.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.884+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0600cbbe0 con 0x7ff0600626d0 2024-03-16T19:31:35.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:35.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.899+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff060150790 con 0x7ff0600626d0 2024-03-16T19:31:35.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.902+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff060150790 con 0x7ff0600626d0 2024-03-16T19:31:35.882 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:35.882 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:35.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.902+0000 7ff0679e9640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ff06002c2f0 0x7ff06002c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.902+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff06015a860 con 0x7ff06002c2f0 2024-03-16T19:31:35.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.902+0000 7ff065785640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ff06002c2f0 0x7ff06002c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.903+0000 7ff065785640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7ff06002c2f0 0x7ff06002c710 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:35.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.903+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff06015a860 con 0x7ff06002c2f0 2024-03-16T19:31:35.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.918+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff06015a860 con 0x7ff06002c2f0 2024-03-16T19:31:35.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.919+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff06015a860 con 0x7ff06002c2f0 2024-03-16T19:31:35.899 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:35.900 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:35.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.920+0000 7ff0679e9640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff060204d80 0x7ff060207170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.920+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff060150900 con 0x7ff060204d80 2024-03-16T19:31:35.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.920+0000 7ff064f84640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff060204d80 0x7ff060207170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.920+0000 7ff064f84640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff060204d80 0x7ff060207170 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:35.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.921+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff060150900 con 0x7ff060204d80 2024-03-16T19:31:35.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.935+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff060150900 con 0x7ff060204d80 2024-03-16T19:31:35.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.938+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff060150900 con 0x7ff060204d80 2024-03-16T19:31:35.918 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.938+0000 7ff0679e9640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff060207bc0 0x7ff060209fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.938+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff06002d2c0 con 0x7ff060207bc0 2024-03-16T19:31:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.938+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff060207bc0 0x7ff060209fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.939+0000 7ff065f86640 1 -- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff060207bc0 msgr2=0x7ff060209fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 27 2024-03-16T19:31:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.939+0000 7ff065f86640 1 -- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff060207bc0 msgr2=0x7ff060209fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:31:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.939+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff060207bc0 0x7ff060209fb0 crc :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:31:35.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:35.939+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff060207bc0 0x7ff060209fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:31:35.941 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:35.961+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:35.977 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:35.977 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:35.986 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:36.043 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:36.062+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:36.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:36.140+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff060207bc0 0x7ff060209fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:36.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:36.140+0000 7ff065f86640 1 --2- 172.21.15.172:0/2230556564 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7ff060207bc0 0x7ff060209fb0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:36.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:36.141+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff06002d2c0 con 0x7ff060207bc0 2024-03-16T19:31:36.132 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:36.132 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:36.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:36.155+0000 7ff0679e9640 1 -- 172.21.15.172:0/2230556564 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff0601505a0 con 0x7ff060207bc0 2024-03-16T19:31:36.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:36.157+0000 7ff055ffb640 1 -- 172.21.15.172:0/2230556564 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff0601505a0 con 0x7ff060207bc0 2024-03-16T19:31:36.137 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:36.137 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:36.142 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:36.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:36.143 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:36.164+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:36.244 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:36.264+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:36.249 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:36.250 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:36.259 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:36.344 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:36.364+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:36.378 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:36.379 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:36.388 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:36.446 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:36.466+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:36.534 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:36.534 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:36.543 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:36.587 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:36.606+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:36.648 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:36.668+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:36.653 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:36.653 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:36.662 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:36.749 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:36.769+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:36.809 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:36.810 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:36.818 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:36.862 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:36.881+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:36.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:31:36.926 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:36.926 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:36.935 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:36.950 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:36.971+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:36.992 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.012+0000 7f8ff30c3640 1 Processor -- start 2024-03-16T19:31:36.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.012+0000 7f8ff30c3640 1 -- start start 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8ff30c3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8fec0b5f10 0x7f8fec06b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8ff30c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec0caf40 0x7f8fec06bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8ff30c3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fec06b110 con 0x7f8fec154e50 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8ff30c3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec15b8d0 con 0x7f8fec0b5f10 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8ff30c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec15ad50 con 0x7f8fec0caf40 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8febfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8fec0b5f10 0x7f8fec06b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8ff1660640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec0caf40 0x7f8fec06bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8ff0e5f640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f8fec154e50 0x7f8fec0cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:59098/0 (socket says 172.21.15.172:59098) 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8ff1660640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec0caf40 0x7f8fec06bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:38850/0 (socket says 172.21.15.172:38850) 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8febfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8fec0b5f10 0x7f8fec06b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:37430/0 (socket says 172.21.15.172:37430) 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.013+0000 7f8ff0e5f640 1 -- 172.21.15.172:0/3657224670 learned_addr learned my addr 172.21.15.172:0/3657224670 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:36.994 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.014+0000 7f8febfff640 1 -- 172.21.15.172:0/3657224670 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec0caf40 msgr2=0x7f8fec06bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:36.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.014+0000 7f8febfff640 1 --2- 172.21.15.172:0/3657224670 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec0caf40 0x7f8fec06bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:36.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.014+0000 7f8febfff640 1 -- 172.21.15.172:0/3657224670 >> v1:172.21.15.174:6789/0 conn(0x7f8fec154e50 legacy=0x7f8fec0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:36.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.014+0000 7f8febfff640 1 -- 172.21.15.172:0/3657224670 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec15b030 con 0x7f8fec0b5f10 2024-03-16T19:31:36.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.014+0000 7f8ff1660640 1 --2- 172.21.15.172:0/3657224670 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec0caf40 0x7f8fec06bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8febfff640 1 --2- 172.21.15.172:0/3657224670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 0x7f8fec06b5a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00097a0 tx=0x7f8fe002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad29837e4c4f4f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8feb7fe640 1 -- 172.21.15.172:0/3657224670 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0030610 con 0x7f8fec0b5f10 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8feb7fe640 1 -- 172.21.15.172:0/3657224670 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe0030770 con 0x7f8fec0b5f10 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8feb7fe640 1 -- 172.21.15.172:0/3657224670 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0030b00 con 0x7f8fec0b5f10 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8ff30c3640 1 -- 172.21.15.172:0/3657224670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 msgr2=0x7f8fec06b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8ff30c3640 1 --2- 172.21.15.172:0/3657224670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 0x7f8fec06b5a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00097a0 tx=0x7f8fe002b960 comp rx=0 tx=0).stop 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8ff30c3640 1 -- 172.21.15.172:0/3657224670 shutdown_connections 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8ff30c3640 1 --2- 172.21.15.172:0/3657224670 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec0caf40 0x7f8fec06bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8ff30c3640 1 --2- 172.21.15.172:0/3657224670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 0x7f8fec06b5a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:36.996 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8ff30c3640 1 -- 172.21.15.172:0/3657224670 >> 172.21.15.172:0/3657224670 conn(0x7f8fec066c20 msgr2=0x7f8fec067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:36.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8ff30c3640 1 -- 172.21.15.172:0/3657224670 shutdown_connections 2024-03-16T19:31:36.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.015+0000 7f8ff30c3640 1 -- 172.21.15.172:0/3657224670 wait complete. 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.016+0000 7f8ff30c3640 1 Processor -- start 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.017+0000 7f8ff30c3640 1 -- start start 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.017+0000 7f8ff30c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 0x7f8fec148fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.017+0000 7f8ff30c3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fec0caf40 0x7f8fec1f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.017+0000 7f8ff30c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec154e50 0x7f8fec1467e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.017+0000 7f8ff30c3640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec157ee0 con 0x7f8fec0caf40 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.017+0000 7f8ff30c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec14a8c0 con 0x7f8fec0b5f10 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.017+0000 7f8ff30c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec14aba0 con 0x7f8fec154e50 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.017+0000 7f8febfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fec0caf40 0x7f8fec1f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff1660640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec154e50 0x7f8fec1467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:36.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8febfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fec0caf40 0x7f8fec1f19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34418/0 (socket says 172.21.15.172:34418) 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8febfff640 1 -- 172.21.15.172:0/1773256312 learned_addr learned my addr 172.21.15.172:0/1773256312 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff0e5f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 0x7f8fec148fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff1660640 1 -- 172.21.15.172:0/1773256312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 msgr2=0x7f8fec148fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff1660640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 0x7f8fec148fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff1660640 1 -- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fec0caf40 msgr2=0x7f8fec1f19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff1660640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fec0caf40 0x7f8fec1f19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff1660640 1 -- 172.21.15.172:0/1773256312 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe4009a70 con 0x7f8fec154e50 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff0e5f640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 0x7f8fec148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8febfff640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fec0caf40 0x7f8fec1f19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff1660640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec154e50 0x7f8fec1467e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400ef90 tx=0x7f8fe400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8fe97fa640 1 -- 172.21.15.172:0/1773256312 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe400cd30 con 0x7f8fec154e50 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8fe97fa640 1 -- 172.21.15.172:0/1773256312 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe400f040 con 0x7f8fec154e50 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8fe97fa640 1 -- 172.21.15.172:0/1773256312 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4003bd0 con 0x7f8fec154e50 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe0009400 con 0x7f8fec154e50 2024-03-16T19:31:37.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.018+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fec15ab20 con 0x7f8fec154e50 2024-03-16T19:31:37.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.019+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fec0c9ab0 con 0x7f8fec154e50 2024-03-16T19:31:37.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.020+0000 7f8fe97fa640 1 -- 172.21.15.172:0/1773256312 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8fe4014480 con 0x7f8fec154e50 2024-03-16T19:31:37.001 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.020+0000 7f8fe97fa640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8fd806b3e0 0x7f8fd806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:37.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.020+0000 7f8fe97fa640 1 -- 172.21.15.172:0/1773256312 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f8fe408a700 con 0x7f8fec154e50 2024-03-16T19:31:37.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.020+0000 7f8ff0e5f640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8fd806b3e0 0x7f8fd806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:37.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.021+0000 7f8ff0e5f640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8fd806b3e0 0x7f8fd806d8a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc007fd0 tx=0x7f8fdc007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:37.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.026+0000 7f8fe97fa640 1 -- 172.21.15.172:0/1773256312 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fe4055370 con 0x7f8fec154e50 2024-03-16T19:31:37.051 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:37.071+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:37.085 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:37.085 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:37.094 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:37.152 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:37.172+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:37.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.208+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f8fec075a70 con 0x7f8fec154e50 2024-03-16T19:31:37.201 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:37.201 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:37.210 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:37.252 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:37.273+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:37.315 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:37.316 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:37.325 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:37.353 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:37.374+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:37.432 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:37.432 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:37.440 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:37.481 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-16T19:31:37.481 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-16T19:31:37.481 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-16T19:31:37.482 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-03-16T19:31:37.485 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:37.504+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:37.546 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:37.547 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:37.555 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:37.555 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:37.574+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:37.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.606+0000 7f4eeefd0640 1 Processor -- start 2024-03-16T19:31:37.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.606+0000 7f4eeefd0640 1 -- start start 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.606+0000 7f4eeefd0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4ee8156dc0 0x7f4ee80c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.606+0000 7f4eeefd0640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ee806dbd0 0x7f4ee806b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.606+0000 7f4eeefd0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee806e080 con 0x7f4ee8150ce0 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.606+0000 7f4eeefd0640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee80c1da0 con 0x7f4ee806dbd0 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.606+0000 7f4eeefd0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee8157370 con 0x7f4ee8156dc0 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.607+0000 7f4eed56d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ee806dbd0 0x7f4ee806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.607+0000 7f4eed56d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ee806dbd0 0x7f4ee806b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34420/0 (socket says 172.21.15.172:34420) 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.607+0000 7f4eed56d640 1 -- 172.21.15.172:0/2401196157 learned_addr learned my addr 172.21.15.172:0/2401196157 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.607+0000 7f4ee7fff640 1 --2- 172.21.15.172:0/2401196157 >> v2:172.21.15.174:3300/0 conn(0x7f4ee8156dc0 0x7f4ee80c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.607+0000 7f4eed56d640 1 -- 172.21.15.172:0/2401196157 >> v2:172.21.15.174:3300/0 conn(0x7f4ee8156dc0 msgr2=0x7f4ee80c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.607+0000 7f4eed56d640 1 --2- 172.21.15.172:0/2401196157 >> v2:172.21.15.174:3300/0 conn(0x7f4ee8156dc0 0x7f4ee80c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.607+0000 7f4eed56d640 1 -- 172.21.15.172:0/2401196157 >> v1:172.21.15.174:6789/0 conn(0x7f4ee8150ce0 legacy=0x7f4ee80c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.607+0000 7f4eed56d640 1 -- 172.21.15.172:0/2401196157 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee8157650 con 0x7f4ee806dbd0 2024-03-16T19:31:37.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4eed56d640 1 --2- 172.21.15.172:0/2401196157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee806dbd0 0x7f4ee806b760 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8070390 tx=0x7f4ee002d120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3317be6bee1664 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:37.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4ee77fe640 1 -- 172.21.15.172:0/2401196157 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee002dcc0 con 0x7f4ee806dbd0 2024-03-16T19:31:37.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4ee77fe640 1 -- 172.21.15.172:0/2401196157 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee002de20 con 0x7f4ee806dbd0 2024-03-16T19:31:37.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4ee77fe640 1 -- 172.21.15.172:0/2401196157 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0035610 con 0x7f4ee806dbd0 2024-03-16T19:31:37.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4eeefd0640 1 -- 172.21.15.172:0/2401196157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee806dbd0 msgr2=0x7f4ee806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4eeefd0640 1 --2- 172.21.15.172:0/2401196157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee806dbd0 0x7f4ee806b760 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8070390 tx=0x7f4ee002d120 comp rx=0 tx=0).stop 2024-03-16T19:31:37.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4eeefd0640 1 -- 172.21.15.172:0/2401196157 shutdown_connections 2024-03-16T19:31:37.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4eeefd0640 1 --2- 172.21.15.172:0/2401196157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee806dbd0 0x7f4ee806b760 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4eeefd0640 1 --2- 172.21.15.172:0/2401196157 >> v2:172.21.15.174:3300/0 conn(0x7f4ee8156dc0 0x7f4ee80c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4eeefd0640 1 -- 172.21.15.172:0/2401196157 >> 172.21.15.172:0/2401196157 conn(0x7f4ee80cc030 msgr2=0x7f4ee80cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:37.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4eeefd0640 1 -- 172.21.15.172:0/2401196157 shutdown_connections 2024-03-16T19:31:37.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.608+0000 7f4eeefd0640 1 -- 172.21.15.172:0/2401196157 wait complete. 2024-03-16T19:31:37.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.609+0000 7f4eeefd0640 1 Processor -- start 2024-03-16T19:31:37.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.609+0000 7f4eeefd0640 1 -- start start 2024-03-16T19:31:37.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4eeefd0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ee806dbd0 0x7f4ee81460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:37.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4eeefd0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ee8150ce0 0x7f4ee8154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:37.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4eeefd0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee8156dc0 0x7f4ee8155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:37.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4eeefd0640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8156380 con 0x7f4ee8156dc0 2024-03-16T19:31:37.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4eeefd0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8146610 con 0x7f4ee8150ce0 2024-03-16T19:31:37.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4eeefd0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee80cbd00 con 0x7f4ee806dbd0 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4eed56d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee8156dc0 0x7f4ee8155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4eed56d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee8156dc0 0x7f4ee8155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34424/0 (socket says 172.21.15.172:34424) 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4eed56d640 1 -- 172.21.15.172:0/612002946 learned_addr learned my addr 172.21.15.172:0/612002946 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.610+0000 7f4ee7fff640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ee8150ce0 0x7f4ee8154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eecd6c640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ee806dbd0 0x7f4ee81460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eed56d640 1 -- 172.21.15.172:0/612002946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ee806dbd0 msgr2=0x7f4ee81460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eed56d640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ee806dbd0 0x7f4ee81460d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eed56d640 1 -- 172.21.15.172:0/612002946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ee8150ce0 msgr2=0x7f4ee8154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eed56d640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ee8150ce0 0x7f4ee8154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eed56d640 1 -- 172.21.15.172:0/612002946 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0009b30 con 0x7f4ee8156dc0 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eecd6c640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ee806dbd0 0x7f4ee81460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eed56d640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee8156dc0 0x7f4ee8155830 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0037040 tx=0x7f4ee0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4ee57fa640 1 -- 172.21.15.172:0/612002946 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee003f620 con 0x7f4ee8156dc0 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4ee57fa640 1 -- 172.21.15.172:0/612002946 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee003f780 con 0x7f4ee8156dc0 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4ee57fa640 1 -- 172.21.15.172:0/612002946 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee003fb70 con 0x7f4ee8156dc0 2024-03-16T19:31:37.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee80cb740 con 0x7f4ee8156dc0 2024-03-16T19:31:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.611+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee81455f0 con 0x7f4ee8156dc0 2024-03-16T19:31:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.612+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee81510c0 con 0x7f4ee8156dc0 2024-03-16T19:31:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.612+0000 7f4ee57fa640 1 -- 172.21.15.172:0/612002946 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4ee003fcd0 con 0x7f4ee8156dc0 2024-03-16T19:31:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.613+0000 7f4ee57fa640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ecc06b3b0 0x7f4ecc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.613+0000 7f4ee57fa640 1 -- 172.21.15.172:0/612002946 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f4ee00ac610 con 0x7f4ee8156dc0 2024-03-16T19:31:37.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.613+0000 7f4eecd6c640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ecc06b3b0 0x7f4ecc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:37.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.613+0000 7f4eecd6c640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ecc06b3b0 0x7f4ecc06d870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4edc001660 tx=0x7f4edc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:37.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.617+0000 7f4ee57fa640 1 -- 172.21.15.172:0/612002946 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ee00771d0 con 0x7f4ee8156dc0 2024-03-16T19:31:37.656 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:37.675+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:37.702 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:37.702 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:37.711 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:37.756 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:37.776+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:37.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.797+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd out", "ids": ["0"]} v 0) v1 -- 0x7f4ee8070c70 con 0x7f4ee8156dc0 2024-03-16T19:31:37.777 INFO:tasks.ceph.mon.a.smithi172.stderr:2024-03-16T19:31:37.797+0000 7f2459211640 -1 mon.a@0(leader).osd e79 definitely_dead 0 2024-03-16T19:31:37.857 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:37.877+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:37.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.894+0000 7f4ee57fa640 1 -- 172.21.15.172:0/612002946 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["0"]}]=0 marked out osd.0. v80) v1 ==== 87+0+0 (secure 0 0 0) 0x7f4ee0009510 con 0x7f4ee8156dc0 2024-03-16T19:31:37.875 INFO:teuthology.orchestra.run.smithi172.stderr:marked out osd.0. 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ecc06b3b0 msgr2=0x7f4ecc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ecc06b3b0 0x7f4ecc06d870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4edc001660 tx=0x7f4edc009040 comp rx=0 tx=0).stop 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee8156dc0 msgr2=0x7f4ee8155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee8156dc0 0x7f4ee8155830 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0037040 tx=0x7f4ee0004ab0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 shutdown_connections 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ee8156dc0 0x7f4ee8155830 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ee8150ce0 0x7f4ee8154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ecc06b3b0 0x7f4ecc06d870 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 --2- 172.21.15.172:0/612002946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ee806dbd0 0x7f4ee81460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.896+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 >> 172.21.15.172:0/612002946 conn(0x7f4ee80cc030 msgr2=0x7f4ee80726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:37.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.899+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 shutdown_connections 2024-03-16T19:31:37.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.899+0000 7f4eeefd0640 1 -- 172.21.15.172:0/612002946 wait complete. 2024-03-16T19:31:37.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.899+0000 7f8fe97fa640 1 -- 172.21.15.172:0/1773256312 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v80) v1 ==== 91+0+0 (secure 0 0 0) 0x7f8fe4058a30 con 0x7f8fec154e50 2024-03-16T19:31:37.880 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:31:37.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8fd806b3e0 msgr2=0x7f8fd806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8fd806b3e0 0x7f8fd806d8a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc007fd0 tx=0x7f8fdc007e80 comp rx=0 tx=0).stop 2024-03-16T19:31:37.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec154e50 msgr2=0x7f8fec1467e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:37.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec154e50 0x7f8fec1467e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400ef90 tx=0x7f8fe400c490 comp rx=0 tx=0).stop 2024-03-16T19:31:37.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 shutdown_connections 2024-03-16T19:31:37.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8fd806b3e0 0x7f8fd806d8a0 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc007fd0 tx=0x7f8fdc007e80 comp rx=0 tx=0).stop 2024-03-16T19:31:37.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8fec154e50 0x7f8fec1467e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8fec0caf40 0x7f8fec1f19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 --2- 172.21.15.172:0/1773256312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fec0b5f10 0x7f8fec148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:37.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 >> 172.21.15.172:0/1773256312 conn(0x7f8fec066c20 msgr2=0x7f8fec147450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:37.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.904+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 shutdown_connections 2024-03-16T19:31:37.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:37.907+0000 7f8ff30c3640 1 -- 172.21.15.172:0/1773256312 wait complete. 2024-03-16T19:31:37.891 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:37.891 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:37.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-16T19:31:37.957 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:37.977+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:38.059 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:38.078+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:38.069 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:38.069 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:38.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-16T19:31:38.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-16T19:31:38.160 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:38.179+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:38.188 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-16T19:31:38.188 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-16T19:31:38.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-16T19:31:38.245 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.265+0000 7f83ec5e2640 1 Processor -- start 2024-03-16T19:31:38.245 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.265+0000 7f83ec5e2640 1 -- start start 2024-03-16T19:31:38.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ec5e2640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83e4154ed0 0x7f83e415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ec5e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b4700 0x7f83e40b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ec5e2640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e40b4270 con 0x7f83e40b3630 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ec5e2640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e41552b0 con 0x7f83e4154ed0 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ec5e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e415b4b0 con 0x7f83e40b4700 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ea37e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83e4154ed0 0x7f83e415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83e9b7d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f83e40b3630 0x7f83e40cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:56080/0 (socket says 172.21.15.172:56080) 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83e9b7d640 1 -- 172.21.15.172:0/1700933126 learned_addr learned my addr 172.21.15.172:0/1700933126 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83eab7f640 1 --2- 172.21.15.172:0/1700933126 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b4700 0x7f83e40b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ea37e640 1 -- 172.21.15.172:0/1700933126 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b4700 msgr2=0x7f83e40b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ea37e640 1 --2- 172.21.15.172:0/1700933126 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b4700 0x7f83e40b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ea37e640 1 -- 172.21.15.172:0/1700933126 >> v1:172.21.15.172:6789/0 conn(0x7f83e40b3630 legacy=0x7f83e40cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83e937c640 1 -- 172.21.15.172:0/1700933126 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238152426 0 0) 0x7f83e40b4270 con 0x7f83e40b3630 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83ea37e640 1 -- 172.21.15.172:0/1700933126 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e415b790 con 0x7f83e4154ed0 2024-03-16T19:31:38.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.266+0000 7f83eab7f640 1 --2- 172.21.15.172:0/1700933126 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b4700 0x7f83e40b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:38.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.267+0000 7f83ea37e640 1 --2- 172.21.15.172:0/1700933126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83e4154ed0 0x7f83e415ab40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0097a0 tx=0x7f83dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4b3f8c4f47c07 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.267+0000 7f83e937c640 1 -- 172.21.15.172:0/1700933126 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc031830 con 0x7f83e4154ed0 2024-03-16T19:31:38.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.267+0000 7f83e937c640 1 -- 172.21.15.172:0/1700933126 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83dc031990 con 0x7f83e4154ed0 2024-03-16T19:31:38.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.267+0000 7f83e937c640 1 -- 172.21.15.172:0/1700933126 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc031d20 con 0x7f83e4154ed0 2024-03-16T19:31:38.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.268+0000 7f83ec5e2640 1 -- 172.21.15.172:0/1700933126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83e4154ed0 msgr2=0x7f83e415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:38.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.268+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/1700933126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83e4154ed0 0x7f83e415ab40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0097a0 tx=0x7f83dc02b8d0 comp rx=0 tx=0).stop 2024-03-16T19:31:38.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.268+0000 7f83ec5e2640 1 -- 172.21.15.172:0/1700933126 shutdown_connections 2024-03-16T19:31:38.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.268+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/1700933126 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b4700 0x7f83e40b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:38.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.268+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/1700933126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83e4154ed0 0x7f83e415ab40 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:38.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.268+0000 7f83ec5e2640 1 -- 172.21.15.172:0/1700933126 >> 172.21.15.172:0/1700933126 conn(0x7f83e406bf60 msgr2=0x7f83e406d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:38.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.268+0000 7f83ec5e2640 1 -- 172.21.15.172:0/1700933126 shutdown_connections 2024-03-16T19:31:38.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.268+0000 7f83ec5e2640 1 -- 172.21.15.172:0/1700933126 wait complete. 2024-03-16T19:31:38.251 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.269+0000 7f83ec5e2640 1 Processor -- start 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.269+0000 7f83ec5e2640 1 -- start start 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83ec5e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b3630 0x7f83e40c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83ec5e2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83e40b4700 0x7f83e40c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83ec5e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83e4154ed0 0x7f83e415e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83ec5e2640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40d0260 con 0x7f83e40b4700 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83ec5e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40ca210 con 0x7f83e4154ed0 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83ec5e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40cf6d0 con 0x7f83e40b3630 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83e9b7d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83e40b4700 0x7f83e40c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83e9b7d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83e40b4700 0x7f83e40c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34436/0 (socket says 172.21.15.172:34436) 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83e9b7d640 1 -- 172.21.15.172:0/268350722 learned_addr learned my addr 172.21.15.172:0/268350722 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83e4154ed0 0x7f83e415e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83ea37e640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b3630 0x7f83e40c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83e9b7d640 1 -- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b3630 msgr2=0x7f83e40c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83e9b7d640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83e40b3630 0x7f83e40c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:38.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83e9b7d640 1 -- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83e4154ed0 msgr2=0x7f83e415e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83e9b7d640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83e4154ed0 0x7f83e415e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83e9b7d640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e0009a70 con 0x7f83e40b4700 2024-03-16T19:31:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.270+0000 7f83e9b7d640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83e40b4700 0x7f83e40c9cd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f83d400c500 tx=0x7f83d400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.271+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d40074e0 con 0x7f83e40b4700 2024-03-16T19:31:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.271+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d4007660 con 0x7f83e40b4700 2024-03-16T19:31:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.271+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83dc009400 con 0x7f83e40b4700 2024-03-16T19:31:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.271+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4004a20 con 0x7f83e40b4700 2024-03-16T19:31:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.271+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e40cf8c0 con 0x7f83e40b4700 2024-03-16T19:31:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.271+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f83e40d03a0 con 0x7f83e40b4700 2024-03-16T19:31:38.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.272+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f83d401c3b0 con 0x7f83e40b4700 2024-03-16T19:31:38.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.273+0000 7f83daffd640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f83c806b3e0 0x7f83c806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.273+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f83d4089390 con 0x7f83e40b4700 2024-03-16T19:31:38.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.273+0000 7f83ea37e640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f83c806b3e0 0x7f83c806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.273+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v80) v1 ==== 54+0+16 (secure 0 0 0) 0x7f83d40897e0 con 0x7f83e40b4700 2024-03-16T19:31:38.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.273+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f83e41595b0 0x7f83e40c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.273+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83e4158fd0 con 0x7f83e41595b0 2024-03-16T19:31:38.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.273+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f83e41595b0 0x7f83e40c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.273+0000 7f83ea37e640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f83c806b3e0 0x7f83c806d8a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f83dc036000 tx=0x7f83dc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.273+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7f83e41595b0 0x7f83e40c4880 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.274+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f83e4158fd0 con 0x7f83e41595b0 2024-03-16T19:31:38.261 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:38.280+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:38.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.289+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f83e415ba70 con 0x7f83e41595b0 2024-03-16T19:31:38.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.291+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f83e000f040 con 0x7f83e41595b0 2024-03-16T19:31:38.271 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0: {} 2024-03-16T19:31:38.271 INFO:teuthology.orchestra.run.smithi172.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-16T19:31:38.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.291+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f83e4159990 0x7f83e41f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.291+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83e40b2e20 con 0x7f83e4159990 2024-03-16T19:31:38.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.291+0000 7f83ea37e640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f83e4159990 0x7f83e41f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.291+0000 7f83ea37e640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f83e4159990 0x7f83e41f48d0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.292+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f83dc031b00 con 0x7f83e4159990 2024-03-16T19:31:38.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.307+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f83e40d03a0 con 0x7f83e4159990 2024-03-16T19:31:38.289 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.309+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f83e40d03a0 con 0x7f83e4159990 2024-03-16T19:31:38.289 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1: {} 2024-03-16T19:31:38.290 INFO:teuthology.orchestra.run.smithi172.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-16T19:31:38.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.309+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=81}) v3 -- 0x7f83e4068d20 con 0x7f83e40b4700 2024-03-16T19:31:38.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.309+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f83e40cfff0 con 0x7f83e40b4700 2024-03-16T19:31:38.290 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.310+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83d4054090 con 0x7f83e40b4700 2024-03-16T19:31:38.291 INFO:teuthology.orchestra.run.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-16T19:31:38.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.310+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f83e40c4dc0 0x7f83e40c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.310+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83e414b9d0 con 0x7f83e40c4dc0 2024-03-16T19:31:38.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.310+0000 7f83e9b7d640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f83e40c4dc0 0x7f83e40c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.310+0000 7f83e9b7d640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f83e40c4dc0 0x7f83e40c7d30 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.311+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f83e414b9d0 con 0x7f83e40c4dc0 2024-03-16T19:31:38.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.325+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f83e40c8540 con 0x7f83e40c4dc0 2024-03-16T19:31:38.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.327+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f83e40c8540 con 0x7f83e40c4dc0 2024-03-16T19:31:38.307 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3: {} 2024-03-16T19:31:38.307 INFO:teuthology.orchestra.run.smithi172.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-16T19:31:38.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.327+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f83e4062200 0x7f83e415bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.327+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83e40c86b0 con 0x7f83e4062200 2024-03-16T19:31:38.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.327+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f83e4062200 0x7f83e415bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.327+0000 7f83eab7f640 1 -- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f83e4062200 msgr2=0x7f83e415bf10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 24 2024-03-16T19:31:38.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.327+0000 7f83eab7f640 1 -- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f83e4062200 msgr2=0x7f83e415bf10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T19:31:38.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.327+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f83e4062200 0x7f83e415bf10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T19:31:38.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.327+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f83e4062200 0x7f83e415bf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T19:31:38.310 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:32 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'18, mlcod=16'14)", 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.94628369399999, 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003833534, 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383616+0000", 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6000000000000004e-07 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383809+0000", 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019270100000000001 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383827+0000", 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7744000000000002e-05 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383832+0000", 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2329999999999997e-06 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383875+0000", 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3662999999999999e-05 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.383894+0000", 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-03-16T19:31:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387415+0000", 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003521087 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387449+0000", 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3679000000000001e-05 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:33 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'19, mlcod=16'14)", 2024-03-16T19:31:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.94552345899999, 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058615079999999996, 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384376+0000", 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384381+0000", 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.352e-06 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384388+0000", 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4329999999999998e-06 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384393+0000", 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5029999999999999e-06 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384468+0000", 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5846000000000005e-05 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.384481+0000", 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952000000000001e-05 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390193+0000", 2024-03-16T19:31:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057114569999999996 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390237+0000", 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4396e-05 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:34 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'20, mlcod=17'16)", 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.944709504, 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050688649999999997, 2024-03-16T19:31:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385190+0000", 2024-03-16T19:31:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7800000000000001e-07 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385196+0000", 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1310000000000003e-06 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385206+0000", 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7089999999999997e-06 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385212+0000", 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8390000000000003e-06 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385251+0000", 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8850000000000002e-05 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385263+0000", 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.236e-05 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390240+0000", 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004977286 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390259+0000", 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8111999999999999e-05 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:35 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.944069579, 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048423229999999999, 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385830+0000", 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4099999999999997e-07 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385942+0000", 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111253 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385955+0000", 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.385967+0000", 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1770000000000001e-05 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386006+0000", 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8609999999999998e-05 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386023+0000", 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.719e-05 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390618+0000", 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045947740000000003 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390672+0000", 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4301000000000001e-05 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:36 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'16)", 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.94341148199999, 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042106549999999998, 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386488+0000", 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386489+0000", 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386495+0000", 2024-03-16T19:31:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6250000000000001e-06 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386508+0000", 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.386515+0000", 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2269999999999998e-06 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387195+0000", 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068068499999999999 2024-03-16T19:31:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387210+0000", 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4769e-05 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390675+0000", 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034652559999999999 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390698+0000", 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3119999999999999e-05 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:37 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'16)", 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.94287890699999, 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003708687, 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387020+0000", 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387021+0000", 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6400000000000001e-07 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387055+0000", 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4413999999999999e-05 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387069+0000", 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3332999999999999e-05 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387075+0000", 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7760000000000003e-06 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387481+0000", 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000406489 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.387495+0000", 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4154e-05 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390700+0000", 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032050300000000002 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.390729+0000", 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8727000000000001e-05 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:84 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.88470700400001, 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056402759999999996, 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445192+0000", 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445193+0000", 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445297+0000", 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104585 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445305+0000", 2024-03-16T19:31:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2449999999999998e-06 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445309+0000", 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8510000000000001e-06 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445350+0000", 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0405999999999999e-05 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445367+0000", 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6988999999999999e-05 2024-03-16T19:31:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450786+0000", 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054195520000000002 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450832+0000", 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6248999999999997e-05 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:85 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-03-16T19:31:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.88410481, 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050706680000000004, 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445794+0000", 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445795+0000", 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.15e-07 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445800+0000", 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.967e-06 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445823+0000", 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3411999999999999e-05 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445826+0000", 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1920000000000001e-06 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445882+0000", 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5436000000000003e-05 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.445901+0000", 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023999999999999e-05 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450835+0000", 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049347210000000004 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450865+0000", 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9601e-05 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:86 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'51)", 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.88346186499999, 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004454576, 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446437+0000", 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446438+0000", 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3399999999999998e-07 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446443+0000", 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1449999999999999e-06 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446455+0000", 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1824000000000001e-05 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446460+0000", 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4160000000000003e-06 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446505+0000", 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446524+0000", 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9021e-05 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450866+0000", 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043426740000000004 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450892+0000", 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5471999999999999e-05 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:87 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.88295790399999, 2024-03-16T19:31:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003979799, 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446941+0000", 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446942+0000", 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5099999999999999e-07 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.446998+0000", 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6465000000000002e-05 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447010+0000", 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447016+0000", 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5649999999999998e-06 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447075+0000", 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9147000000000001e-05 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.447094+0000", 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8488e-05 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450893+0000", 2024-03-16T19:31:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037992880000000001 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:17.450921+0000", 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7956000000000001e-05 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4203.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'106)", 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 139.724281502, 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082665380000000004, 2024-03-16T19:31:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605618+0000", 2024-03-16T19:31:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7199999999999999e-07 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605621+0000", 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7269999999999999e-06 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605627+0000", 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1469999999999998e-06 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605630+0000", 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3160000000000001e-06 2024-03-16T19:31:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605659+0000", 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8546000000000001e-05 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.605669+0000", 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0149999999999999e-05 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613806+0000", 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081367179999999994 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:18.613884+0000", 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8362000000000006e-05 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.972312243, 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077008600000000003, 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357587+0000", 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357609+0000", 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4429999999999997e-06 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357657+0000", 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849000000000002e-05 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357667+0000", 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0208e-05 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357869+0000", 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202229 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357967+0000", 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7765999999999999e-05 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115946 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362263+0000", 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041798599999999997 2024-03-16T19:31:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365196+0000", 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029334740000000002 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365246+0000", 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0127000000000003e-05 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365288+0000", 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1711000000000001e-05 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.97228770700001, 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078147860000000006, 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357611+0000", 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357612+0000", 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357615+0000", 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6630000000000002e-06 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357695+0000", 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0266999999999996e-05 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357701+0000", 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358344+0000", 2024-03-16T19:31:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064315599999999998 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358416+0000", 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1899999999999999e-05 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012741599999999999 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362270+0000", 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037268370000000002 2024-03-16T19:31:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365333+0000", 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030631949999999999 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365392+0000", 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8199999999999998e-05 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365426+0000", 2024-03-16T19:31:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4632999999999999e-05 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.97228317699999, 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074250339999999996, 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357616+0000", 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7000000000000001e-07 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357624+0000", 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7349999999999996e-06 2024-03-16T19:31:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357714+0000", 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0428999999999998e-05 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357720+0000", 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4689999999999996e-06 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358771+0000", 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010506039999999999 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358893+0000", 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122438 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359040+0000", 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147088 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362287+0000", 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003246657 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364943+0000", 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026561380000000002 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6243999999999999e-05 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365041+0000", 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1962000000000002e-05 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.97227336500001, 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079450319999999994, 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357626+0000", 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357629+0000", 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357740+0000", 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111344 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357745+0000", 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5450000000000003e-06 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358765+0000", 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010199219999999999 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011216499999999999 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358985+0000", 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107577 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362710+0000", 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037245939999999999 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365481+0000", 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027715980000000001 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365524+0000", 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3000000000000002e-05 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365571+0000", 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6609e-05 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.972269512, 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007999028, 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357630+0000", 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-07 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357632+0000", 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0420000000000001e-06 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357758+0000", 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012591600000000001 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357763+0000", 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6369999999999998e-06 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359304+0000", 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001541433 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359413+0000", 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108961 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359520+0000", 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010696699999999999 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362795+0000", 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032749599999999999 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365569+0000", 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027740640000000001 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365598+0000", 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8782e-05 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365629+0000", 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0987e-05 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.377 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.97226619600001, 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080638199999999993, 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357633+0000", 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8200000000000001e-07 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357637+0000", 2024-03-16T19:31:38.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6160000000000002e-06 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357773+0000", 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013625900000000001 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357778+0000", 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8729999999999998e-06 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359568+0000", 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017900699999999999 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359667+0000", 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9193999999999999e-05 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.365697+0000", 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060295260000000003 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:2 2.f e21/18 2:f1f69726:::400.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.97107061299999, 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039420669999999996, 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358829+0000", 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358830+0000", 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3399999999999997e-07 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358836+0000", 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6189999999999996e-06 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358848+0000", 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1505000000000001e-05 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358856+0000", 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1669999999999999e-06 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359318+0000", 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046209400000000002 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359334+0000", 2024-03-16T19:31:38.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892e-05 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362729+0000", 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003394963 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362771+0000", 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1893e-05 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.385 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4588.0:1 3.32 3.2b4ef7b2 (undecoded) ondisk+read+known_if_redirected e21)", 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:29.176862+0000", 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.1530371319999997, 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036772179999999999, 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4588", 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/2627359036", 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176862+0000", 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176862+0000", 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176864+0000", 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8640000000000001e-06 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176869+0000", 2024-03-16T19:31:38.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849999999999999e-06 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176887+0000", 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8048999999999999e-05 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176897+0000", 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0441999999999999e-05 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.180528+0000", 2024-03-16T19:31:38.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036310629999999999 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.180539+0000", 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1015e-05 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4588.0:2 3.32 3.2b4ef7b2 (undecoded) ondisk+write+known_if_redirected e21)", 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:29.176872+0000", 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.1530276159999993, 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037192620000000001, 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:38.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4588", 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/2627359036", 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176872+0000", 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176872+0000", 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176872+0000", 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1899999999999999e-07 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176873+0000", 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-06 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176927+0000", 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4358999999999999e-05 2024-03-16T19:31:38.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.176934+0000", 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5039999999999999e-06 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.180586+0000", 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036517640000000001 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.180591+0000", 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3020000000000002e-06 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.393 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.393 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:38.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-16T19:31:38.394 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:38.413+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:38.494 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:38.513+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:38.501 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:38.501 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:38.501 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:38.501 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:38.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.528+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f83e4062200 0x7f83e415bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.528+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7f83e4062200 0x7f83e415bf10 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.509 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.529+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f83e40c86b0 con 0x7f83e4062200 2024-03-16T19:31:38.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-16T19:31:38.523 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.543+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f83e4062890 con 0x7f83e4062200 2024-03-16T19:31:38.525 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.545+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f83e4062890 con 0x7f83e4062200 2024-03-16T19:31:38.525 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4: {} 2024-03-16T19:31:38.526 INFO:teuthology.orchestra.run.smithi172.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-16T19:31:38.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.545+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f83e415c960 0x7f83e402bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.545+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83e41538a0 con 0x7f83e415c960 2024-03-16T19:31:38.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.545+0000 7f83ea37e640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f83e415c960 0x7f83e402bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.546+0000 7f83ea37e640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f83e415c960 0x7f83e402bde0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.547+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f83e41538a0 con 0x7f83e415c960 2024-03-16T19:31:38.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.561+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f83e41538a0 con 0x7f83e415c960 2024-03-16T19:31:38.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.563+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f83e41538a0 con 0x7f83e415c960 2024-03-16T19:31:38.543 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5: {} 2024-03-16T19:31:38.544 INFO:teuthology.orchestra.run.smithi172.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-16T19:31:38.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.563+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f83e402c830 0x7f83e402cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.563+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83e406d8c0 con 0x7f83e402c830 2024-03-16T19:31:38.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.564+0000 7f83e9b7d640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f83e402c830 0x7f83e402cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.564+0000 7f83e9b7d640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f83e402c830 0x7f83e402cc90 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.544 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.565+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f83e406d8c0 con 0x7f83e402c830 2024-03-16T19:31:38.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.579+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f83e406d8c0 con 0x7f83e402c830 2024-03-16T19:31:38.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.580+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f83e406d8c0 con 0x7f83e402c830 2024-03-16T19:31:38.560 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6: {} 2024-03-16T19:31:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-16T19:31:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.581+0000 7f83ec5e2640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f83e4207830 0x7f83e4209c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.581+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83e402d380 con 0x7f83e4207830 2024-03-16T19:31:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.581+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f83e4207830 0x7f83e4209c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.581+0000 7f83eab7f640 1 --2- 172.21.15.172:0/268350722 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f83e4207830 0x7f83e4209c20 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:38.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.582+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f83e402d380 con 0x7f83e4207830 2024-03-16T19:31:38.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.596+0000 7f83ec5e2640 1 -- 172.21.15.172:0/268350722 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f83e402d350 con 0x7f83e4207830 2024-03-16T19:31:38.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:38.598+0000 7f83daffd640 1 -- 172.21.15.172:0/268350722 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f83e402d350 con 0x7f83e4207830 2024-03-16T19:31:38.578 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7: {} 2024-03-16T19:31:38.578 INFO:teuthology.orchestra.run.smithi172.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-16T19:31:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-16T19:31:38.581 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-16T19:31:38.595 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:38.614+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:38.617 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:38.617 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:38.617 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:38.617 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:38.617 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:38.626 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-16T19:31:38.698 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:38.717+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:38.774 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:38.774 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:38.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:38.774 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:38.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.43618977200001, 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052245640000000001, 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357337+0000", 2024-03-16T19:31:38.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357344+0000", 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0400000000000004e-06 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357351+0000", 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8990000000000004e-06 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357396+0000", 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5714999999999999e-05 2024-03-16T19:31:38.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357409+0000", 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3045e-05 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357498+0000", 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8387999999999995e-05 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357621+0000", 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012360600000000001 2024-03-16T19:31:38.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357739+0000", 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011739 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359995+0000", 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022557739999999999 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362462+0000", 2024-03-16T19:31:38.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002467534 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362511+0000", 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9348999999999998e-05 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362561+0000", 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9824000000000003e-05 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.779 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.436172658, 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054979800000000004, 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:38.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357354+0000", 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4499999999999998e-07 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357357+0000", 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4619999999999999e-06 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357446+0000", 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9377999999999997e-05 2024-03-16T19:31:38.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2929999999999997e-06 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358021+0000", 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000569182 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358090+0000", 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8399999999999996e-05 2024-03-16T19:31:38.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358211+0000", 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121286 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359999+0000", 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017880260000000001 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362759+0000", 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027594350000000002 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362814+0000", 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5083000000000001e-05 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362852+0000", 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090000000000003e-05 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.784 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.436165378, 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055859129999999996, 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357361+0000", 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357373+0000", 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1909000000000001e-05 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357467+0000", 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4135999999999995e-05 2024-03-16T19:31:38.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357472+0000", 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0839999999999999e-06 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359043+0000", 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015704040000000001 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359145+0000", 2024-03-16T19:31:38.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102295 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359266+0000", 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121178 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362714+0000", 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034474739999999999 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362888+0000", 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017408500000000001 2024-03-16T19:31:38.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362928+0000", 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0534999999999997e-05 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362947+0000", 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8535000000000001e-05 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.789 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.43615151100001, 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053283640000000004, 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357375+0000", 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357378+0000", 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7970000000000001e-06 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357493+0000", 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114881 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3719999999999998e-06 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358417+0000", 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092010900000000001 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358514+0000", 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6082000000000006e-05 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358628+0000", 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114543 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360003+0000", 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013747550000000001 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362612+0000", 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026092760000000002 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6354999999999998e-05 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362703+0000", 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4832999999999997e-05 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:6 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.435478676, 2024-03-16T19:31:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017506430000000001, 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358048+0000", 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358049+0000", 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358057+0000", 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0839999999999992e-06 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358068+0000", 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1518999999999999e-05 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358075+0000", 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358135+0000", 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9416000000000002e-05 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358157+0000", 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1722999999999999e-05 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359762+0000", 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001605914 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359798+0000", 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5967000000000003e-05 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:7 2.10 e21/18 2:0d82a743:::600.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:38.798 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.43318042600001, 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044706099999999999, 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360346+0000", 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360347+0000", 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.23e-07 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360353+0000", 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.353e-06 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360363+0000", 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0362e-05 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360372+0000", 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0650000000000005e-06 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360417+0000", 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4027999999999999e-05 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7669999999999999e-05 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364776+0000", 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043417070000000002 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364817+0000", 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701999999999998e-05 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:8 2.10 e21/18 2:09401a81:::601.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.43313532499999, 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044556020000000003, 2024-03-16T19:31:38.802 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360391+0000", 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7799999999999997e-07 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360395+0000", 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0260000000000001e-06 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360403+0000", 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2049999999999996e-06 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360407+0000", 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5909999999999998e-06 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360604+0000", 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196632 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360623+0000", 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9394000000000001e-05 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364820+0000", 2024-03-16T19:31:38.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041966909999999998 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364847+0000", 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6784999999999999e-05 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:9 2.10 e21/18 2:09401a81:::601.00000000:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.43310726999999, 2024-03-16T19:31:38.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044568899999999998, 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360419+0000", 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1199999999999999e-07 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360424+0000", 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6099999999999999e-06 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360430+0000", 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6579999999999999e-06 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360434+0000", 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1350000000000002e-06 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360824+0000", 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038990500000000002 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360843+0000", 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712e-05 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364848+0000", 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040048130000000003 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364876+0000", 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7744999999999999e-05 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:10 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'5, mlcod=0'0)", 2024-03-16T19:31:38.810 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.43308585600001, 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044643449999999998, 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360441+0000", 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360444+0000", 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8049999999999998e-06 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360449+0000", 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5300000000000004e-06 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360454+0000", 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5460000000000002e-06 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361028+0000", 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057449200000000001 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361047+0000", 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8804000000000002e-05 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364877+0000", 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003830436 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364905+0000", 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7508000000000002e-05 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:11 2.10 e21/18 2:0bd6d154:::602.00000000:head v 21'6, mlcod=0'0)", 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.43306350500001, 2024-03-16T19:31:38.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044742540000000004, 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360463+0000", 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0600000000000001e-07 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360467+0000", 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2990000000000001e-06 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360472+0000", 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8289999999999996e-06 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360476+0000", 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7249999999999999e-06 2024-03-16T19:31:38.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361188+0000", 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00071183800000000001 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361200+0000", 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2235999999999999e-05 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364909+0000", 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037086570000000002 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364937+0000", 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8364000000000002e-05 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:12 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'7, mlcod=0'0)", 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.433043106, 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044815970000000004, 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360483+0000", 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360484+0000", 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.99e-07 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360486+0000", 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360491+0000", 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3000000000000001e-06 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360495+0000", 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8079999999999998e-06 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361314+0000", 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081868999999999996 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5792000000000001e-05 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364939+0000", 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036087760000000002 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364965+0000", 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6392000000000001e-05 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:13 2.10 e21/18 2:0d89b25e:::603.00000000:head v 21'8, mlcod=0'0)", 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.432937114, 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043994070000000001, 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360589+0000", 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1500000000000004e-07 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360596+0000", 2024-03-16T19:31:38.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.117e-06 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360607+0000", 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798000000000001e-05 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360614+0000", 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6710000000000002e-06 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361502+0000", 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088875 2024-03-16T19:31:38.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361522+0000", 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9830000000000002e-05 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364967+0000", 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003444786 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364989+0000", 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1739999999999999e-05 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:14 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.431145403, 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018658800000000001, 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362381+0000", 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362382+0000", 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2900000000000002e-07 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362386+0000", 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5990000000000004e-06 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362398+0000", 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362405+0000", 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2089999999999999e-06 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362449+0000", 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3698000000000001e-05 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362457+0000", 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362568+0000", 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111081 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:31 2.d e21/18 ondisk, result = 0)", 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:38.829 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.431097902, 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028962500000000001, 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362429+0000", 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.58e-07 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362431+0000", 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4719999999999998e-06 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362439+0000", 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4989999999999997e-06 2024-03-16T19:31:38.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362448+0000", 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0469999999999999e-06 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362602+0000", 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015380099999999999 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362607+0000", 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6160000000000001e-06 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362718+0000", 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110832 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:15 2.15 e21/18 ondisk, result = 0)", 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.43085865099999, 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019029100000000001, 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362668+0000", 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8100000000000003e-07 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362672+0000", 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5039999999999998e-06 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362681+0000", 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3600000000000002e-06 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362687+0000", 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7300000000000002e-06 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362745+0000", 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7569999999999999e-05 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362752+0000", 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362858+0000", 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105627 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:39 2.6 2:654134d2:::mds0_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 122.422360506, 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014856890000000001, 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371166+0000", 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371167+0000", 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-06 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371173+0000", 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7710000000000003e-06 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371191+0000", 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8337000000000001e-05 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371216+0000", 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4678e-05 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371299+0000", 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2756999999999994e-05 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371396+0000", 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7581e-05 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371477+0000", 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0668999999999996e-05 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.371979+0000", 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050189500000000003 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372587+0000", 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060842999999999997 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372618+0000", 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1004999999999999e-05 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:36.372652+0000", 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3337999999999998e-05 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:46 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e36)", 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.418029648000001, 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014027969999999999, 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375497+0000", 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375500+0000", 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.807e-06 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375517+0000", 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7867e-05 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375581+0000", 2024-03-16T19:31:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831000000000006e-05 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375598+0000", 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6716999999999999e-05 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375742+0000", 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014393000000000001 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375856+0000", 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113997 2024-03-16T19:31:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.375972+0000", 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115827 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376609+0000", 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063765700000000002 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376835+0000", 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022546099999999999 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376867+0000", 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1973e-05 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:19.376900+0000", 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.273e-05 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:47 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals,omap-rm-keys] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-03-16T19:31:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 73.417398845999998, 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001653439, 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376128+0000", 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376130+0000", 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4159999999999998e-06 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376141+0000", 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.097e-05 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376183+0000", 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2194999999999998e-05 2024-03-16T19:31:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376199+0000", 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5840000000000001e-05 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376323+0000", 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012406099999999999 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376435+0000", 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011244200000000001 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.376555+0000", 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119117 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377214+0000", 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065930399999999999 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377717+0000", 2024-03-16T19:31:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050282400000000002 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377752+0000", 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5337999999999999e-05 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:30:25.377781+0000", 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8932e-05 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:48 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e77)", 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:29.381986+0000", 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.4115406410000002, 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001303532, 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.381986+0000", 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.381986+0000", 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.381988+0000", 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.869e-06 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.381997+0000", 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6730000000000004e-06 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382025+0000", 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8092000000000001e-05 2024-03-16T19:31:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382038+0000", 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2333e-05 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382109+0000", 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1104000000000002e-05 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382176+0000", 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7533999999999994e-05 2024-03-16T19:31:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382274+0000", 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7132000000000004e-05 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.382979+0000", 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070523000000000005 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383190+0000", 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021099899999999999 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383243+0000", 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3579999999999999e-05 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383289+0000", 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5986000000000003e-05 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(mds.0.4:48 2.6 e76/18 ondisk, result = 0)", 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:29.383116+0000", 2024-03-16T19:31:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.4104103709999993, 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017887099999999999, 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383116+0000", 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383116+0000", 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383117+0000", 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5700000000000002e-07 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383120+0000", 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5410000000000001e-06 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383131+0000", 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0866999999999999e-05 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383136+0000", 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0699999999999997e-06 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383174+0000", 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8293000000000001e-05 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383182+0000", 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5089999999999996e-06 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:29.383295+0000", 2024-03-16T19:31:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112934 2024-03-16T19:31:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:38.860 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:38.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-16T19:31:38.861 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:38.880+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:38.962 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:38.981+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:38.976 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:38.976 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:38.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-16T19:31:39.062 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:31:39.082+0000 7ff53dd8b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38406) UID: 0 2024-03-16T19:31:39.092 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:39.093 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:39.093 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 180, 2024-03-16T19:31:39.093 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-16T19:31:39.093 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:39.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-16T19:31:39.163 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:39.182+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:39.210 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:39.210 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-16T19:31:39.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-16T19:31:39.210 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-16T19:31:39.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.872215321, 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040458179999999996, 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357449+0000", 2024-03-16T19:31:39.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357452+0000", 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4230000000000001e-06 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357459+0000", 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.844e-06 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357487+0000", 2024-03-16T19:31:39.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7580999999999999e-05 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357497+0000", 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357601+0000", 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010423 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357707+0000", 2024-03-16T19:31:39.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105307 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361495+0000", 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037880549999999998 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.87220084800001, 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050336540000000003, 2024-03-16T19:31:39.214 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357463+0000", 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357464+0000", 2024-03-16T19:31:39.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357466+0000", 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.855e-06 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357530+0000", 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4432999999999998e-05 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357537+0000", 2024-03-16T19:31:39.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.703e-06 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357599+0000", 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2129000000000001e-05 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357704+0000", 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010519899999999999 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362497+0000", 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047929340000000004 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.217 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:18 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.871696359, 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004314755, 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357968+0000", 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357969+0000", 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8000000000000004e-07 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357988+0000", 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9279999999999998e-05 2024-03-16T19:31:39.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357998+0000", 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9820000000000003e-06 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358005+0000", 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.798e-06 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358058+0000", 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3522e-05 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358083+0000", 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4845999999999999e-05 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362184+0000", 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041010090000000001 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362283+0000", 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8437999999999995e-05 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:22 2.e e21/18 2:717a0223:::608.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.871680963, 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041796009999999998, 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357983+0000", 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357984+0000", 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.68e-07 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.357990+0000", 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4160000000000001e-06 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358000+0000", 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4609999999999997e-06 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358006+0000", 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4930000000000003e-06 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358278+0000", 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000271401 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:39.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.579e-05 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038198059999999998 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362163+0000", 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9565999999999999e-05 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.225 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:23 2.e e21/18 2:717a0223:::608.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.87122228, 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038120720000000001, 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358442+0000", 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358443+0000", 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.37e-07 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358447+0000", 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5580000000000002e-06 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358456+0000", 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909999999999993e-06 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358462+0000", 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3960000000000003e-06 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358506+0000", 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4197999999999999e-05 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358526+0000", 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9854999999999999e-05 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362165+0000", 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036387279999999999 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362254+0000", 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9009000000000001e-05 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:19 2.14 e21/18 2:2f55791f:::606.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:39.229 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.87067611000001, 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039823419999999998, 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358988+0000", 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358989+0000", 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9109999999999998e-06 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359008+0000", 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.366e-05 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359015+0000", 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359361+0000", 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034562800000000001 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359380+0000", 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8862e-05 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362930+0000", 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003550419 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362970+0000", 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0561e-05 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:1 2.14 e21/18 2:292cf221:::200.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.870119019, 2024-03-16T19:31:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003449012, 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359545+0000", 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359546+0000", 2024-03-16T19:31:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359557+0000", 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1113e-05 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359566+0000", 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8890000000000008e-06 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359572+0000", 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0959999999999997e-06 2024-03-16T19:31:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359668+0000", 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6508000000000005e-05 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.359685+0000", 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6818000000000001e-05 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362973+0000", 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003287749 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362994+0000", 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1390999999999999e-05 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:3 2.14 e21/18 2:287034c4:::1.00000000.inode:head v 21'4, mlcod=0'0)", 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.868917271, 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040265470000000001, 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360747+0000", 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360748+0000", 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-07 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360754+0000", 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0700000000000003e-06 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360766+0000", 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2068e-05 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360772+0000", 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4690000000000002e-06 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360822+0000", 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699e-05 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.360844+0000", 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1984000000000002e-05 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364734+0000", 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389025 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364774+0000", 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9313999999999997e-05 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1b epoch 21/18 rep_tid 1 lcod 21'1)", 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.86833856200001, 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017400599999999999, 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361326+0000", 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.13e-07 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361330+0000", 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1250000000000003e-06 2024-03-16T19:31:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361340+0000", 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3700000000000001e-06 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9139999999999999e-06 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361393+0000", 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8288000000000003e-05 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361500+0000", 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106696 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(2.14 epoch 21/18 rep_tid 7 entries 21'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.868289035, 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034163900000000001, 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361375+0000", 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361376+0000", 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1399999999999997e-07 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361383+0000", 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.221e-06 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361395+0000", 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2058999999999999e-05 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.361408+0000", 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.308e-05 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362078+0000", 2024-03-16T19:31:39.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067025500000000003 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.364792+0000", 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027133610000000001 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing_reply(2.1e epoch 21/18 rep_tid 0 lcod 21'1)", 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.867328392, 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165298, 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362337+0000", 2024-03-16T19:31:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999997e-07 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362340+0000", 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6660000000000001e-06 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362350+0000", 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6059999999999992e-06 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362356+0000", 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0979999999999999e-06 2024-03-16T19:31:39.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362404+0000", 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8146000000000003e-05 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.362501+0000", 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7082000000000003e-05 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:38 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'6, mlcod=21'4)", 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.86361419399999, 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065734300000000001, 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366050+0000", 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366051+0000", 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9500000000000002e-07 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366056+0000", 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6609999999999997e-06 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366065+0000", 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1710000000000008e-06 2024-03-16T19:31:39.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366071+0000", 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0930000000000002e-06 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366113+0000", 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2302999999999998e-05 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366132+0000", 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9471e-05 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366670+0000", 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053722399999999999 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:31.366707+0000", 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7824999999999998e-05 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:40 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'7, mlcod=21'5)", 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:39.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 119.42321328200001, 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00094280999999999998, 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806451+0000", 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806453+0000", 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806463+0000", 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806481+0000", 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8233e-05 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806492+0000", 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0798999999999999e-05 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806558+0000", 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6304999999999996e-05 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.806576+0000", 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8179999999999999e-05 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807360+0000", 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00078339299999999996 2024-03-16T19:31:39.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:39.807394+0000", 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4143999999999997e-05 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:41 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'8, mlcod=21'6)", 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 117.857660279, 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072340700000000002, 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.258 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372004+0000", 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372005+0000", 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372017+0000", 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1423999999999999e-05 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372033+0000", 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6022e-05 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372043+0000", 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0083e-05 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372105+0000", 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2500000000000001e-05 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372123+0000", 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7782999999999999e-05 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372696+0000", 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057261900000000001 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:41.372727+0000", 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1816000000000002e-05 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:42 2.14 e21/18 2:2e2fa760:::200.00000001:head v 21'9, mlcod=21'7)", 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 116.35006155400001, 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012172629999999999, 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879603+0000", 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879605+0000", 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8950000000000001e-06 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879620+0000", 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5809999999999999e-05 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879646+0000", 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6040000000000001e-05 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879665+0000", 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8144000000000001e-05 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879765+0000", 2024-03-16T19:31:39.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010021100000000001 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.879794+0000", 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8875000000000001e-05 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880768+0000", 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097426900000000005 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:29:42.880820+0000", 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2018999999999998e-05 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPull(2.e e67/66 cost 1048576)", 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.445387644999997, 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026662799999999998, 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784277+0000", 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0500000000000003e-07 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784292+0000", 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5139e-05 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784304+0000", 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1501000000000001e-05 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784313+0000", 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8189999999999998e-06 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784384+0000", 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1192999999999994e-05 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:05.784543+0000", 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159371 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:49 2.14 e77/74 2:2e2fa760:::200.00000001:head v 77'12, mlcod=36'10)", 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:31.373368+0000", 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.8562958370000002, 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011458099999999999, 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.373368+0000", 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.373368+0000", 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.373369+0000", 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2799999999999995e-07 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.373381+0000", 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1962e-05 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.373395+0000", 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3839000000000001e-05 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.373403+0000", 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0679999999999997e-06 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.373467+0000", 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3561999999999999e-05 2024-03-16T19:31:39.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.373490+0000", 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3427000000000001e-05 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374460+0000", 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00097031799999999996 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374514+0000", 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3805999999999999e-05 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.272 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(mds.0.4:50 2.14 e77/74 2:292cf221:::200.00000000:head v 77'13, mlcod=36'10)", 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:31.374012+0000", 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.855652504, 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072767399999999999, 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374012+0000", 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374012+0000", 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374012+0000", 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999996e-07 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374017+0000", 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5140000000000003e-06 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374027+0000", 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8989999999999996e-06 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374031+0000", 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4449999999999998e-06 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374082+0000", 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0630999999999997e-05 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374104+0000", 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1744999999999999e-05 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374702+0000", 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00059820900000000005 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:31.374739+0000", 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7626999999999997e-05 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(2.13 81/80 [PushOp(2:cd5a64a3:::100.00000000:head, version: 21'1, data_included: [], data_size: 0, omap_header_size: 274, omap_entries_size: 10, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:cd5a64a3:::100.00000000:head@21'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:38.907647+0000", 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.32201700300000002, 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041587800000000001, 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.907647+0000", 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.907647+0000", 2024-03-16T19:31:39.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.907648+0000", 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1699999999999998e-07 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.907677+0000", 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9614000000000001e-05 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.907694+0000", 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6895999999999999e-05 2024-03-16T19:31:39.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.907712+0000", 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7612e-05 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.907797+0000", 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5233000000000002e-05 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.907810+0000", 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.313e-05 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.908063+0000", 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025287600000000002 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(2.13 81/80 [PushOp(2:c91d4a1d:::mds0_inotable:head, version: 21'2, data_included: [0~34], data_size: 34, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c91d4a1d:::mds0_inotable:head@21'2, size: 34, copy_subset: [(0, 34)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 34, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-16T19:31:38.910343+0000", 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.31932120800000002, 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051692100000000005, 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-16T19:31:39.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.910343+0000", 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.910343+0000", 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.910344+0000", 2024-03-16T19:31:39.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2300000000000001e-07 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.910355+0000", 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1769e-05 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.910372+0000", 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6175e-05 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.910384+0000", 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2464e-05 2024-03-16T19:31:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.910465+0000", 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0952999999999996e-05 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.910479+0000", 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4022999999999999e-05 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-16T19:31:38.910860+0000", 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038101399999999999 2024-03-16T19:31:39.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.284 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-16T19:31:39.284 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-16T19:31:39.284 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:39.285 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-16T19:31:39.285 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:31:39.305+0000 7fbcd8ae2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37740) UID: 0 2024-03-16T19:31:39.386 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:39.405+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:39.391 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:39.392 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:39.392 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:39.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-16T19:31:39.487 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:31:39.507+0000 7fda8712b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37737) UID: 0 2024-03-16T19:31:39.546 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:39.546 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:39.546 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:39.546 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:39.546 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:39.555 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-16T19:31:39.598 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:39.618+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:39.618 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-16T19:31:39.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T19:31:39.662 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:39.662 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:39.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:39.662 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.32490848399999, 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059387429999999998, 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357175+0000", 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357184+0000", 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7880000000000001e-06 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357194+0000", 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1060000000000006e-06 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357218+0000", 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4358999999999998e-05 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357242+0000", 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4051999999999999e-05 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357431+0000", 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000188739 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018728399999999999 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357772+0000", 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015376100000000001 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359771+0000", 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019989830000000002 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363009+0000", 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032384530000000001 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363067+0000", 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7941e-05 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363113+0000", 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6276999999999998e-05 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.667 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.1a epoch 21/18 rep_tid 2 entries 21'1 (0'0) error 2:5e41ab69:::200.00000005:head by mds.0.4:36 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.32368106199999, 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014118799999999999, 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358402+0000", 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358403+0000", 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1429999999999999e-06 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358414+0000", 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0845999999999999e-05 2024-03-16T19:31:39.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358436+0000", 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1682999999999999e-05 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358450+0000", 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4239999999999999e-05 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358559+0000", 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010880899999999999 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359814+0000", 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001255159 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:31 2.d e21/18 2:b50e409b:::mds_snaptable:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.323278356, 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00330116, 2024-03-16T19:31:39.671 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358805+0000", 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358806+0000", 2024-03-16T19:31:39.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4479999999999999e-06 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358877+0000", 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0560000000000002e-05 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358889+0000", 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1783999999999999e-05 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358897+0000", 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1359999999999997e-06 2024-03-16T19:31:39.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358969+0000", 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2611e-05 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358995+0000", 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5446999999999999e-05 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362056+0000", 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030613210000000001 2024-03-16T19:31:39.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362106+0000", 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9852999999999999e-05 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:28 2.13 e21/18 2:cd5a64a3:::100.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.32266960499999, 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005311395, 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.675 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359414+0000", 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3900000000000004e-07 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359427+0000", 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2293000000000001e-05 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359439+0000", 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1945e-05 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359446+0000", 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7500000000000003e-06 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359906+0000", 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045915099999999998 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359933+0000", 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7404000000000001e-05 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364679+0000", 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0047458359999999998 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364725+0000", 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6177000000000002e-05 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:29 2.b e21/18 2:d0630e4c:::mds0_sessionmap:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.32222573600001, 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049093440000000004, 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.679 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359857+0000", 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359858+0000", 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1699999999999997e-07 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359868+0000", 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2269999999999999e-06 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359972+0000", 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104444 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359981+0000", 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3999999999999998e-06 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360110+0000", 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012870500000000001 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360207+0000", 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7249000000000004e-05 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364730+0000", 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045230679999999999 2024-03-16T19:31:39.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364767+0000", 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6334000000000001e-05 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "pg_update_log_missing(2.b epoch 21/18 rep_tid 120 entries 21'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.32208295999999, 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004781352, 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:39.683 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360000+0000", 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360001+0000", 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360010+0000", 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0769999999999997e-06 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360029+0000", 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8927e-05 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360042+0000", 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2841e-05 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360508+0000", 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046653500000000002 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364782+0000", 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004273471 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:27 2.13 e21/18 2:c91d4a1d:::mds0_inotable:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.322068419, 2024-03-16T19:31:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005372511, 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360015+0000", 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360016+0000", 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2470000000000001e-06 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360030+0000", 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4184e-05 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360043+0000", 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.275e-05 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360053+0000", 2024-03-16T19:31:39.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7680000000000003e-06 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360415+0000", 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036213499999999997 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360435+0000", 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9879000000000001e-05 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365336+0000", 2024-03-16T19:31:39.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004901529 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365387+0000", 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1019000000000001e-05 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(mds.0.4:2 2.f e21/18 ondisk, result = 0)", 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.31919072400001, 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00022989500000000001, 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.690 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362892+0000", 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362893+0000", 2024-03-16T19:31:39.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7700000000000003e-07 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362898+0000", 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000004e-06 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362923+0000", 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4604e-05 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362932+0000", 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1679999999999993e-06 2024-03-16T19:31:39.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362993+0000", 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1049000000000005e-05 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363004+0000", 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0335e-05 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363122+0000", 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011877199999999999 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(2.13 81/80 [PushReplyOp(2:cd5a64a3:::100.00000000:head)])", 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:38.909339+0000", 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.77274375900000003, 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00034751699999999999, 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909339+0000", 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909339+0000", 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909340+0000", 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1099999999999995e-07 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909345+0000", 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6310000000000002e-06 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909356+0000", 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0899e-05 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909365+0000", 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3419999999999997e-06 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909438+0000", 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3161e-05 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909687+0000", 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00024887299999999999 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(2.13 81/80 [PushReplyOp(2:c91d4a1d:::mds0_inotable:head)])", 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:38.916021+0000", 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.76606175899999995, 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027596700000000001, 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916021+0000", 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916021+0000", 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916022+0000", 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6100000000000002e-07 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916025+0000", 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8179999999999997e-06 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916032+0000", 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.922e-06 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916042+0000", 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2019999999999993e-06 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916090+0000", 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8418000000000003e-05 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916297+0000", 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00020734599999999999 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPush(2.1f 81/80 [PushOp(2:ff5b34d6:::1.00000000:head, version: 21'2, data_included: [], data_size: 0, omap_header_size: 274, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(2:ff5b34d6:::1.00000000:head@21'2, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-16T19:31:39.700 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:38.916341+0000", 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.76574234799999996, 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00031574199999999999, 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916341+0000", 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916341+0000", 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916341+0000", 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4099999999999999e-07 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916355+0000", 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3859000000000001e-05 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916372+0000", 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6838e-05 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916388+0000", 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5719999999999999e-05 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916466+0000", 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8586999999999998e-05 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916477+0000", 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1151999999999999e-05 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.916657+0000", 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00017904499999999999 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPush(2.1f 81/80 [PushOp(2:fe07bba1:::100.00000000.inode:head, version: 21'3, data_included: [0~582], data_size: 582, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:fe07bba1:::100.00000000.inode:head@21'3, size: 582, copy_subset: [(0, 582)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 582, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:38.917907+0000", 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.76417599599999997, 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000374311, 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.917907+0000", 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.917907+0000", 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.917908+0000", 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9999999999999997e-07 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.917918+0000", 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0465999999999999e-05 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.917935+0000", 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6597999999999999e-05 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.917950+0000", 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4805e-05 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918040+0000", 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0334000000000006e-05 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918050+0000", 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9510000000000001e-06 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918281+0000", 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023155699999999999 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.707 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(2.f 81/80 [PushReplyOp(2:f1f69726:::400.00000000:head)])", 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:38.918242+0000", 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.76384131300000002, 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000148901, 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918242+0000", 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918242+0000", 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918242+0000", 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6100000000000002e-07 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918244+0000", 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3889999999999999e-06 2024-03-16T19:31:39.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918249+0000", 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8679999999999998e-06 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918255+0000", 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0449999999999997e-06 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:39.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918280+0000", 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4613e-05 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.918391+0000", 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011072499999999999 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:39.711 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:39.712 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-16T19:31:39.713 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:31:39.731+0000 7fb3a8878640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38400) UID: 0 2024-03-16T19:31:39.813 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:39.832+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:39.860 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:39.860 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:39.860 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:39.860 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:39.869 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-16T19:31:39.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:31:39.953 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:39.972+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:39.955 INFO:tasks.workunit:Stopping ['fs/misc/trivial_sync.sh'] on client.0... 2024-03-16T19:31:39.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-16T19:31:40.005 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.025+0000 7f258bab5640 1 Processor -- start 2024-03-16T19:31:40.006 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.025+0000 7f258bab5640 1 -- start start 2024-03-16T19:31:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f258bab5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2584152a20 0x7f258406f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f258bab5640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f25840ce4e0 0x7f25840ce950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f258bab5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25840c9cf0 con 0x7f25840b0de0 2024-03-16T19:31:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f258bab5640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25840c9f90 con 0x7f25840ce4e0 2024-03-16T19:31:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f258bab5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25840ca250 con 0x7f2584152a20 2024-03-16T19:31:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f258a052640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f25840ce4e0 0x7f25840ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f2589851640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2584152a20 0x7f258406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f2589851640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2584152a20 0x7f258406f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:58792/0 (socket says 172.21.15.172:58792) 2024-03-16T19:31:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f2589050640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f25840b0de0 0x7f258415c560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:52926/0 (socket says 172.21.15.172:52926) 2024-03-16T19:31:40.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f2589851640 1 -- 172.21.15.172:0/2755820229 learned_addr learned my addr 172.21.15.172:0/2755820229 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:40.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.026+0000 7f258a052640 1 --2- 172.21.15.172:0/2755820229 >> v2:172.21.15.172:3300/0 conn(0x7f25840ce4e0 0x7f25840ce950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42606/0 (socket says 172.21.15.172:42606) 2024-03-16T19:31:40.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.027+0000 7f2589851640 1 -- 172.21.15.172:0/2755820229 >> v2:172.21.15.172:3300/0 conn(0x7f25840ce4e0 msgr2=0x7f25840ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:40.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.027+0000 7f2589851640 1 --2- 172.21.15.172:0/2755820229 >> v2:172.21.15.172:3300/0 conn(0x7f25840ce4e0 0x7f25840ce950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.027+0000 7f2589851640 1 -- 172.21.15.172:0/2755820229 >> v1:172.21.15.174:6789/0 conn(0x7f25840b0de0 legacy=0x7f258415c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:40.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.027+0000 7f2589851640 1 -- 172.21.15.172:0/2755820229 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25840ca530 con 0x7f2584152a20 2024-03-16T19:31:40.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.027+0000 7f258884f640 1 -- 172.21.15.172:0/2755820229 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382317378 0 0) 0x7f25840c9cf0 con 0x7f25840b0de0 2024-03-16T19:31:40.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.027+0000 7f258a052640 1 --2- 172.21.15.172:0/2755820229 >> v2:172.21.15.172:3300/0 conn(0x7f25840ce4e0 0x7f25840ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:40.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f2589851640 1 --2- 172.21.15.172:0/2755820229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2584152a20 0x7f258406f070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2584067b90 tx=0x7f257c02c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c91b3a1310aa4ad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258884f640 1 -- 172.21.15.172:0/2755820229 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c02cbb0 con 0x7f2584152a20 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258884f640 1 -- 172.21.15.172:0/2755820229 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c02cd10 con 0x7f2584152a20 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258884f640 1 -- 172.21.15.172:0/2755820229 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c034610 con 0x7f2584152a20 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258bab5640 1 -- 172.21.15.172:0/2755820229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2584152a20 msgr2=0x7f258406f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258bab5640 1 --2- 172.21.15.172:0/2755820229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2584152a20 0x7f258406f070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2584067b90 tx=0x7f257c02c010 comp rx=0 tx=0).stop 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258bab5640 1 -- 172.21.15.172:0/2755820229 shutdown_connections 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258bab5640 1 --2- 172.21.15.172:0/2755820229 >> v2:172.21.15.172:3300/0 conn(0x7f25840ce4e0 0x7f25840ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258bab5640 1 --2- 172.21.15.172:0/2755820229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2584152a20 0x7f258406f070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258bab5640 1 -- 172.21.15.172:0/2755820229 >> 172.21.15.172:0/2755820229 conn(0x7f2584066e00 msgr2=0x7f25840671e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258bab5640 1 -- 172.21.15.172:0/2755820229 shutdown_connections 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.028+0000 7f258bab5640 1 -- 172.21.15.172:0/2755820229 wait complete. 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.029+0000 7f258bab5640 1 Processor -- start 2024-03-16T19:31:40.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f258bab5640 1 -- start start 2024-03-16T19:31:40.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f258bab5640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25840b0de0 0x7f25840c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:40.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f258bab5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25840ce4e0 0x7f25840cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:40.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f258bab5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2584152a20 0x7f25841f1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:40.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f258bab5640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25840c6a60 con 0x7f25840b0de0 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f258bab5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25841eed10 con 0x7f25840ce4e0 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f258bab5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25841ee190 con 0x7f2584152a20 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f2589851640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25840b0de0 0x7f25840c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f2589851640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25840b0de0 0x7f25840c54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42614/0 (socket says 172.21.15.172:42614) 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f2589851640 1 -- 172.21.15.172:0/2099962814 learned_addr learned my addr 172.21.15.172:0/2099962814 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f258a052640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2584152a20 0x7f25841f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.030+0000 7f2589050640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25840ce4e0 0x7f25840cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f2589851640 1 -- 172.21.15.172:0/2099962814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2584152a20 msgr2=0x7f25841f1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f2589851640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2584152a20 0x7f25841f1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f2589851640 1 -- 172.21.15.172:0/2099962814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25840ce4e0 msgr2=0x7f25840cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f2589851640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25840ce4e0 0x7f25840cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f2589851640 1 -- 172.21.15.172:0/2099962814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2580009400 con 0x7f25840b0de0 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f2589851640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25840b0de0 0x7f25840c54d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f257c000f80 tx=0x7f257c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f257a7fc640 1 -- 172.21.15.172:0/2099962814 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c009510 con 0x7f25840b0de0 2024-03-16T19:31:40.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f257a7fc640 1 -- 172.21.15.172:0/2099962814 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c03e450 con 0x7f25840b0de0 2024-03-16T19:31:40.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c009b30 con 0x7f25840b0de0 2024-03-16T19:31:40.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f257a7fc640 1 -- 172.21.15.172:0/2099962814 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c03e600 con 0x7f25840b0de0 2024-03-16T19:31:40.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.031+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25841ee750 con 0x7f25840b0de0 2024-03-16T19:31:40.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.032+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2584075a70 con 0x7f25840b0de0 2024-03-16T19:31:40.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.033+0000 7f257a7fc640 1 -- 172.21.15.172:0/2099962814 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f257c03e760 con 0x7f25840b0de0 2024-03-16T19:31:40.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.033+0000 7f257a7fc640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f256806b3b0 0x7f256806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:40.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.033+0000 7f2589050640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f256806b3b0 0x7f256806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:40.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.033+0000 7f257a7fc640 1 -- 172.21.15.172:0/2099962814 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f257c0ac180 con 0x7f25840b0de0 2024-03-16T19:31:40.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.033+0000 7f2589050640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f256806b3b0 0x7f256806d870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f25841e8720 tx=0x7f2574007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:40.015 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:31:40.035+0000 7f3900bdc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37730) UID: 0 2024-03-16T19:31:40.017 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.037+0000 7f257a7fc640 1 -- 172.21.15.172:0/2099962814 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f257c076c70 con 0x7f25840b0de0 2024-03-16T19:31:40.021 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:40.021 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:40.021 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:40.021 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:40.021 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:40.030 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-16T19:31:40.116 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:31:40.135+0000 7f9f79944640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38401) UID: 0 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:14 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.839626952, 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039045479999999999, 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358260+0000", 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358261+0000", 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5540000000000001e-06 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358274+0000", 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2391999999999999e-05 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358294+0000", 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0542000000000001e-05 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358308+0000", 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3556e-05 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359511+0000", 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012031450000000001 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359529+0000", 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8210999999999999e-05 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362120+0000", 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025906509999999998 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362164+0000", 2024-03-16T19:31:40.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4496999999999998e-05 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:15 2.15 e21/18 2:a93a17c2:::604.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.839547834, 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042166870000000002, 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358339+0000", 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7100000000000002e-07 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358346+0000", 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7050000000000002e-06 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358357+0000", 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0613e-05 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358363+0000", 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0680000000000002e-06 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359789+0000", 2024-03-16T19:31:40.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014261709999999999 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359811+0000", 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1942000000000001e-05 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362515+0000", 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027040609999999998 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362555+0000", 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0655999999999998e-05 2024-03-16T19:31:40.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:20 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.83893370999999, 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00098915500000000003, 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:40.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358953+0000", 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358954+0000", 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3200000000000001e-06 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358963+0000", 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1540000000000008e-06 2024-03-16T19:31:40.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358981+0000", 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7122e-05 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358991+0000", 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0339e-05 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359095+0000", 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000104072 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359123+0000", 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7874000000000001e-05 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359890+0000", 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076696700000000002 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359942+0000", 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2306999999999999e-05 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:21 2.1c e21/18 2:3ed09add:::607.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.83849364100001, 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027657160000000001, 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359393+0000", 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359394+0000", 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8200000000000008e-07 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359402+0000", 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2719999999999997e-06 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359417+0000", 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.523e-05 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359430+0000", 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2945999999999999e-05 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359505+0000", 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4714000000000003e-05 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359526+0000", 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1044e-05 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362113+0000", 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025868309999999999 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362159+0000", 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5697e-05 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:30 2.15 e21/18 2:aa448500:::500.00000000:head v 21'3, mlcod=0'0)", 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 128.83825205100001, 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0029513510000000001, 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359635+0000", 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359636+0000", 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4300000000000001e-07 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359659+0000", 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3873999999999999e-05 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359674+0000", 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4433999999999999e-05 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359684+0000", 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8460000000000003e-06 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360036+0000", 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035209400000000001 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360055+0000", 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9083e-05 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362559+0000", 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0025040069999999999 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362586+0000", 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.707e-05 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPush(3.32 81/80 [PushOp(3:4def72d4:::10000000003.00000000:head, version: 79'2, data_included: [0~4], data_size: 4, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4def72d4:::10000000003.00000000:head@79'2, size: 4, copy_subset: [(0, 4)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:38.909987+0000", 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 1.287900131, 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000444089, 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909987+0000", 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909987+0000", 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.909987+0000", 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7899999999999997e-07 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.910001+0000", 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3658e-05 2024-03-16T19:31:40.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.910021+0000", 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0276e-05 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.910033+0000", 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1836e-05 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.910117+0000", 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3660999999999997e-05 2024-03-16T19:31:40.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.910130+0000", 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3419e-05 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:38.910431+0000", 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00030046000000000001 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.201 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:40.202 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-16T19:31:40.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.249+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f258415bb00 con 0x7f25840b0de0 2024-03-16T19:31:40.244 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:40.263+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:40.249 DEBUG:teuthology.parallel:result is None 2024-03-16T19:31:40.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T19:31:40.317 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:31:40.336+0000 7ff521acb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 38402) UID: 0 2024-03-16T19:31:40.323 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T19:31:40.323 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-16T19:31:40.334 INFO:tasks.thrashosds:joining thrashosds 2024-03-16T19:31:40.382 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:40.382 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:40.382 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:40.382 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:40.391 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-16T19:31:40.537 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:40.537 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:40.537 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:40.538 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:40.538 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:40.546 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-16T19:31:40.654 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:40.654 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:40.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:40.654 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.31625253600001, 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005472406, 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357219+0000", 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357228+0000", 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0780000000000002e-06 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357239+0000", 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0793e-05 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357309+0000", 2024-03-16T19:31:40.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0038999999999998e-05 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357329+0000", 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0180999999999999e-05 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357486+0000", 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015646 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013255799999999999 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357809+0000", 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019051199999999999 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361345+0000", 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035365040000000002 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362620+0000", 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012751850000000001 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362658+0000", 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7897e-05 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362691+0000", 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3198999999999998e-05 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.31622836299999, 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058853480000000003, 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357243+0000", 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357244+0000", 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0699999999999997e-07 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357247+0000", 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7620000000000001e-06 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357364+0000", 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000116549 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357371+0000", 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9650000000000002e-06 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358435+0000", 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010639969999999999 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358543+0000", 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000107771 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358688+0000", 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000145332 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362365+0000", 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036774260000000001 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363040+0000", 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00067442900000000004 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363087+0000", 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7156e-05 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363128+0000", 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1454e-05 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.316222536, 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003765904, 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357249+0000", 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.67e-07 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357252+0000", 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3e-06 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357388+0000", 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136403 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357394+0000", 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4919999999999998e-06 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357454+0000", 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0047000000000003e-05 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357618+0000", 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016324500000000001 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357792+0000", 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000174561 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360214+0000", 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024220489999999999 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360902+0000", 2024-03-16T19:31:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687296 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360972+0000", 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0567000000000004e-05 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361015+0000", 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2676999999999998e-05 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.31621862399999, 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00798951, 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357253+0000", 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8999999999999998e-07 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357257+0000", 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2810000000000002e-06 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357419+0000", 2024-03-16T19:31:40.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000161292 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357425+0000", 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358125+0000", 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069914100000000002 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358214+0000", 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8962000000000006e-05 2024-03-16T19:31:40.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 7", 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.358345+0000", 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013114800000000001 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362336+0000", 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039911979999999996 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365151+0000", 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002815176 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365204+0000", 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2771e-05 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365242+0000", 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8485999999999997e-05 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.674 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.31621264200001, 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060130180000000002, 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357259+0000", 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6899999999999998e-07 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357261+0000", 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.979e-06 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357501+0000", 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00023945300000000001 2024-03-16T19:31:40.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357507+0000", 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1340000000000001e-06 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359003+0000", 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001496418 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359111+0000", 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010802999999999999 2024-03-16T19:31:40.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.359255+0000", 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014408199999999999 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.362371+0000", 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003115423 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363190+0000", 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081926600000000003 2024-03-16T19:31:40.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363230+0000", 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9535e-05 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.363272+0000", 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2329000000000001e-05 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.679 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.31620897600001, 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078090670000000003, 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357263+0000", 2024-03-16T19:31:40.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357267+0000", 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9520000000000002e-06 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357272+0000", 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0779999999999997e-06 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357524+0000", 2024-03-16T19:31:40.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025254800000000001 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357532+0000", 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5290000000000003e-06 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360278+0000", 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027464249999999998 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360408+0000", 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012961099999999999 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360590+0000", 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000181898 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364882+0000", 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042920090000000003 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364948+0000", 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6400000000000001e-05 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364999+0000", 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1291000000000001e-05 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365072+0000", 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2325999999999998e-05 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.316198316, 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079262159999999998, 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357273+0000", 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357274+0000", 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1800000000000001e-07 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357276+0000", 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2400000000000002e-06 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357543+0000", 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000266698 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357547+0000", 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.172e-06 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361010+0000", 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034632249999999999 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361119+0000", 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108991 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365199+0000", 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040804719999999999 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:36 2.1a 2:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.316194427, 2024-03-16T19:31:40.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0031265849999999999, 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357277+0000", 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7500000000000001e-07 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357279+0000", 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4360000000000001e-06 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357556+0000", 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00027722599999999999 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357565+0000", 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357635+0000", 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9439999999999999e-05 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.357741+0000", 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010653 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360404+0000", 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002662429 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:16 2.1d e21/18 2:b871830b:::605.00000000:head v 21'1, mlcod=0'0)", 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.31260730100001, 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041337559999999997, 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360864+0000", 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360865+0000", 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0100000000000005e-07 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360871+0000", 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3779999999999998e-06 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360878+0000", 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3950000000000003e-06 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360885+0000", 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5409999999999997e-06 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360959+0000", 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4325000000000004e-05 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.360983+0000", 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3300999999999999e-05 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364955+0000", 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039725350000000001 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.364998+0000", 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.278e-05 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:17 2.1d e21/18 2:b871830b:::605.00000000:head v 21'2, mlcod=0'0)", 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.31222495399999, 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037929069999999999, 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361247+0000", 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361248+0000", 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.049e-06 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361256+0000", 2024-03-16T19:31:40.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7010000000000002e-06 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361266+0000", 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0098000000000001e-05 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361272+0000", 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9610000000000002e-06 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361324+0000", 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2042000000000001e-05 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.361343+0000", 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8430000000000001e-05 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365003+0000", 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003660562 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365039+0000", 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6063999999999999e-05 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:38 2.14 2:2e2fa760:::200.00000001:head [write 0~940] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 129.30779000000001, 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019266019999999999, 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365681+0000", 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365683+0000", 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5859999999999999e-06 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365691+0000", 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365714+0000", 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2552999999999999e-05 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365727+0000", 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3355e-05 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365791+0000", 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3633999999999994e-05 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365841+0000", 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9521999999999998e-05 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.365922+0000", 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1717000000000004e-05 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.366767+0000", 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084456400000000001 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367531+0000", 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076376799999999996 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367566+0000", 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5023e-05 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:31.367608+0000", 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2509000000000002e-05 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:40 2.14 2:2e2fa760:::200.00000001:head [write 940~256 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 120.86937802600001, 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034699560000000002, 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804093+0000", 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804095+0000", 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2079999999999999e-06 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804100+0000", 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4e-06 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804116+0000", 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-05 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804125+0000", 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8580000000000006e-06 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804176+0000", 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0544000000000003e-05 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804240+0000", 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3715999999999998e-05 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.804338+0000", 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8737e-05 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.805019+0000", 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068049500000000004 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807448+0000", 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024289799999999999 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807513+0000", 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5506000000000006e-05 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:39.807563+0000", 2024-03-16T19:31:40.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0092000000000001e-05 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:41 2.14 2:2e2fa760:::200.00000001:head [write 1196~987 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 119.30188432, 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012473600000000001, 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371587+0000", 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371588+0000", 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2500000000000001e-06 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371595+0000", 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3539999999999996e-06 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371613+0000", 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8508999999999998e-05 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371621+0000", 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9750000000000007e-06 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371667+0000", 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5892e-05 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371730+0000", 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3226999999999996e-05 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.371815+0000", 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4971999999999999e-05 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372517+0000", 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070139300000000003 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372771+0000", 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025434500000000001 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372805+0000", 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3572999999999999e-05 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:41.372835+0000", 2024-03-16T19:31:40.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.987e-05 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:42 2.14 2:2e2fa760:::200.00000001:head [write 2183~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 117.794272069, 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001804714, 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879199+0000", 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879201+0000", 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1170000000000001e-06 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879206+0000", 2024-03-16T19:31:40.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3329999999999996e-06 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879223+0000", 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7096999999999999e-05 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879231+0000", 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6179999999999997e-06 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879291+0000", 2024-03-16T19:31:40.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0102000000000002e-05 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879356+0000", 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5427999999999998e-05 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.879443+0000", 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7026000000000003e-05 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880135+0000", 2024-03-16T19:31:40.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00069139799999999997 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880941+0000", 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080650700000000004 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.880974+0000", 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3328999999999999e-05 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:42.881004+0000", 2024-03-16T19:31:40.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9759000000000001e-05 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop(mds.0.4:47 2.6 e40/18 2:654134d2:::mds0_openfiles.0:head v 40'4, mlcod=21'2)", 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.296872927999999, 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089267600000000002, 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376599+0000", 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376600+0000", 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8899999999999998e-07 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376608+0000", 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3010000000000007e-06 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376620+0000", 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919999999999999e-05 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376631+0000", 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.119e-05 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376698+0000", 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7150999999999995e-05 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.376726+0000", 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7693999999999999e-05 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377451+0000", 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072499399999999996 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:30:25.377491+0000", 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0436999999999999e-05 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:49 2.14 2:2e2fa760:::200.00000001:head [write 5990~7422 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e77)", 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:31.372820+0000", 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.3006513660000003, 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001781929, 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372820+0000", 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372820+0000", 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372822+0000", 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1959999999999998e-06 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372849+0000", 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6591e-05 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372885+0000", 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5865999999999998e-05 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372896+0000", 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1212e-05 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372960+0000", 2024-03-16T19:31:40.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3788999999999995e-05 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373030+0000", 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9993000000000006e-05 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373130+0000", 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000100214 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374163+0000", 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010331839999999999 2024-03-16T19:31:40.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374534+0000", 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037083 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374567+0000", 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2549999999999998e-05 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374602+0000", 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5503999999999998e-05 2024-03-16T19:31:40.728 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:50 2.14 2:292cf221:::200.00000000:head [writefull 0~90 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e77)", 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:31.372854+0000", 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.300617484, 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001920001, 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372854+0000", 2024-03-16T19:31:40.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372854+0000", 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372854+0000", 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5900000000000003e-07 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372857+0000", 2024-03-16T19:31:40.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3140000000000002e-06 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372925+0000", 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8816e-05 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.372930+0000", 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9429999999999999e-06 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373562+0000", 2024-03-16T19:31:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00063115399999999998 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373649+0000", 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7378000000000003e-05 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.373777+0000", 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012839500000000001 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374622+0000", 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00084469299999999998 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374726+0000", 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000103853 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374753+0000", 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6825000000000002e-05 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:31.374774+0000", 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1271e-05 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.733 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:54 2.14 2:2e2fa760:::200.00000001:head [write 19040~8125 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e82)", 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:39.963760+0000", 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.70971128800000005, 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075248210000000001, 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.963760+0000", 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.963760+0000", 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.963763+0000", 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.762e-06 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.963782+0000", 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9460999999999999e-05 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.963816+0000", 2024-03-16T19:31:40.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3586e-05 2024-03-16T19:31:40.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.963829+0000", 2024-03-16T19:31:40.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3015e-05 2024-03-16T19:31:40.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.963903+0000", 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3479000000000006e-05 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.963908+0000", 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2560000000000002e-06 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.965902+0000", 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00199402 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.965990+0000", 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8684000000000007e-05 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.966124+0000", 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000133421 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.968415+0000", 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0022914760000000002 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.971191+0000", 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027752720000000001 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.971244+0000", 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3363999999999999e-05 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.971285+0000", 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1025000000000001e-05 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPush(1.0 82/81 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:39.974097+0000", 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.69937461700000003, 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010601460000000001, 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.974097+0000", 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.974097+0000", 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.974098+0000", 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2399999999999997e-07 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.974263+0000", 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016500400000000001 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.974280+0000", 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6905000000000002e-05 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.974297+0000", 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7186000000000001e-05 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.974398+0000", 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101352 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.974414+0000", 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6310000000000001e-05 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:39.975157+0000", 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00074256500000000004 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(mds.0.4:55 2.14 2:2e2fa760:::200.00000001:head [write 27165~3298 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e82)", 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:31:40.337067+0000", 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.33640419500000002, 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015499999999999999, 2024-03-16T19:31:40.744 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.337067+0000", 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.337067+0000", 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.337069+0000", 2024-03-16T19:31:40.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.874e-06 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.337082+0000", 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.309e-05 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.337107+0000", 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4956e-05 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.337117+0000", 2024-03-16T19:31:40.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5489999999999995e-06 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.337191+0000", 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4710999999999997e-05 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.337260+0000", 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8063000000000002e-05 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 3", 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.337349+0000", 2024-03-16T19:31:40.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9662999999999999e-05 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.338113+0000", 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00076428599999999998 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.338552+0000", 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00043806200000000002 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.338585+0000", 2024-03-16T19:31:40.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3399999999999999e-05 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:31:40.338617+0000", 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2345999999999998e-05 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:40.749 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:40.750 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-16T19:31:40.855 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:40.855 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:40.855 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-03-16T19:31:40.856 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:40.864 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-16T19:31:40.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.922+0000 7f257a7fc640 1 -- 172.21.15.172:0/2099962814 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v83) v1 ==== 101+0+0 (secure 0 0 0) 0x7f257c07a330 con 0x7f25840b0de0 2024-03-16T19:31:40.902 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:31:40.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f256806b3b0 msgr2=0x7f256806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f256806b3b0 0x7f256806d870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f25841e8720 tx=0x7f2574007de0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25840b0de0 msgr2=0x7f25840c54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25840b0de0 0x7f25840c54d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f257c000f80 tx=0x7f257c004a40 comp rx=0 tx=0).stop 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 shutdown_connections 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f256806b3b0 0x7f256806d870 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2584152a20 0x7f25841f1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25840ce4e0 0x7f25840cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 --2- 172.21.15.172:0/2099962814 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25840b0de0 0x7f25840c54d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 >> 172.21.15.172:0/2099962814 conn(0x7f2584066e00 msgr2=0x7f25840ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 shutdown_connections 2024-03-16T19:31:40.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:40.924+0000 7f258bab5640 1 -- 172.21.15.172:0/2099962814 wait complete. 2024-03-16T19:31:40.974 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:40.974 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-03-16T19:31:40.974 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 180, 2024-03-16T19:31:40.974 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-03-16T19:31:40.974 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:40.982 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-16T19:31:41.090 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 144.87892420899999, 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34622022000000002, 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.091 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229470+0000", 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229472+0000", 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.097e-06 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.229480+0000", 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3069999999999997e-06 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573946+0000", 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.34446627699999999 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.573954+0000", 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1289999999999996e-06 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574044+0000", 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9889000000000006e-05 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574255+0000", 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021096299999999999 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.574444+0000", 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000189378 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575331+0000", 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088656599999999998 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575612+0000", 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00028075499999999999 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575658+0000", 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6134000000000002e-05 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.575691+0000", 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2725000000000003e-05 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.096 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:26 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [getxattr striper.excl (26) in=12b,getxattr striper.size (16) in=12b,getxattr striper.allocated (16) in=17b,getxattr striper.version (16) in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 144.50222866199999, 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.772430057, 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.097 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 26 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606166+0000", 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606167+0000", 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1699999999999997e-07 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:16.606171+0000", 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4660000000000001e-06 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378048+0000", 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.77187724700000004 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378064+0000", 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.59e-05 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378277+0000", 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021325700000000001 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378498+0000", 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000220232 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.378596+0000", 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8238000000000004e-05 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.72625221800001, 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086448389999999996, 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382142+0000", 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382143+0000", 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1399999999999998e-07 2024-03-16T19:31:41.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382222+0000", 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9690000000000004e-05 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382321+0000", 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8429000000000002e-05 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382325+0000", 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7419999999999997e-06 2024-03-16T19:31:41.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385412+0000", 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003086232 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385513+0000", 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101215 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.385653+0000", 2024-03-16T19:31:41.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014050099999999999 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387104+0000", 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014508800000000001 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390732+0000", 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036272409999999998 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390764+0000", 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2867999999999998e-05 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390787+0000", 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2727000000000001e-05 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.107 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.72616780499999, 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087014970000000007, 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382227+0000", 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-03-16T19:31:41.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382232+0000", 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0509999999999998e-06 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382342+0000", 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000110055 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382349+0000", 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9349999999999994e-06 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386185+0000", 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038351499999999998 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386236+0000", 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1802000000000003e-05 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386329+0000", 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2108000000000006e-05 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387108+0000", 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077987900000000003 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390839+0000", 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003730747 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390893+0000", 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3511000000000003e-05 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390928+0000", 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5509000000000002e-05 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.113 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.72616044899999, 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087938180000000001, 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382234+0000", 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3500000000000002e-07 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382238+0000", 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382365+0000", 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012767 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.382370+0000", 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7600000000000002e-06 2024-03-16T19:31:41.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386642+0000", 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004271703 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386729+0000", 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6824999999999996e-05 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.386865+0000", 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000136379 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.387671+0000", 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00080619700000000003 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390969+0000", 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003297229 2024-03-16T19:31:41.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.390999+0000", 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0753000000000002e-05 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.391028+0000", 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8608000000000001e-05 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.119 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.53362125699999, 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008547519, 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 113 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574773+0000", 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574774+0000", 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9699999999999996e-07 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574777+0000", 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1140000000000002e-06 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574872+0000", 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5267999999999997e-05 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574876+0000", 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640000000000004e-06 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580139+0000", 2024-03-16T19:31:41.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052626840000000001 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580215+0000", 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6174000000000006e-05 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580338+0000", 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012277999999999999 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583099+0000", 2024-03-16T19:31:41.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00276071 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583242+0000", 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000143032 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583285+0000", 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2831999999999999e-05 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583321+0000", 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6263999999999997e-05 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.53361545000001, 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086238110000000003, 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 114 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:41.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574779+0000", 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1300000000000001e-07 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574782+0000", 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7669999999999999e-06 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574883+0000", 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010074299999999999 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574885+0000", 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.588e-06 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580884+0000", 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059980759999999998 2024-03-16T19:31:41.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.580927+0000", 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3713e-05 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581012+0000", 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5192999999999996e-05 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583133+0000", 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021204779999999999 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583327+0000", 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019430900000000001 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583369+0000", 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1505999999999999e-05 2024-03-16T19:31:41.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583403+0000", 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4125000000000003e-05 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.53349731700001, 2024-03-16T19:31:41.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096109309999999996, 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 115 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:41.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574897+0000", 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574898+0000", 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4499999999999997e-07 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574989+0000", 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1742000000000004e-05 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575008+0000", 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8529000000000002e-05 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575012+0000", 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6820000000000001e-06 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581266+0000", 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062543859999999998 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581341+0000", 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5406000000000003e-05 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581468+0000", 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012647799999999999 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583409+0000", 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001941356 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584419+0000", 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001009759 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584472+0000", 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3263000000000001e-05 2024-03-16T19:31:41.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584508+0000", 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5784999999999997e-05 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.53340212000001, 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096255820000000006, 2024-03-16T19:31:41.137 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 116 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574992+0000", 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574993+0000", 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2800000000000002e-07 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574995+0000", 2024-03-16T19:31:41.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3180000000000001e-06 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575022+0000", 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6622e-05 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575024+0000", 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6929999999999999e-06 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.581961+0000", 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069366269999999999 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582055+0000", 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4123999999999995e-05 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582170+0000", 2024-03-16T19:31:41.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000115154 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583414+0000", 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012431810000000001 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584555+0000", 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001141925 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584587+0000", 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1112000000000003e-05 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584618+0000", 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1398e-05 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.143 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.53339810899999, 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097488720000000004, 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 117 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574996+0000", 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574997+0000", 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-03-16T19:31:41.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.574998+0000", 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.823e-06 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575031+0000", 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2911999999999999e-05 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.575034+0000", 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7259999999999998e-06 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582506+0000", 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074723050000000003 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582596+0000", 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9399000000000002e-05 2024-03-16T19:31:41.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.582761+0000", 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165558 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.583420+0000", 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065819999999999995 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584658+0000", 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001238168 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584710+0000", 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2259999999999998e-05 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.584745+0000", 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5317000000000001e-05 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.11446623500001, 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095492319999999995, 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.150 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 133 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993928+0000", 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993929+0000", 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1500000000000004e-07 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993932+0000", 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.977e-06 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993941+0000", 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6489999999999997e-06 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993944+0000", 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.19e-06 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996083+0000", 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021388539999999999 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996129+0000", 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6409000000000002e-05 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996198+0000", 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8712000000000002e-05 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997165+0000", 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000967029 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003309+0000", 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0061444760000000003 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003392+0000", 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2966000000000005e-05 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003478+0000", 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5354999999999998e-05 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.114416866, 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096701549999999997, 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 134 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.993978+0000", 2024-03-16T19:31:41.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-07 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994054+0000", 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6413000000000002e-05 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994108+0000", 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3875999999999997e-05 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994112+0000", 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6430000000000001e-06 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996443+0000", 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023308920000000002 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996522+0000", 2024-03-16T19:31:41.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9399000000000003e-05 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.996644+0000", 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012196700000000001 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998669+0000", 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020247020000000002 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003492+0000", 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048235650000000001 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003558+0000", 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5358999999999995e-05 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003648+0000", 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9956000000000004e-05 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.161 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.11433803400001, 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096327960000000008, 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.162 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 135 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994056+0000", 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994057+0000", 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4400000000000001e-07 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994059+0000", 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7740000000000001e-06 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994123+0000", 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4220999999999994e-05 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994125+0000", 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5790000000000002e-06 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997186+0000", 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0030606320000000002 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997230+0000", 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3513000000000002e-05 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997302+0000", 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2614999999999995e-05 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998819+0000", 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001516744 2024-03-16T19:31:41.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003651+0000", 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048318340000000001 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003672+0000", 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1233000000000002e-05 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003689+0000", 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7306999999999999e-05 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.11433482300001, 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099077609999999993, 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 136 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994060+0000", 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0200000000000001e-07 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994062+0000", 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6169999999999999e-06 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994131+0000", 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9530999999999995e-05 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994137+0000", 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6180000000000003e-06 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997568+0000", 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034311319999999999 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997643+0000", 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5016000000000001e-05 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.997765+0000", 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000122531 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000130+0000", 2024-03-16T19:31:41.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023649769999999999 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003924+0000", 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003793959 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003948+0000", 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3581e-05 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003967+0000", 2024-03-16T19:31:41.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9596999999999999e-05 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.11433187, 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099452289999999999, 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 137 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994063+0000", 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994176+0000", 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011289399999999999 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994255+0000", 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9017999999999995e-05 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994260+0000", 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9259999999999999e-06 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998089+0000", 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038289940000000001 2024-03-16T19:31:41.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998135+0000", 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5991000000000001e-05 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998205+0000", 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0704000000000006e-05 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000508+0000", 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0023023589999999999 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003971+0000", 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034628599999999999 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.003991+0000", 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0888000000000001e-05 2024-03-16T19:31:41.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004008+0000", 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6392000000000002e-05 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:41.180 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.11421689599999, 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098709450000000008, 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 138 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:41.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994178+0000", 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6300000000000001e-07 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994180+0000", 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.838e-06 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994267+0000", 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7609999999999996e-05 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994270+0000", 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4200000000000001e-06 2024-03-16T19:31:41.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998853+0000", 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045827669999999997 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998900+0000", 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7484999999999997e-05 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.998972+0000", 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2371000000000002e-05 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001172+0000", 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021992140000000001 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004011+0000", 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028392040000000001 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004031+0000", 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0092999999999999e-05 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004049+0000", 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7580000000000001e-05 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.11421354399999, 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099071339999999997, 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 139 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.187 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994181+0000", 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2800000000000003e-07 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994183+0000", 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.683e-06 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994275+0000", 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2262999999999994e-05 2024-03-16T19:31:41.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994278+0000", 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699999999999999e-06 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999211+0000", 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049330939999999998 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999261+0000", 2024-03-16T19:31:41.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9549000000000003e-05 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999335+0000", 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4059000000000004e-05 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001200+0000", 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001864859 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004052+0000", 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028526089999999999 2024-03-16T19:31:41.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004073+0000", 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0367999999999999e-05 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004088+0000", 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5352000000000001e-05 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.192 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.11421042800001, 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009943136, 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994184+0000", 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.03e-07 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994243+0000", 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8477000000000003e-05 2024-03-16T19:31:41.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994284+0000", 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0924000000000003e-05 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994286+0000", 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3870000000000002e-06 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999671+0000", 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005384865 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999714+0000", 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3438e-05 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.999784+0000", 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0042000000000005e-05 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001947+0000", 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0021628609999999999 2024-03-16T19:31:41.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004091+0000", 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002143804 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004112+0000", 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0474000000000001e-05 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004127+0000", 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5661000000000001e-05 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 143.114149959, 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099197079999999993, 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.198 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994245+0000", 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994247+0000", 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.725e-06 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994293+0000", 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6045999999999999e-05 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:17.994296+0000", 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1540000000000002e-06 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000152+0000", 2024-03-16T19:31:41.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058560599999999997 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000198+0000", 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6156999999999999e-05 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.000271+0000", 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3045000000000003e-05 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.001969+0000", 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001697804 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004130+0000", 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002161152 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004149+0000", 2024-03-16T19:31:41.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8760999999999999e-05 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.004164+0000", 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5523e-05 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4203.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 142.505469909, 2024-03-16T19:31:41.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011219396, 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4203", 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.172:0/1142089520", 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602925+0000", 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1800000000000002e-07 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602927+0000", 2024-03-16T19:31:41.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9970000000000001e-06 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602954+0000", 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7236000000000001e-05 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.602957+0000", 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6309999999999999e-06 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-03-16T19:31:41.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605392+0000", 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00243531 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605436+0000", 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4042999999999998e-05 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 0", 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.605518+0000", 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2006000000000001e-05 2024-03-16T19:31:41.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.608784+0000", 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032661679999999998 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614078+0000", 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052937669999999996 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-03-16T19:31:41.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614108+0000", 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9539999999999998e-05 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-03-16T19:29:18.614144+0000", 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6380000000000001e-05 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-16T19:31:41.210 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-16T19:31:42.902 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-16T19:31:42.903 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-03-16T19:31:42.903 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-16T19:31:42.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-16T19:31:42.906 INFO:tasks.ceph.osd.2:Started 2024-03-16T19:31:42.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:31:42.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-16T19:31:43.021 INFO:teuthology.orchestra.run.smithi172.stderr:no valid command found; 10 closest matches: 2024-03-16T19:31:43.021 INFO:teuthology.orchestra.run.smithi172.stderr:0 2024-03-16T19:31:43.021 INFO:teuthology.orchestra.run.smithi172.stderr:1 2024-03-16T19:31:43.021 INFO:teuthology.orchestra.run.smithi172.stderr:2 2024-03-16T19:31:43.021 INFO:teuthology.orchestra.run.smithi172.stderr:abort 2024-03-16T19:31:43.021 INFO:teuthology.orchestra.run.smithi172.stderr:assert 2024-03-16T19:31:43.021 INFO:teuthology.orchestra.run.smithi172.stderr:config diff 2024-03-16T19:31:43.021 INFO:teuthology.orchestra.run.smithi172.stderr:config diff get 2024-03-16T19:31:43.021 INFO:teuthology.orchestra.run.smithi172.stderr:config get 2024-03-16T19:31:43.022 INFO:teuthology.orchestra.run.smithi172.stderr:config help [] 2024-03-16T19:31:43.022 INFO:teuthology.orchestra.run.smithi172.stderr:config set ... 2024-03-16T19:31:43.022 INFO:teuthology.orchestra.run.smithi172.stderr:admin_socket: invalid command 2024-03-16T19:31:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-16T19:31:43.025 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-03-16T19:31:43.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.081+0000 7f3441d2c640 1 Processor -- start 2024-03-16T19:31:43.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f3441d2c640 1 -- start start 2024-03-16T19:31:43.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f3441d2c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c152ab0 0x7f343c0c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:43.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f3441d2c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f343c0c9af0 0x7f343c15bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:43.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f3441d2c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c152e90 con 0x7f343c06fde0 2024-03-16T19:31:43.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f3441d2c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c0c85e0 con 0x7f343c0c9af0 2024-03-16T19:31:43.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f3441d2c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0c30e0 con 0x7f343c152ab0 2024-03-16T19:31:43.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f343bfff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f343c06fde0 0x7f343c06e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51292/0 (socket says 172.21.15.172:51292) 2024-03-16T19:31:43.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f343bfff640 1 -- 172.21.15.172:0/1566228741 learned_addr learned my addr 172.21.15.172:0/1566228741 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:43.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f343b7fe640 1 --2- 172.21.15.172:0/1566228741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c152ab0 0x7f343c0c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:43.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.082+0000 7f343affd640 1 --2- 172.21.15.172:0/1566228741 >> v2:172.21.15.174:3300/0 conn(0x7f343c0c9af0 0x7f343c15bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:43.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.083+0000 7f343a7fc640 1 -- 172.21.15.172:0/1566228741 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478412600 0 0) 0x7f342c001110 con 0x7f343c06fde0 2024-03-16T19:31:43.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.083+0000 7f343b7fe640 1 -- 172.21.15.172:0/1566228741 >> v2:172.21.15.174:3300/0 conn(0x7f343c0c9af0 msgr2=0x7f343c15bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:43.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.083+0000 7f343b7fe640 1 --2- 172.21.15.172:0/1566228741 >> v2:172.21.15.174:3300/0 conn(0x7f343c0c9af0 0x7f343c15bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:43.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.083+0000 7f343b7fe640 1 -- 172.21.15.172:0/1566228741 >> v1:172.21.15.172:6789/0 conn(0x7f343c06fde0 legacy=0x7f343c06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:43.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.083+0000 7f343b7fe640 1 -- 172.21.15.172:0/1566228741 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0c33c0 con 0x7f343c152ab0 2024-03-16T19:31:43.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f343b7fe640 1 --2- 172.21.15.172:0/1566228741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c152ab0 0x7f343c0c8ac0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3434009d90 tx=0x7f343402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd58782b2114208d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:43.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f343a7fc640 1 -- 172.21.15.172:0/1566228741 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343402cc50 con 0x7f343c152ab0 2024-03-16T19:31:43.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f343a7fc640 1 -- 172.21.15.172:0/1566228741 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343402cdb0 con 0x7f343c152ab0 2024-03-16T19:31:43.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f343a7fc640 1 -- 172.21.15.172:0/1566228741 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434034660 con 0x7f343c152ab0 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f3441d2c640 1 -- 172.21.15.172:0/1566228741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c152ab0 msgr2=0x7f343c0c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f3441d2c640 1 --2- 172.21.15.172:0/1566228741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c152ab0 0x7f343c0c8ac0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3434009d90 tx=0x7f343402bea0 comp rx=0 tx=0).stop 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f3441d2c640 1 -- 172.21.15.172:0/1566228741 shutdown_connections 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f3441d2c640 1 --2- 172.21.15.172:0/1566228741 >> v2:172.21.15.174:3300/0 conn(0x7f343c0c9af0 0x7f343c15bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f3441d2c640 1 --2- 172.21.15.172:0/1566228741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c152ab0 0x7f343c0c8ac0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f3441d2c640 1 -- 172.21.15.172:0/1566228741 >> 172.21.15.172:0/1566228741 conn(0x7f343c070810 msgr2=0x7f343c070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f3441d2c640 1 -- 172.21.15.172:0/1566228741 shutdown_connections 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.084+0000 7f3441d2c640 1 -- 172.21.15.172:0/1566228741 wait complete. 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.085+0000 7f3441d2c640 1 Processor -- start 2024-03-16T19:31:43.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f3441d2c640 1 -- start start 2024-03-16T19:31:43.068 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f3441d2c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c06fde0 0x7f343c158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:43.068 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f3441d2c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343c0c9af0 0x7f343c1e30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:43.068 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f3441d2c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f343c152ab0 0x7f343c14bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f3441d2c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c1f2530 con 0x7f343c0c9af0 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f3441d2c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c1e2970 con 0x7f343c152ab0 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f3441d2c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c1e2690 con 0x7f343c06fde0 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f343affd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343c0c9af0 0x7f343c1e30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f343affd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343c0c9af0 0x7f343c1e30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42626/0 (socket says 172.21.15.172:42626) 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f343affd640 1 -- 172.21.15.172:0/4167824060 learned_addr learned my addr 172.21.15.172:0/4167824060 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f343b7fe640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c06fde0 0x7f343c158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.086+0000 7f343bfff640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f343c152ab0 0x7f343c14bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f343affd640 1 -- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c06fde0 msgr2=0x7f343c158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f343affd640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c06fde0 0x7f343c158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f343affd640 1 -- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f343c152ab0 msgr2=0x7f343c14bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f343affd640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f343c152ab0 0x7f343c14bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f343affd640 1 -- 172.21.15.172:0/4167824060 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434009a70 con 0x7f343c0c9af0 2024-03-16T19:31:43.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f343b7fe640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c06fde0 0x7f343c158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:43.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f343bfff640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f343c152ab0 0x7f343c14bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:43.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f343affd640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343c0c9af0 0x7f343c1e30b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f342400e9b0 tx=0x7f342400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:43.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f3433fff640 1 -- 172.21.15.172:0/4167824060 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424009670 con 0x7f343c0c9af0 2024-03-16T19:31:43.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f3433fff640 1 -- 172.21.15.172:0/4167824060 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342400c9a0 con 0x7f343c0c9af0 2024-03-16T19:31:43.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f3433fff640 1 -- 172.21.15.172:0/4167824060 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342400cd30 con 0x7f343c0c9af0 2024-03-16T19:31:43.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c1f1c00 con 0x7f343c0c9af0 2024-03-16T19:31:43.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.087+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c1f21c0 con 0x7f343c0c9af0 2024-03-16T19:31:43.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.088+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c1f26e0 con 0x7f343c0c9af0 2024-03-16T19:31:43.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.089+0000 7f3433fff640 1 -- 172.21.15.172:0/4167824060 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3424013470 con 0x7f343c0c9af0 2024-03-16T19:31:43.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.089+0000 7f3433fff640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f341c06b3b0 0x7f341c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:43.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.089+0000 7f343b7fe640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f341c06b3b0 0x7f341c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:43.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.089+0000 7f3433fff640 1 -- 172.21.15.172:0/4167824060 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f3424089f10 con 0x7f343c0c9af0 2024-03-16T19:31:43.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.090+0000 7f343b7fe640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f341c06b3b0 0x7f341c06d870 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3434036000 tx=0x7f3434030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:43.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.094+0000 7f3433fff640 1 -- 172.21.15.172:0/4167824060 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f343c1f26e0 con 0x7f343c0c9af0 2024-03-16T19:31:43.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.275+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f343c1f26e0 con 0x7f343c0c9af0 2024-03-16T19:31:43.505 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:31:43.525+0000 7f1aaf424640 -1 Falling back to public interface 2024-03-16T19:31:43.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.938+0000 7f3433fff640 1 -- 172.21.15.172:0/4167824060 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v85) v1 ==== 91+0+0 (secure 0 0 0) 0x7f343c1f26e0 con 0x7f343c0c9af0 2024-03-16T19:31:43.918 INFO:teuthology.orchestra.run.smithi172.stderr:noscrub is unset 2024-03-16T19:31:43.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.940+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f341c06b3b0 msgr2=0x7f341c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.940+0000 7f3441d2c640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f341c06b3b0 0x7f341c06d870 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3434036000 tx=0x7f3434030000 comp rx=0 tx=0).stop 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.940+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343c0c9af0 msgr2=0x7f343c1e30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.941+0000 7f3441d2c640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343c0c9af0 0x7f343c1e30b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f342400e9b0 tx=0x7f342400ee70 comp rx=0 tx=0).stop 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.941+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 shutdown_connections 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.941+0000 7f3441d2c640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f343c152ab0 0x7f343c14bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.941+0000 7f3441d2c640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f343c0c9af0 0x7f343c1e30b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.941+0000 7f3441d2c640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f341c06b3b0 0x7f341c06d870 secure :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3434036000 tx=0x7f3434030000 comp rx=0 tx=0).stop 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.941+0000 7f3441d2c640 1 --2- 172.21.15.172:0/4167824060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f343c06fde0 0x7f343c158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.941+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 >> 172.21.15.172:0/4167824060 conn(0x7f343c070810 msgr2=0x7f343c0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.941+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 shutdown_connections 2024-03-16T19:31:43.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:43.941+0000 7f3441d2c640 1 -- 172.21.15.172:0/4167824060 wait complete. 2024-03-16T19:31:43.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-16T19:31:44.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.098+0000 7f0ac76c0640 1 Processor -- start 2024-03-16T19:31:44.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.099+0000 7f0ac76c0640 1 -- start start 2024-03-16T19:31:44.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.099+0000 7f0ac76c0640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0ac0154e50 0x7f0ac00cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:44.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.099+0000 7f0ac76c0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0ac00caf40 0x7f0ac00cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:44.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.099+0000 7f0ac76c0640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac006b110 con 0x7f0ac00b5f10 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.099+0000 7f0ac76c0640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac015b8d0 con 0x7f0ac0154e50 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.099+0000 7f0ac76c0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac015ad50 con 0x7f0ac00caf40 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.100+0000 7f0ac545c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0ac0154e50 0x7f0ac00cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.100+0000 7f0ac4c5b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0ac00b5f10 0x7f0ac006b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51300/0 (socket says 172.21.15.172:51300) 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.100+0000 7f0ac4c5b640 1 -- 172.21.15.172:0/2871847229 learned_addr learned my addr 172.21.15.172:0/2871847229 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.100+0000 7f0ac5c5d640 1 --2- 172.21.15.172:0/2871847229 >> v2:172.21.15.174:3300/0 conn(0x7f0ac00caf40 0x7f0ac00cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.100+0000 7f0ac545c640 1 -- 172.21.15.172:0/2871847229 >> v2:172.21.15.174:3300/0 conn(0x7f0ac00caf40 msgr2=0x7f0ac00cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.100+0000 7f0ac545c640 1 --2- 172.21.15.172:0/2871847229 >> v2:172.21.15.174:3300/0 conn(0x7f0ac00caf40 0x7f0ac00cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.100+0000 7f0ac545c640 1 -- 172.21.15.172:0/2871847229 >> v1:172.21.15.172:6789/0 conn(0x7f0ac00b5f10 legacy=0x7f0ac006b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.100+0000 7f0ac545c640 1 -- 172.21.15.172:0/2871847229 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac015b030 con 0x7f0ac0154e50 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.100+0000 7f0ab7fff640 1 -- 172.21.15.172:0/2871847229 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212532368 0 0) 0x7f0ac006b110 con 0x7f0ac00b5f10 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ac545c640 1 --2- 172.21.15.172:0/2871847229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 0x7f0ac00cbc30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0075200 tx=0x7f0ab002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588f349ee153f466 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ab7fff640 1 -- 172.21.15.172:0/2871847229 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0030610 con 0x7f0ac0154e50 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ab7fff640 1 -- 172.21.15.172:0/2871847229 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab0030770 con 0x7f0ac0154e50 2024-03-16T19:31:44.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ab7fff640 1 -- 172.21.15.172:0/2871847229 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0030b00 con 0x7f0ac0154e50 2024-03-16T19:31:44.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2871847229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 msgr2=0x7f0ac00cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:44.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ac76c0640 1 --2- 172.21.15.172:0/2871847229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 0x7f0ac00cbc30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0075200 tx=0x7f0ab002b960 comp rx=0 tx=0).stop 2024-03-16T19:31:44.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2871847229 shutdown_connections 2024-03-16T19:31:44.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ac76c0640 1 --2- 172.21.15.172:0/2871847229 >> v2:172.21.15.174:3300/0 conn(0x7f0ac00caf40 0x7f0ac00cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ac76c0640 1 --2- 172.21.15.172:0/2871847229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 0x7f0ac00cbc30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2871847229 >> 172.21.15.172:0/2871847229 conn(0x7f0ac0066c20 msgr2=0x7f0ac0067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2871847229 shutdown_connections 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.101+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2871847229 wait complete. 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.102+0000 7f0ac76c0640 1 Processor -- start 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac76c0640 1 -- start start 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac76c0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ac00b5f10 0x7f0ac01529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac76c0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ac00caf40 0x7f0ac00c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac76c0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 0x7f0ac00c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac76c0640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac00c4c90 con 0x7f0ac0154e50 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac76c0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0158240 con 0x7f0ac00b5f10 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac76c0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01f0d70 con 0x7f0ac00caf40 2024-03-16T19:31:44.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac5c5d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 0x7f0ac00c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac5c5d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 0x7f0ac00c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42648/0 (socket says 172.21.15.172:42648) 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac5c5d640 1 -- 172.21.15.172:0/2641032175 learned_addr learned my addr 172.21.15.172:0/2641032175 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac4c5b640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ac00caf40 0x7f0ac00c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.103+0000 7f0ac545c640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ac00b5f10 0x7f0ac01529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ac5c5d640 1 -- 172.21.15.172:0/2641032175 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ac00caf40 msgr2=0x7f0ac00c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ac5c5d640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ac00caf40 0x7f0ac00c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ac5c5d640 1 -- 172.21.15.172:0/2641032175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ac00b5f10 msgr2=0x7f0ac01529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ac5c5d640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ac00b5f10 0x7f0ac01529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ac5c5d640 1 -- 172.21.15.172:0/2641032175 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0009400 con 0x7f0ac0154e50 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ac5c5d640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 0x7f0ac00c44c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800efc0 tx=0x7f0ab800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ab5ffb640 1 -- 172.21.15.172:0/2641032175 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8014480 con 0x7f0ac0154e50 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ab5ffb640 1 -- 172.21.15.172:0/2641032175 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab800f040 con 0x7f0ac0154e50 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ab5ffb640 1 -- 172.21.15.172:0/2641032175 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8014820 con 0x7f0ac0154e50 2024-03-16T19:31:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac015ab20 con 0x7f0ac0154e50 2024-03-16T19:31:44.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.104+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac01f0720 con 0x7f0ac0154e50 2024-03-16T19:31:44.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.105+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac00c9ab0 con 0x7f0ac0154e50 2024-03-16T19:31:44.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.105+0000 7f0ab5ffb640 1 -- 172.21.15.172:0/2641032175 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0ab8014980 con 0x7f0ac0154e50 2024-03-16T19:31:44.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.106+0000 7f0ab5ffb640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0aa406b3e0 0x7f0aa406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:44.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.106+0000 7f0ab5ffb640 1 -- 172.21.15.172:0/2641032175 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f0ab808a680 con 0x7f0ac0154e50 2024-03-16T19:31:44.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.106+0000 7f0ac545c640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0aa406b3e0 0x7f0aa406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:44.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.106+0000 7f0ac545c640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0aa406b3e0 0x7f0aa406d8a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0036000 tx=0x7f0ab0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:44.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.110+0000 7f0ab5ffb640 1 -- 172.21.15.172:0/2641032175 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ab8055350 con 0x7f0ac0154e50 2024-03-16T19:31:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.292+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f0ac0075a70 con 0x7f0ac0154e50 2024-03-16T19:31:44.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.946+0000 7f0ab5ffb640 1 -- 172.21.15.172:0/2641032175 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v86) v1 ==== 101+0+0 (secure 0 0 0) 0x7f0ab8018020 con 0x7f0ac0154e50 2024-03-16T19:31:44.927 INFO:teuthology.orchestra.run.smithi172.stderr:nodeep-scrub is unset 2024-03-16T19:31:44.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0aa406b3e0 msgr2=0x7f0aa406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:44.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0aa406b3e0 0x7f0aa406d8a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0036000 tx=0x7f0ab0008030 comp rx=0 tx=0).stop 2024-03-16T19:31:44.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 msgr2=0x7f0ac00c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:44.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 0x7f0ac00c44c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800efc0 tx=0x7f0ab800c490 comp rx=0 tx=0).stop 2024-03-16T19:31:44.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 shutdown_connections 2024-03-16T19:31:44.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ac0154e50 0x7f0ac00c44c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:44.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0aa406b3e0 0x7f0aa406d8a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:44.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ac00caf40 0x7f0ac00c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:44.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 --2- 172.21.15.172:0/2641032175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ac00b5f10 0x7f0ac01529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:44.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 >> 172.21.15.172:0/2641032175 conn(0x7f0ac0066c20 msgr2=0x7f0ac00c8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:44.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 shutdown_connections 2024-03-16T19:31:44.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:44.950+0000 7f0ac76c0640 1 -- 172.21.15.172:0/2641032175 wait complete. 2024-03-16T19:31:46.962 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:31:46.982+0000 7f1aaf424640 -1 osd.2 58 log_to_monitors true 2024-03-16T19:31:47.964 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:31:47.984+0000 7f1aa31c9640 -1 osd.2 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T19:31:48.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-16T19:31:48.140 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:31:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-16T19:31:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-16T19:31:48.140 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:31:48.149 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-16T19:31:48.149 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-03-16T19:31:48.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-03-16T19:31:48.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.273+0000 7f7d2651e640 1 Processor -- start 2024-03-16T19:31:48.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.273+0000 7f7d2651e640 1 -- start start 2024-03-16T19:31:48.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.273+0000 7f7d2651e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7d20150ce0 0x7f7d200c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:48.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.273+0000 7f7d2651e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2006e080 con 0x7f7d2006dbd0 2024-03-16T19:31:48.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.274+0000 7f7d2651e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200c1da0 con 0x7f7d20156dc0 2024-03-16T19:31:48.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.274+0000 7f7d2651e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d20157370 con 0x7f7d20150ce0 2024-03-16T19:31:48.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.274+0000 7f7d1ffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7d20150ce0 0x7f7d200c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:48.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.274+0000 7f7d24abb640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7d2006dbd0 0x7f7d2006b760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51308/0 (socket says 172.21.15.172:51308) 2024-03-16T19:31:48.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.274+0000 7f7d24abb640 1 -- 172.21.15.172:0/1086103150 learned_addr learned my addr 172.21.15.172:0/1086103150 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:48.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.274+0000 7f7d1ffff640 1 -- 172.21.15.172:0/1086103150 >> v1:172.21.15.174:6789/0 conn(0x7f7d20156dc0 legacy=0x7f7d200c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:48.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.274+0000 7f7d1ffff640 1 -- 172.21.15.172:0/1086103150 >> v1:172.21.15.172:6789/0 conn(0x7f7d2006dbd0 legacy=0x7f7d2006b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:48.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.274+0000 7f7d1effd640 1 -- 172.21.15.172:0/1086103150 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277140033 0 0) 0x7f7d200c1da0 con 0x7f7d20156dc0 2024-03-16T19:31:48.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.274+0000 7f7d1ffff640 1 -- 172.21.15.172:0/1086103150 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d20157650 con 0x7f7d20150ce0 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d1ffff640 1 --2- 172.21.15.172:0/1086103150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20150ce0 0x7f7d200c84b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000ba40 tx=0x7f7d1002dff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b83795cfb1d72d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d1effd640 1 -- 172.21.15.172:0/1086103150 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1002ed20 con 0x7f7d20150ce0 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d1effd640 1 -- 172.21.15.172:0/1086103150 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1002ee80 con 0x7f7d20150ce0 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d1effd640 1 -- 172.21.15.172:0/1086103150 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10036660 con 0x7f7d20150ce0 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d2651e640 1 -- 172.21.15.172:0/1086103150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20150ce0 msgr2=0x7f7d200c84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d2651e640 1 --2- 172.21.15.172:0/1086103150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20150ce0 0x7f7d200c84b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000ba40 tx=0x7f7d1002dff0 comp rx=0 tx=0).stop 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d2651e640 1 -- 172.21.15.172:0/1086103150 shutdown_connections 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d2651e640 1 --2- 172.21.15.172:0/1086103150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20150ce0 0x7f7d200c84b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d2651e640 1 -- 172.21.15.172:0/1086103150 >> 172.21.15.172:0/1086103150 conn(0x7f7d200cc030 msgr2=0x7f7d200cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d2651e640 1 -- 172.21.15.172:0/1086103150 shutdown_connections 2024-03-16T19:31:48.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.275+0000 7f7d2651e640 1 -- 172.21.15.172:0/1086103150 wait complete. 2024-03-16T19:31:48.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.277+0000 7f7d2651e640 1 Processor -- start 2024-03-16T19:31:48.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.277+0000 7f7d2651e640 1 -- start start 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d2651e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d2006dbd0 0x7f7d2014da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d2651e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d20150ce0 0x7f7d201556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d2651e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20156dc0 0x7f7d20145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d2651e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201ef2c0 con 0x7f7d20156dc0 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d2651e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d200ca030 con 0x7f7d20150ce0 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d2651e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2014c5b0 con 0x7f7d2006dbd0 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d24abb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20156dc0 0x7f7d20145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d1ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d2006dbd0 0x7f7d2014da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d1f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d20150ce0 0x7f7d201556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d24abb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20156dc0 0x7f7d20145900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42682/0 (socket says 172.21.15.172:42682) 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d24abb640 1 -- 172.21.15.172:0/30743367 learned_addr learned my addr 172.21.15.172:0/30743367 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:48.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d24abb640 1 -- 172.21.15.172:0/30743367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d2006dbd0 msgr2=0x7f7d2014da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d24abb640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d2006dbd0 0x7f7d2014da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d24abb640 1 -- 172.21.15.172:0/30743367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d20150ce0 msgr2=0x7f7d201556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d24abb640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d20150ce0 0x7f7d201556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d24abb640 1 -- 172.21.15.172:0/30743367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1000b6a0 con 0x7f7d20156dc0 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d24abb640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20156dc0 0x7f7d20145900 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800caf0 tx=0x7f7d18002130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d1cff9640 1 -- 172.21.15.172:0/30743367 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1800a760 con 0x7f7d20156dc0 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d1cff9640 1 -- 172.21.15.172:0/30743367 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1800a8e0 con 0x7f7d20156dc0 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d201ef4d0 con 0x7f7d20156dc0 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d1cff9640 1 -- 172.21.15.172:0/30743367 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1800ad10 con 0x7f7d20156dc0 2024-03-16T19:31:48.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.278+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d201efa90 con 0x7f7d20156dc0 2024-03-16T19:31:48.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.279+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d20075a70 con 0x7f7d20156dc0 2024-03-16T19:31:48.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.280+0000 7f7d1cff9640 1 -- 172.21.15.172:0/30743367 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7d180070d0 con 0x7f7d20156dc0 2024-03-16T19:31:48.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.280+0000 7f7d1cff9640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d0c06b3e0 0x7f7d0c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:48.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.280+0000 7f7d1cff9640 1 -- 172.21.15.172:0/30743367 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f7d180894c0 con 0x7f7d20156dc0 2024-03-16T19:31:48.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.280+0000 7f7d1ffff640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d0c06b3e0 0x7f7d0c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:48.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.281+0000 7f7d1ffff640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d0c06b3e0 0x7f7d0c06d8a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7d10038000 tx=0x7f7d10032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:48.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.285+0000 7f7d1cff9640 1 -- 172.21.15.172:0/30743367 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d18054100 con 0x7f7d20156dc0 2024-03-16T19:31:48.444 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.465+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd in", "ids": ["0"]} v 0) v1 -- 0x7f7d2006e080 con 0x7f7d20156dc0 2024-03-16T19:31:48.445 INFO:tasks.ceph.mon.a.smithi172.stderr:2024-03-16T19:31:48.465+0000 7f2459211640 -1 mon.a@0(leader).osd e87 definitely_dead 0 2024-03-16T19:31:48.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.960+0000 7f7d1cff9640 1 -- 172.21.15.172:0/30743367 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["0"]}]=0 marked in osd.0. v88) v1 ==== 85+0+0 (secure 0 0 0) 0x7f7d2006e080 con 0x7f7d20156dc0 2024-03-16T19:31:48.940 INFO:teuthology.orchestra.run.smithi172.stderr:marked in osd.0. 2024-03-16T19:31:48.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.963+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d0c06b3e0 msgr2=0x7f7d0c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:48.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.963+0000 7f7d2651e640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d0c06b3e0 0x7f7d0c06d8a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7d10038000 tx=0x7f7d10032000 comp rx=0 tx=0).stop 2024-03-16T19:31:48.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.963+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20156dc0 msgr2=0x7f7d20145900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:48.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.963+0000 7f7d2651e640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20156dc0 0x7f7d20145900 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800caf0 tx=0x7f7d18002130 comp rx=0 tx=0).stop 2024-03-16T19:31:48.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.966+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 shutdown_connections 2024-03-16T19:31:48.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.966+0000 7f7d2651e640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d20156dc0 0x7f7d20145900 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:48.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.967+0000 7f7d2651e640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d20150ce0 0x7f7d201556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:48.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.967+0000 7f7d2651e640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d0c06b3e0 0x7f7d0c06d8a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:48.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.967+0000 7f7d2651e640 1 --2- 172.21.15.172:0/30743367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d2006dbd0 0x7f7d2014da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:48.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.967+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 >> 172.21.15.172:0/30743367 conn(0x7f7d200cc030 msgr2=0x7f7d2014fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:48.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.973+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 shutdown_connections 2024-03-16T19:31:48.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:48.974+0000 7f7d2651e640 1 -- 172.21.15.172:0/30743367 wait complete. 2024-03-16T19:31:48.968 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-03-16T19:31:48.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-16T19:31:49.068 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.088+0000 7ff3b692c640 1 Processor -- start 2024-03-16T19:31:49.068 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.088+0000 7ff3b692c640 1 -- start start 2024-03-16T19:31:49.069 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3b692c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b00cb560 0x7ff3b015e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3b692c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff3b006bc60 0x7ff3b006c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3b692c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b00cb360 con 0x7ff3b0154b90 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3b692c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b015ede0 con 0x7ff3b006bc60 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3b692c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00cb9a0 con 0x7ff3b00cb560 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3affff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff3b0154b90 0x7ff3b00cd1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:58124/0 (socket says 172.21.15.172:58124) 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3affff640 1 -- 172.21.15.172:0/2224248259 learned_addr learned my addr 172.21.15.172:0/2224248259 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3b4ec9640 1 --2- 172.21.15.172:0/2224248259 >> v2:172.21.15.172:3300/0 conn(0x7ff3b006bc60 0x7ff3b006c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/2224248259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b00cb560 0x7ff3b015e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.089+0000 7ff3aeffd640 1 -- 172.21.15.172:0/2224248259 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556849363 0 0) 0x7ff3b00cb360 con 0x7ff3b0154b90 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.090+0000 7ff3aeffd640 1 -- 172.21.15.172:0/2224248259 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff398003520 con 0x7ff3b0154b90 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.090+0000 7ff3b4ec9640 1 -- 172.21.15.172:0/2224248259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b00cb560 msgr2=0x7ff3b015e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.090+0000 7ff3b4ec9640 1 --2- 172.21.15.172:0/2224248259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b00cb560 0x7ff3b015e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.090+0000 7ff3b4ec9640 1 -- 172.21.15.172:0/2224248259 >> v1:172.21.15.172:6789/0 conn(0x7ff3b0154b90 legacy=0x7ff3b00cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.090+0000 7ff3b4ec9640 1 -- 172.21.15.172:0/2224248259 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b015a950 con 0x7ff3b006bc60 2024-03-16T19:31:49.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.090+0000 7ff3b4ec9640 1 --2- 172.21.15.172:0/2224248259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b006bc60 0x7ff3b006c0b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009d90 tx=0x7ff3a802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee28e1d7c38ed9b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.090+0000 7ff3aeffd640 1 -- 172.21.15.172:0/2224248259 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a802cc50 con 0x7ff3b006bc60 2024-03-16T19:31:49.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.090+0000 7ff3aeffd640 1 -- 172.21.15.172:0/2224248259 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a802cdb0 con 0x7ff3b006bc60 2024-03-16T19:31:49.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.090+0000 7ff3aeffd640 1 -- 172.21.15.172:0/2224248259 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8034660 con 0x7ff3b006bc60 2024-03-16T19:31:49.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.091+0000 7ff3b692c640 1 -- 172.21.15.172:0/2224248259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b006bc60 msgr2=0x7ff3b006c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.091+0000 7ff3b692c640 1 --2- 172.21.15.172:0/2224248259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b006bc60 0x7ff3b006c0b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009d90 tx=0x7ff3a802bea0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.091+0000 7ff3b692c640 1 -- 172.21.15.172:0/2224248259 shutdown_connections 2024-03-16T19:31:49.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.091+0000 7ff3b692c640 1 --2- 172.21.15.172:0/2224248259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b00cb560 0x7ff3b015e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.072 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.091+0000 7ff3b692c640 1 --2- 172.21.15.172:0/2224248259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b006bc60 0x7ff3b006c0b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.091+0000 7ff3b692c640 1 -- 172.21.15.172:0/2224248259 >> 172.21.15.172:0/2224248259 conn(0x7ff3b0066e00 msgr2=0x7ff3b0067210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:49.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.091+0000 7ff3b692c640 1 -- 172.21.15.172:0/2224248259 shutdown_connections 2024-03-16T19:31:49.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.091+0000 7ff3b692c640 1 -- 172.21.15.172:0/2224248259 wait complete. 2024-03-16T19:31:49.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.092+0000 7ff3b692c640 1 Processor -- start 2024-03-16T19:31:49.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.092+0000 7ff3b692c640 1 -- start start 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b692c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b006bc60 0x7ff3b014bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b692c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b00cb560 0x7ff3b014c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b692c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b0154b90 0x7ff3b01488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b692c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01f0ee0 con 0x7ff3b0154b90 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b692c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b0148450 con 0x7ff3b00cb560 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b692c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00b5eb0 con 0x7ff3b006bc60 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b4ec9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b0154b90 0x7ff3b01488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b4ec9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b0154b90 0x7ff3b01488e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52550/0 (socket says 172.21.15.172:52550) 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b4ec9640 1 -- 172.21.15.172:0/1776792525 learned_addr learned my addr 172.21.15.172:0/1776792525 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3affff640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b006bc60 0x7ff3b014bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.075 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b00cb560 0x7ff3b014c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b4ec9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b006bc60 msgr2=0x7ff3b014bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b4ec9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b006bc60 0x7ff3b014bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b4ec9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b00cb560 msgr2=0x7ff3b014c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b4ec9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b00cb560 0x7ff3b014c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.093+0000 7ff3b4ec9640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a8009a70 con 0x7ff3b0154b90 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.094+0000 7ff3b4ec9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b0154b90 0x7ff3b01488e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009eb0 tx=0x7ff3a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.094+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8040650 con 0x7ff3b0154b90 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.094+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a80407b0 con 0x7ff3b0154b90 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.094+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8040b60 con 0x7ff3b0154b90 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.094+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b00b6190 con 0x7ff3b0154b90 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.094+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b01480e0 con 0x7ff3b0154b90 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.095+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3b0075a70 con 0x7ff3b0154b90 2024-03-16T19:31:49.076 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.095+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff3a8040cc0 con 0x7ff3b0154b90 2024-03-16T19:31:49.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.095+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff39c06b3b0 0x7ff39c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5886+0+0 (secure 0 0 0) 0x7ff3a80ac7b0 con 0x7ff3b0154b90 2024-03-16T19:31:49.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3affff640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff39c06b3b0 0x7ff39c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+1289 (secure 0 0 0) 0x7ff3a80b3090 con 0x7ff3b0154b90 2024-03-16T19:31:49.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3b692c640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b5380 0x7ff3b0148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b0074d00 con 0x7ff3b00b5380 2024-03-16T19:31:49.078 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3affff640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff39c06b3b0 0x7ff39c06d870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4005880 tx=0x7ff3a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b5380 0x7ff3b0148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3af7fe640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3b015aed0 con 0x7ff3b00b5380 2024-03-16T19:31:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b5380 0x7ff3b0148e20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff3b01587e0 tx=0x7ff3a0011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.096+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0004300 con 0x7ff3b00b5380 2024-03-16T19:31:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.097+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7ff3a0004460 con 0x7ff3b00b5380 2024-03-16T19:31:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.097+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b5380 msgr2=0x7ff3b0148e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.097+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b5380 0x7ff3b0148e20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff3b01587e0 tx=0x7ff3a0011d40 comp rx=0 tx=0).stop 2024-03-16T19:31:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.097+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b5380 msgr2=0x7ff3b0148e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:31:49.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.097+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b5380 0x7ff3b0148e20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.103+0000 7ff3b692c640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b42f0 0x7ff3b00cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.103+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01482d0 con 0x7ff3b00b42f0 2024-03-16T19:31:49.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.103+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b42f0 0x7ff3b00cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.103+0000 7ff3af7fe640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7ff3b015abf0 con 0x7ff3b00b42f0 2024-03-16T19:31:49.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.104+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b42f0 0x7ff3b00cf6b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff3b014cd20 tx=0x7ff3a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.104+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a001e650 con 0x7ff3b00b42f0 2024-03-16T19:31:49.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.105+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.0 v2:172.21.15.172:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7ff3a001e870 con 0x7ff3b00b42f0 2024-03-16T19:31:49.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.105+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b42f0 msgr2=0x7ff3b00cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.105+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b42f0 0x7ff3b00cf6b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff3b014cd20 tx=0x7ff3a00046c0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.105+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b42f0 msgr2=0x7ff3b00cf6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:31:49.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.105+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b00b42f0 0x7ff3b00cf6b0 secure :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff3b014cd20 tx=0x7ff3a00046c0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.085 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a: {} 2024-03-16T19:31:49.086 INFO:teuthology.orchestra.run.smithi172.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.105+0000 7ff3b692c640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01f0910 0x7ff3b00cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.105+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00d0130 con 0x7ff3b01f0910 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.105+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01f0910 0x7ff3b00cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.106+0000 7ff3af7fe640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3b0148590 con 0x7ff3b01f0910 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.106+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01f0910 0x7ff3b00cfbf0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0012010 tx=0x7ff3a0011d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.106+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00065d0 con 0x7ff3b01f0910 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.106+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7ff3a0006730 con 0x7ff3b01f0910 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.106+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01f0910 msgr2=0x7ff3b00cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.106+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01f0910 0x7ff3b00cfbf0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0012010 tx=0x7ff3a0011d40 comp rx=0 tx=0).stop 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.106+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01f0910 msgr2=0x7ff3b00cfbf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.106+0000 7ff3b4ec9640 1 -- 172.21.15.172:0/1776792525 reap_dead start 2024-03-16T19:31:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.106+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01f0910 0x7ff3b00cfbf0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.112+0000 7ff3b692c640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b014bf60 0x7ff3b014c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.112+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b014c880 con 0x7ff3b014bf60 2024-03-16T19:31:49.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.112+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b014bf60 0x7ff3b014c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.113+0000 7ff3af7fe640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7ff3b014e8a0 con 0x7ff3b014bf60 2024-03-16T19:31:49.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.113+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b014bf60 0x7ff3b014c340 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000c00 tx=0x7ff3a002c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.113+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0041070 con 0x7ff3b014bf60 2024-03-16T19:31:49.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.114+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7ff3a003c0e0 con 0x7ff3b014bf60 2024-03-16T19:31:49.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.114+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b014bf60 msgr2=0x7ff3b014c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.114+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b014bf60 0x7ff3b014c340 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000c00 tx=0x7ff3a002c000 comp rx=0 tx=0).stop 2024-03-16T19:31:49.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.114+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b014bf60 msgr2=0x7ff3b014c340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:31:49.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.114+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b014bf60 0x7ff3b014c340 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.095 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b: {} 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.114+0000 7ff3b692c640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3480 0x7ff3b01f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.114+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00764e0 con 0x7ff3b01f3480 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.115+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3480 0x7ff3b01f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.115+0000 7ff3af7fe640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3b00d10d0 con 0x7ff3b01f3480 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.115+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3480 0x7ff3b01f3860 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009cf0 tx=0x7ff3a0011d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.115+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a004f640 con 0x7ff3b01f3480 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.115+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.2 v2:172.21.15.174:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7ff3a004f7a0 con 0x7ff3b01f3480 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.115+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3480 msgr2=0x7ff3b01f3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.115+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3480 0x7ff3b01f3860 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009cf0 tx=0x7ff3a0011d40 comp rx=0 tx=0).stop 2024-03-16T19:31:49.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.116+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3480 msgr2=0x7ff3b01f3860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:31:49.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.116+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3480 0x7ff3b01f3860 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.121+0000 7ff3b692c640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3da0 0x7ff3b01593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.121+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01f4180 con 0x7ff3b01f3da0 2024-03-16T19:31:49.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.122+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3da0 0x7ff3b01593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.122+0000 7ff3af7fe640 1 -- 172.21.15.172:0/1776792525 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7ff3b0159910 con 0x7ff3b01f3da0 2024-03-16T19:31:49.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.122+0000 7ff3af7fe640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3da0 0x7ff3b01593d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0011300 tx=0x7ff3a0048000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.122+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0055310 con 0x7ff3b01f3da0 2024-03-16T19:31:49.103 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.123+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 <== mon.2 v2:172.21.15.174:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7ff3a004fbe0 con 0x7ff3b01f3da0 2024-03-16T19:31:49.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.123+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3da0 msgr2=0x7ff3b01593d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.123+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3da0 0x7ff3b01593d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0011300 tx=0x7ff3a0048000 comp rx=0 tx=0).stop 2024-03-16T19:31:49.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.123+0000 7ff3acff9640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3da0 msgr2=0x7ff3b01593d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-16T19:31:49.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.123+0000 7ff3acff9640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3b01f3da0 0x7ff3b01593d0 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0011300 tx=0x7ff3a0048000 comp rx=0 tx=0).stop 2024-03-16T19:31:49.104 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c: {} 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.124+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff39c06b3b0 msgr2=0x7ff39c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.124+0000 7ff3b692c640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff39c06b3b0 0x7ff39c06d870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4005880 tx=0x7ff3a4009040 comp rx=0 tx=0).stop 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.124+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b0154b90 msgr2=0x7ff3b01488e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.124+0000 7ff3b692c640 1 --2- 172.21.15.172:0/1776792525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3b0154b90 0x7ff3b01488e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009eb0 tx=0x7ff3a8004a40 comp rx=0 tx=0).stop 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.124+0000 7ff3b4ec9640 1 -- 172.21.15.172:0/1776792525 reap_dead start 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.124+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 shutdown_connections 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.124+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 >> 172.21.15.172:0/1776792525 conn(0x7ff3b0066e00 msgr2=0x7ff3b00d0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.124+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 shutdown_connections 2024-03-16T19:31:49.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.124+0000 7ff3b692c640 1 -- 172.21.15.172:0/1776792525 wait complete. 2024-03-16T19:31:49.114 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-16T19:31:49.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.237+0000 7f3739ce1640 1 Processor -- start 2024-03-16T19:31:49.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.237+0000 7f3739ce1640 1 -- start start 2024-03-16T19:31:49.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f3739ce1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f373415b480 0x7f3734166490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f3739ce1640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37340ceb20 con 0x7f3734158120 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f3739ce1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f373406c160 con 0x7f37340b0e90 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f3739ce1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f373406c6c0 con 0x7f373415b480 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f37337fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3734158120 0x7f373406f2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:58134/0 (socket says 172.21.15.172:58134) 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f37337fe640 1 -- 172.21.15.172:0/3366569669 learned_addr learned my addr 172.21.15.172:0/3366569669 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f3733fff640 1 --2- 172.21.15.172:0/3366569669 >> v2:172.21.15.174:3300/0 conn(0x7f373415b480 0x7f3734166490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f37327fc640 1 -- 172.21.15.172:0/3366569669 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462401985 0 0) 0x7f37340ceb20 con 0x7f3734158120 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f37327fc640 1 -- 172.21.15.172:0/3366569669 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f371c003520 con 0x7f3734158120 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f37327fc640 1 -- 172.21.15.172:0/3366569669 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659849006 0 0) 0x7f373406c160 con 0x7f37340b0e90 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f3733fff640 1 -- 172.21.15.172:0/3366569669 >> v1:172.21.15.174:6789/0 conn(0x7f37340b0e90 legacy=0x7f373415ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f3733fff640 1 -- 172.21.15.172:0/3366569669 >> v1:172.21.15.172:6789/0 conn(0x7f3734158120 legacy=0x7f373406f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.238+0000 7f3733fff640 1 -- 172.21.15.172:0/3366569669 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373406c400 con 0x7f373415b480 2024-03-16T19:31:49.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.239+0000 7f3733fff640 1 --2- 172.21.15.172:0/3366569669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f373415b480 0x7f3734166490 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f372c009d90 tx=0x7f372c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d320a3ff3b5e8f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.239+0000 7f37327fc640 1 -- 172.21.15.172:0/3366569669 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c02cc60 con 0x7f373415b480 2024-03-16T19:31:49.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.239+0000 7f37327fc640 1 -- 172.21.15.172:0/3366569669 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372c02cdc0 con 0x7f373415b480 2024-03-16T19:31:49.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.239+0000 7f37327fc640 1 -- 172.21.15.172:0/3366569669 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c034640 con 0x7f373415b480 2024-03-16T19:31:49.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.240+0000 7f3739ce1640 1 -- 172.21.15.172:0/3366569669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f373415b480 msgr2=0x7f3734166490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.240+0000 7f3739ce1640 1 --2- 172.21.15.172:0/3366569669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f373415b480 0x7f3734166490 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f372c009d90 tx=0x7f372c02beb0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.240+0000 7f3739ce1640 1 -- 172.21.15.172:0/3366569669 shutdown_connections 2024-03-16T19:31:49.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.240+0000 7f3739ce1640 1 --2- 172.21.15.172:0/3366569669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f373415b480 0x7f3734166490 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.240+0000 7f3739ce1640 1 -- 172.21.15.172:0/3366569669 >> 172.21.15.172:0/3366569669 conn(0x7f3734066c20 msgr2=0x7f3734067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:49.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.240+0000 7f3739ce1640 1 -- 172.21.15.172:0/3366569669 shutdown_connections 2024-03-16T19:31:49.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.240+0000 7f3739ce1640 1 -- 172.21.15.172:0/3366569669 wait complete. 2024-03-16T19:31:49.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.241+0000 7f3739ce1640 1 Processor -- start 2024-03-16T19:31:49.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.241+0000 7f3739ce1640 1 -- start start 2024-03-16T19:31:49.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3739ce1640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37340b0e90 0x7f37340b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3739ce1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3734158120 0x7f37341f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3739ce1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f373415b480 0x7f37340c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3739ce1640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37340b0620 con 0x7f37340b0e90 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3739ce1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37341f41d0 con 0x7f373415b480 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3739ce1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37340c83a0 con 0x7f3734158120 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f37337fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37340b0e90 0x7f37340b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f37337fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37340b0e90 0x7f37340b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52578/0 (socket says 172.21.15.172:52578) 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f37337fe640 1 -- 172.21.15.172:0/3824794176 learned_addr learned my addr 172.21.15.172:0/3824794176 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3732ffd640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3734158120 0x7f37341f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3733fff640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f373415b480 0x7f37340c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3732ffd640 1 -- 172.21.15.172:0/3824794176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f373415b480 msgr2=0x7f37340c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3732ffd640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f373415b480 0x7f37340c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3732ffd640 1 -- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37340b0e90 msgr2=0x7f37340b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3732ffd640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37340b0e90 0x7f37340b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f3732ffd640 1 -- 172.21.15.172:0/3824794176 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c009a70 con 0x7f3734158120 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.242+0000 7f37337fe640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37340b0e90 0x7f37340b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.243+0000 7f3732ffd640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3734158120 0x7f37341f3c90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f372800c450 tx=0x7f372800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.243+0000 7f371bfff640 1 -- 172.21.15.172:0/3824794176 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37280035f0 con 0x7f3734158120 2024-03-16T19:31:49.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.243+0000 7f371bfff640 1 -- 172.21.15.172:0/3824794176 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3728003750 con 0x7f3734158120 2024-03-16T19:31:49.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.243+0000 7f371bfff640 1 -- 172.21.15.172:0/3824794176 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728003b00 con 0x7f3734158120 2024-03-16T19:31:49.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.243+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37340c8680 con 0x7f3734158120 2024-03-16T19:31:49.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.243+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37340af8e0 con 0x7f3734158120 2024-03-16T19:31:49.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.244+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37340af460 con 0x7f3734158120 2024-03-16T19:31:49.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.244+0000 7f371bfff640 1 -- 172.21.15.172:0/3824794176 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3728003c60 con 0x7f3734158120 2024-03-16T19:31:49.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.245+0000 7f371bfff640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f372006b3b0 0x7f372006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:49.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.245+0000 7f371bfff640 1 -- 172.21.15.172:0/3824794176 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5886+0+0 (secure 0 0 0) 0x7f37280891b0 con 0x7f3734158120 2024-03-16T19:31:49.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.245+0000 7f37337fe640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f372006b3b0 0x7f372006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:49.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.245+0000 7f37337fe640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f372006b3b0 0x7f372006d870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3724005880 tx=0x7f3724009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:49.233 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.253+0000 7f371bfff640 1 -- 172.21.15.172:0/3824794176 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37340af460 con 0x7f3734158120 2024-03-16T19:31:49.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.436+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f37340af460 con 0x7f3734158120 2024-03-16T19:31:49.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.965+0000 7f371bfff640 1 -- 172.21.15.172:0/3824794176 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v89) v1 ==== 113+0+0 (secure 0 0 0) 0x7f37340af460 con 0x7f3734158120 2024-03-16T19:31:49.952 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.0 to 1 (10000) 2024-03-16T19:31:49.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f372006b3b0 msgr2=0x7f372006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f372006b3b0 0x7f372006d870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3724005880 tx=0x7f3724009040 comp rx=0 tx=0).stop 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3734158120 msgr2=0x7f37341f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3734158120 0x7f37341f3c90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f372800c450 tx=0x7f372800c910 comp rx=0 tx=0).stop 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 shutdown_connections 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f373415b480 0x7f37340c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f372006b3b0 0x7f372006d870 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3734158120 0x7f37341f3c90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 --2- 172.21.15.172:0/3824794176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37340b0e90 0x7f37340b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.977+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 >> 172.21.15.172:0/3824794176 conn(0x7f3734066c20 msgr2=0x7f37340c39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.978+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 shutdown_connections 2024-03-16T19:31:49.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:49.978+0000 7f3739ce1640 1 -- 172.21.15.172:0/3824794176 wait complete. 2024-03-16T19:31:49.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-16T19:31:50.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.099+0000 7f8f91bc7640 1 Processor -- start 2024-03-16T19:31:50.079 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.099+0000 7f8f91bc7640 1 -- start start 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.099+0000 7f8f91bc7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8f8c152760 0x7f8f8c06dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.099+0000 7f8f91bc7640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8f8c0ca8e0 0x7f8f8c06e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f91bc7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c06b2e0 con 0x7f8f8c0c7b60 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f91bc7640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f8c06bfe0 con 0x7f8f8c0ca8e0 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f91bc7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f8c06cb70 con 0x7f8f8c152760 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f8affd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8f8c0ca8e0 0x7f8f8c06e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f8bfff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8f8c0c7b60 0x7f8f8c15bd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:58136/0 (socket says 172.21.15.172:58136) 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f8bfff640 1 -- 172.21.15.172:0/3603015027 learned_addr learned my addr 172.21.15.172:0/3603015027 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f8b7fe640 1 --2- 172.21.15.172:0/3603015027 >> v2:172.21.15.174:3300/0 conn(0x7f8f8c152760 0x7f8f8c06dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f8affd640 1 -- 172.21.15.172:0/3603015027 >> v2:172.21.15.174:3300/0 conn(0x7f8f8c152760 msgr2=0x7f8f8c06dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:50.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f8affd640 1 --2- 172.21.15.172:0/3603015027 >> v2:172.21.15.174:3300/0 conn(0x7f8f8c152760 0x7f8f8c06dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f8affd640 1 -- 172.21.15.172:0/3603015027 >> v1:172.21.15.172:6789/0 conn(0x7f8f8c0c7b60 legacy=0x7f8f8c15bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:50.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f8affd640 1 -- 172.21.15.172:0/3603015027 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c159580 con 0x7f8f8c0ca8e0 2024-03-16T19:31:50.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.100+0000 7f8f8a7fc640 1 -- 172.21.15.172:0/3603015027 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080333703 0 0) 0x7f8f8c06b2e0 con 0x7f8f8c0c7b60 2024-03-16T19:31:50.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f8affd640 1 --2- 172.21.15.172:0/3603015027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c0ca8e0 0x7f8f8c06e2b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0c6a20 tx=0x7f8f7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33d3fb1474f54eb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:50.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f8a7fc640 1 -- 172.21.15.172:0/3603015027 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c031830 con 0x7f8f8c0ca8e0 2024-03-16T19:31:50.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f8a7fc640 1 -- 172.21.15.172:0/3603015027 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c031990 con 0x7f8f8c0ca8e0 2024-03-16T19:31:50.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f8a7fc640 1 -- 172.21.15.172:0/3603015027 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c031d20 con 0x7f8f8c0ca8e0 2024-03-16T19:31:50.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f91bc7640 1 -- 172.21.15.172:0/3603015027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c0ca8e0 msgr2=0x7f8f8c06e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:50.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f91bc7640 1 --2- 172.21.15.172:0/3603015027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c0ca8e0 0x7f8f8c06e2b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0c6a20 tx=0x7f8f7c02b8d0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f91bc7640 1 -- 172.21.15.172:0/3603015027 shutdown_connections 2024-03-16T19:31:50.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f91bc7640 1 --2- 172.21.15.172:0/3603015027 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c0ca8e0 0x7f8f8c06e2b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f91bc7640 1 --2- 172.21.15.172:0/3603015027 >> v2:172.21.15.174:3300/0 conn(0x7f8f8c152760 0x7f8f8c06dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f91bc7640 1 -- 172.21.15.172:0/3603015027 >> 172.21.15.172:0/3603015027 conn(0x7f8f8c071be0 msgr2=0x7f8f8c071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:50.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f91bc7640 1 -- 172.21.15.172:0/3603015027 shutdown_connections 2024-03-16T19:31:50.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.101+0000 7f8f91bc7640 1 -- 172.21.15.172:0/3603015027 wait complete. 2024-03-16T19:31:50.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.103+0000 7f8f91bc7640 1 Processor -- start 2024-03-16T19:31:50.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.103+0000 7f8f91bc7640 1 -- start start 2024-03-16T19:31:50.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.103+0000 7f8f91bc7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f8c0c7b60 0x7f8f8c0c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:50.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.103+0000 7f8f91bc7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f8c0ca8e0 0x7f8f8c14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:50.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.103+0000 7f8f91bc7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c152760 0x7f8f8c142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:50.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.103+0000 7f8f91bc7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c1e1ba0 con 0x7f8f8c152760 2024-03-16T19:31:50.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.103+0000 7f8f91bc7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c1e1dd0 con 0x7f8f8c0c7b60 2024-03-16T19:31:50.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.103+0000 7f8f91bc7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c157160 con 0x7f8f8c0ca8e0 2024-03-16T19:31:50.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.103+0000 7f8f8bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c152760 0x7f8f8c142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:50.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c152760 0x7f8f8c142cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52596/0 (socket says 172.21.15.172:52596) 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8bfff640 1 -- 172.21.15.172:0/2802549413 learned_addr learned my addr 172.21.15.172:0/2802549413 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8b7fe640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f8c0c7b60 0x7f8f8c0c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8affd640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f8c0ca8e0 0x7f8f8c14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8bfff640 1 -- 172.21.15.172:0/2802549413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f8c0ca8e0 msgr2=0x7f8f8c14fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8bfff640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f8c0ca8e0 0x7f8f8c14fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8bfff640 1 -- 172.21.15.172:0/2802549413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f8c0c7b60 msgr2=0x7f8f8c0c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8bfff640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f8c0c7b60 0x7f8f8c0c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8bfff640 1 -- 172.21.15.172:0/2802549413 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c009400 con 0x7f8f8c152760 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f8bfff640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c152760 0x7f8f8c142cb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400c800 tx=0x7f8f8400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f73fff640 1 -- 172.21.15.172:0/2802549413 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84004d60 con 0x7f8f8c152760 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f73fff640 1 -- 172.21.15.172:0/2802549413 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8400a640 con 0x7f8f8c152760 2024-03-16T19:31:50.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f73fff640 1 -- 172.21.15.172:0/2802549413 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8400aa30 con 0x7f8f8c152760 2024-03-16T19:31:50.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8c1e2090 con 0x7f8f8c152760 2024-03-16T19:31:50.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.104+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8c156680 con 0x7f8f8c152760 2024-03-16T19:31:50.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.105+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f8c0681c0 con 0x7f8f8c152760 2024-03-16T19:31:50.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.106+0000 7f8f73fff640 1 -- 172.21.15.172:0/2802549413 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8f8400ab90 con 0x7f8f8c152760 2024-03-16T19:31:50.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.106+0000 7f8f73fff640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8f7806b3b0 0x7f8f7806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:50.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.106+0000 7f8f73fff640 1 -- 172.21.15.172:0/2802549413 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f8f84089370 con 0x7f8f8c152760 2024-03-16T19:31:50.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.106+0000 7f8f8b7fe640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8f7806b3b0 0x7f8f7806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:50.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.107+0000 7f8f8b7fe640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8f7806b3b0 0x7f8f7806d870 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009aa0 tx=0x7f8f80009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:50.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.110+0000 7f8f73fff640 1 -- 172.21.15.172:0/2802549413 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f84054040 con 0x7f8f8c152760 2024-03-16T19:31:50.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.292+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f8f8c0ca6c0 con 0x7f8f8c152760 2024-03-16T19:31:50.953 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.973+0000 7f8f73fff640 1 -- 172.21.15.172:0/2802549413 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8f84018070 con 0x7f8f8c152760 2024-03-16T19:31:50.954 INFO:teuthology.orchestra.run.smithi172.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-16T19:31:50.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8f7806b3b0 msgr2=0x7f8f7806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:50.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8f7806b3b0 0x7f8f7806d870 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009aa0 tx=0x7f8f80009190 comp rx=0 tx=0).stop 2024-03-16T19:31:50.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c152760 msgr2=0x7f8f8c142cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:50.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c152760 0x7f8f8c142cb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400c800 tx=0x7f8f8400ccc0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 shutdown_connections 2024-03-16T19:31:50.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f8c152760 0x7f8f8c142cb0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8f7806b3b0 0x7f8f7806d870 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f8c0ca8e0 0x7f8f8c14fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 --2- 172.21.15.172:0/2802549413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f8c0c7b60 0x7f8f8c0c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:50.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 >> 172.21.15.172:0/2802549413 conn(0x7f8f8c071be0 msgr2=0x7f8f8c0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:50.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 shutdown_connections 2024-03-16T19:31:50.957 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:50.976+0000 7f8f91bc7640 1 -- 172.21.15.172:0/2802549413 wait complete. 2024-03-16T19:31:50.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-16T19:31:51.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.104+0000 7f9098d3e640 1 Processor -- start 2024-03-16T19:31:51.084 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.104+0000 7f9098d3e640 1 -- start start 2024-03-16T19:31:51.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f9098d3e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f909414d600 0x7f90940c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f9098d3e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f90940cbee0 0x7f9094070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f9098d3e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90940cc2f0 0x7f90940710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f9098d3e640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90940c9930 con 0x7f909414d600 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f9098d3e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90940c6a70 con 0x7f90940cbee0 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f9098d3e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90940cc770 con 0x7f90940cc2f0 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f909259c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f909414d600 0x7f90940c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f909259c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f909414d600 0x7f90940c93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52604/0 (socket says 172.21.15.172:52604) 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f909259c640 1 -- 172.21.15.172:0/2485111310 learned_addr learned my addr 172.21.15.172:0/2485111310 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f9092d9d640 1 --2- 172.21.15.172:0/2485111310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90940cc2f0 0x7f90940710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f9091d9b640 1 --2- 172.21.15.172:0/2485111310 >> v2:172.21.15.174:3300/0 conn(0x7f90940cbee0 0x7f9094070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f909259c640 1 -- 172.21.15.172:0/2485111310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90940cc2f0 msgr2=0x7f90940710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:51.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f909259c640 1 --2- 172.21.15.172:0/2485111310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90940cc2f0 0x7f90940710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f909259c640 1 -- 172.21.15.172:0/2485111310 >> v2:172.21.15.174:3300/0 conn(0x7f90940cbee0 msgr2=0x7f9094070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:51.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f909259c640 1 --2- 172.21.15.172:0/2485111310 >> v2:172.21.15.174:3300/0 conn(0x7f90940cbee0 0x7f9094070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.105+0000 7f909259c640 1 -- 172.21.15.172:0/2485111310 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9094156e20 con 0x7f909414d600 2024-03-16T19:31:51.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.106+0000 7f909259c640 1 --2- 172.21.15.172:0/2485111310 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f909414d600 0x7f90940c93f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9094156d10 tx=0x7f908402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4430e1be39a6c777 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:51.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.106+0000 7f909159a640 1 -- 172.21.15.172:0/2485111310 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908400b0d0 con 0x7f909414d600 2024-03-16T19:31:51.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.106+0000 7f909159a640 1 -- 172.21.15.172:0/2485111310 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908400b230 con 0x7f909414d600 2024-03-16T19:31:51.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.106+0000 7f909159a640 1 -- 172.21.15.172:0/2485111310 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084009600 con 0x7f909414d600 2024-03-16T19:31:51.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.107+0000 7f9098d3e640 1 -- 172.21.15.172:0/2485111310 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f909414d600 msgr2=0x7f90940c93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.107+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2485111310 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f909414d600 0x7f90940c93f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9094156d10 tx=0x7f908402e170 comp rx=0 tx=0).stop 2024-03-16T19:31:51.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.107+0000 7f9098d3e640 1 -- 172.21.15.172:0/2485111310 shutdown_connections 2024-03-16T19:31:51.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.107+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2485111310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90940cc2f0 0x7f90940710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.107+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2485111310 >> v2:172.21.15.174:3300/0 conn(0x7f90940cbee0 0x7f9094070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.107+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2485111310 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f909414d600 0x7f90940c93f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.107+0000 7f9098d3e640 1 -- 172.21.15.172:0/2485111310 >> 172.21.15.172:0/2485111310 conn(0x7f9094066c20 msgr2=0x7f9094067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:51.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.107+0000 7f9098d3e640 1 -- 172.21.15.172:0/2485111310 shutdown_connections 2024-03-16T19:31:51.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.107+0000 7f9098d3e640 1 -- 172.21.15.172:0/2485111310 wait complete. 2024-03-16T19:31:51.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.108+0000 7f9098d3e640 1 Processor -- start 2024-03-16T19:31:51.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.108+0000 7f9098d3e640 1 -- start start 2024-03-16T19:31:51.091 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f9098d3e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90940cbee0 0x7f90941f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.091 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f9098d3e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90940cc2f0 0x7f909415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f9098d3e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f909414d600 0x7f9094152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f9098d3e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90941546a0 con 0x7f90940cbee0 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f9098d3e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90941f0340 con 0x7f90940cc2f0 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f9098d3e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90941f4800 con 0x7f909414d600 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f909259c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90940cbee0 0x7f90941f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f909259c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90940cbee0 0x7f90941f42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52606/0 (socket says 172.21.15.172:52606) 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f909259c640 1 -- 172.21.15.172:0/2789427287 learned_addr learned my addr 172.21.15.172:0/2789427287 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f9092d9d640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f909414d600 0x7f9094152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f9091d9b640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90940cc2f0 0x7f909415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f909259c640 1 -- 172.21.15.172:0/2789427287 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f909414d600 msgr2=0x7f9094152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f909259c640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f909414d600 0x7f9094152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f909259c640 1 -- 172.21.15.172:0/2789427287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90940cc2f0 msgr2=0x7f909415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f909259c640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90940cc2f0 0x7f909415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.109+0000 7f909259c640 1 -- 172.21.15.172:0/2789427287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908400bd40 con 0x7f90940cbee0 2024-03-16T19:31:51.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.110+0000 7f909259c640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90940cbee0 0x7f90941f42c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9084038040 tx=0x7f9084004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.110+0000 7f9082ffd640 1 -- 172.21.15.172:0/2789427287 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084040620 con 0x7f90940cbee0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.110+0000 7f9082ffd640 1 -- 172.21.15.172:0/2789427287 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90840407a0 con 0x7f90940cbee0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.110+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909415aae0 con 0x7f90940cbee0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.110+0000 7f9082ffd640 1 -- 172.21.15.172:0/2789427287 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084040b50 con 0x7f90940cbee0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.110+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90941f0550 con 0x7f90940cbee0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.111+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90940c35e0 con 0x7f90940cbee0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.111+0000 7f9082ffd640 1 -- 172.21.15.172:0/2789427287 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9084040cb0 con 0x7f90940cbee0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.112+0000 7f9082ffd640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f907406b3e0 0x7f907406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.112+0000 7f9082ffd640 1 -- 172.21.15.172:0/2789427287 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f90840ace50 con 0x7f90940cbee0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.112+0000 7f9091d9b640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f907406b3e0 0x7f907406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.112+0000 7f9091d9b640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f907406b3e0 0x7f907406d8a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f90941592e0 tx=0x7f9088009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:51.096 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.116+0000 7f9082ffd640 1 -- 172.21.15.172:0/2789427287 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9084077a60 con 0x7f90940cbee0 2024-03-16T19:31:51.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.297+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f90940681c0 con 0x7f90940cbee0 2024-03-16T19:31:51.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.800+0000 7f9082ffd640 1 -- 172.21.15.172:0/2789427287 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v91) v1 ==== 113+0+0 (secure 0 0 0) 0x7f908407b120 con 0x7f90940cbee0 2024-03-16T19:31:51.780 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.1 to 1 (10000) 2024-03-16T19:31:51.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f907406b3e0 msgr2=0x7f907406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f907406b3e0 0x7f907406d8a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f90941592e0 tx=0x7f9088009190 comp rx=0 tx=0).stop 2024-03-16T19:31:51.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90940cbee0 msgr2=0x7f90941f42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90940cbee0 0x7f90941f42c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9084038040 tx=0x7f9084004a40 comp rx=0 tx=0).stop 2024-03-16T19:31:51.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 shutdown_connections 2024-03-16T19:31:51.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f907406b3e0 0x7f907406d8a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f909414d600 0x7f9094152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90940cc2f0 0x7f909415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 --2- 172.21.15.172:0/2789427287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90940cbee0 0x7f90941f42c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 >> 172.21.15.172:0/2789427287 conn(0x7f9094066c20 msgr2=0x7f909414bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:51.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 shutdown_connections 2024-03-16T19:31:51.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.802+0000 7f9098d3e640 1 -- 172.21.15.172:0/2789427287 wait complete. 2024-03-16T19:31:51.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-16T19:31:51.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.924+0000 7fc508a23640 1 Processor -- start 2024-03-16T19:31:51.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.924+0000 7fc508a23640 1 -- start start 2024-03-16T19:31:51.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc508a23640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc50415c450 0x7fc5041640e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc508a23640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50406b2e0 con 0x7fc504152760 2024-03-16T19:31:51.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc508a23640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50406cb70 con 0x7fc5040ca8e0 2024-03-16T19:31:51.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc508a23640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc50406bfe0 con 0x7fc50415c450 2024-03-16T19:31:51.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc502d9d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc50415c450 0x7fc5041640e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc50259c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc504152760 0x7fc50406dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:58152/0 (socket says 172.21.15.172:58152) 2024-03-16T19:31:51.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc50259c640 1 -- 172.21.15.172:0/293516932 learned_addr learned my addr 172.21.15.172:0/293516932 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:51.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc502d9d640 1 -- 172.21.15.172:0/293516932 >> v1:172.21.15.174:6789/0 conn(0x7fc5040ca8e0 legacy=0x7fc50415bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc502d9d640 1 -- 172.21.15.172:0/293516932 >> v1:172.21.15.172:6789/0 conn(0x7fc504152760 legacy=0x7fc50406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc502d9d640 1 -- 172.21.15.172:0/293516932 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc504159580 con 0x7fc50415c450 2024-03-16T19:31:51.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.925+0000 7fc501d9b640 1 -- 172.21.15.172:0/293516932 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126257470 0 0) 0x7fc50406b2e0 con 0x7fc504152760 2024-03-16T19:31:51.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.926+0000 7fc502d9d640 1 --2- 172.21.15.172:0/293516932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 0x7fc5041640e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009d90 tx=0x7fc4fc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7cf749c4ac725e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:51.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.926+0000 7fc501d9b640 1 -- 172.21.15.172:0/293516932 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc02cd00 con 0x7fc50415c450 2024-03-16T19:31:51.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.926+0000 7fc501d9b640 1 -- 172.21.15.172:0/293516932 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc02ce60 con 0x7fc50415c450 2024-03-16T19:31:51.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.926+0000 7fc501d9b640 1 -- 172.21.15.172:0/293516932 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc034660 con 0x7fc50415c450 2024-03-16T19:31:51.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.927+0000 7fc508a23640 1 -- 172.21.15.172:0/293516932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 msgr2=0x7fc5041640e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.927+0000 7fc508a23640 1 --2- 172.21.15.172:0/293516932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 0x7fc5041640e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009d90 tx=0x7fc4fc02bf50 comp rx=0 tx=0).stop 2024-03-16T19:31:51.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.927+0000 7fc508a23640 1 -- 172.21.15.172:0/293516932 shutdown_connections 2024-03-16T19:31:51.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.927+0000 7fc508a23640 1 --2- 172.21.15.172:0/293516932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 0x7fc5041640e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.927+0000 7fc508a23640 1 -- 172.21.15.172:0/293516932 >> 172.21.15.172:0/293516932 conn(0x7fc504071be0 msgr2=0x7fc504071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:51.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.927+0000 7fc508a23640 1 -- 172.21.15.172:0/293516932 shutdown_connections 2024-03-16T19:31:51.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.927+0000 7fc508a23640 1 -- 172.21.15.172:0/293516932 wait complete. 2024-03-16T19:31:51.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.928+0000 7fc508a23640 1 Processor -- start 2024-03-16T19:31:51.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.928+0000 7fc508a23640 1 -- start start 2024-03-16T19:31:51.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc508a23640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5040ca8e0 0x7fc50414b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc508a23640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc504152760 0x7fc504149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc508a23640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 0x7fc50414afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc508a23640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5041edc60 con 0x7fc50415c450 2024-03-16T19:31:51.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc508a23640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5041eec30 con 0x7fc5040ca8e0 2024-03-16T19:31:51.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc508a23640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5041582a0 con 0x7fc504152760 2024-03-16T19:31:51.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc502d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 0x7fc50414afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc502d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 0x7fc50414afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52616/0 (socket says 172.21.15.172:52616) 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc502d9d640 1 -- 172.21.15.172:0/419899742 learned_addr learned my addr 172.21.15.172:0/419899742 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc50259c640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5040ca8e0 0x7fc50414b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc4fbfff640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc504152760 0x7fc504149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc502d9d640 1 -- 172.21.15.172:0/419899742 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc504152760 msgr2=0x7fc504149410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc502d9d640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc504152760 0x7fc504149410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc502d9d640 1 -- 172.21.15.172:0/419899742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5040ca8e0 msgr2=0x7fc50414b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc502d9d640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5040ca8e0 0x7fc50414b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc502d9d640 1 -- 172.21.15.172:0/419899742 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc009a70 con 0x7fc50415c450 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc502d9d640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 0x7fc50414afb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009d90 tx=0x7fc4fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc4faffd640 1 -- 172.21.15.172:0/419899742 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc03d600 con 0x7fc50415c450 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc4faffd640 1 -- 172.21.15.172:0/419899742 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc03d780 con 0x7fc50415c450 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc504158560 con 0x7fc50415c450 2024-03-16T19:31:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.929+0000 7fc4faffd640 1 -- 172.21.15.172:0/419899742 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc03db30 con 0x7fc50415c450 2024-03-16T19:31:51.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.930+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5041ee150 con 0x7fc50415c450 2024-03-16T19:31:51.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.931+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc50406c390 con 0x7fc50415c450 2024-03-16T19:31:51.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.931+0000 7fc4faffd640 1 -- 172.21.15.172:0/419899742 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc4fc03b040 con 0x7fc50415c450 2024-03-16T19:31:51.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.931+0000 7fc4faffd640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc4e406b3b0 0x7fc4e406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:51.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.931+0000 7fc4faffd640 1 -- 172.21.15.172:0/419899742 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fc4fc0abc10 con 0x7fc50415c450 2024-03-16T19:31:51.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.931+0000 7fc50259c640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc4e406b3b0 0x7fc4e406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:51.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.932+0000 7fc50259c640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc4e406b3b0 0x7fc4e406d870 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4007f70 tx=0x7fc4f4007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:51.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:51.936+0000 7fc4faffd640 1 -- 172.21.15.172:0/419899742 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4fc0768f0 con 0x7fc50415c450 2024-03-16T19:31:52.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.119+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fc5040ca6c0 con 0x7fc50415c450 2024-03-16T19:31:52.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.801+0000 7fc4faffd640 1 -- 172.21.15.172:0/419899742 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v92) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc4fc04d0a0 con 0x7fc50415c450 2024-03-16T19:31:52.781 INFO:teuthology.orchestra.run.smithi172.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-16T19:31:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc4e406b3b0 msgr2=0x7fc4e406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc4e406b3b0 0x7fc4e406d870 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4007f70 tx=0x7fc4f4007f00 comp rx=0 tx=0).stop 2024-03-16T19:31:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 msgr2=0x7fc50414afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 0x7fc50414afb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009d90 tx=0x7fc4fc004a40 comp rx=0 tx=0).stop 2024-03-16T19:31:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 shutdown_connections 2024-03-16T19:31:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc50415c450 0x7fc50414afb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc4e406b3b0 0x7fc4e406d870 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc504152760 0x7fc504149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:52.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 --2- 172.21.15.172:0/419899742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5040ca8e0 0x7fc50414b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:52.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 >> 172.21.15.172:0/419899742 conn(0x7fc504071be0 msgr2=0x7fc50406f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:52.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 shutdown_connections 2024-03-16T19:31:52.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.805+0000 7fc508a23640 1 -- 172.21.15.172:0/419899742 wait complete. 2024-03-16T19:31:52.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-16T19:31:52.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.927+0000 7f605cb48640 1 Processor -- start 2024-03-16T19:31:52.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.927+0000 7f605cb48640 1 -- start start 2024-03-16T19:31:52.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f605cb48640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6058154ad0 0x7f605806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:52.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f605cb48640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605806b880 con 0x7f60580c7fa0 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f605cb48640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605806baf0 con 0x7f60580c7b60 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f605cb48640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605815a900 con 0x7f6058154ad0 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f6056d9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f60580c7fa0 0x7f605815ad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:58166/0 (socket says 172.21.15.172:58166) 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f6056d9d640 1 -- 172.21.15.172:0/3159335920 learned_addr learned my addr 172.21.15.172:0/3159335920 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f605659c640 1 --2- 172.21.15.172:0/3159335920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6058154ad0 0x7f605806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f6055d9b640 1 -- 172.21.15.172:0/3159335920 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183205816 0 0) 0x7f605806b880 con 0x7f60580c7fa0 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f6055d9b640 1 -- 172.21.15.172:0/3159335920 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6044003520 con 0x7f60580c7fa0 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f6055d9b640 1 -- 172.21.15.172:0/3159335920 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202723733 0 0) 0x7f605806baf0 con 0x7f60580c7b60 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f605659c640 1 -- 172.21.15.172:0/3159335920 >> v1:172.21.15.174:6789/0 conn(0x7f60580c7b60 legacy=0x7f605806dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f605659c640 1 -- 172.21.15.172:0/3159335920 >> v1:172.21.15.172:6789/0 conn(0x7f60580c7fa0 legacy=0x7f605815ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:52.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.928+0000 7f605659c640 1 -- 172.21.15.172:0/3159335920 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605806bdd0 con 0x7f6058154ad0 2024-03-16T19:31:52.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.929+0000 7f605659c640 1 --2- 172.21.15.172:0/3159335920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6058154ad0 0x7f605806afb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f60480097d0 tx=0x7f604802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39798d5550765978 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:52.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.929+0000 7f6055d9b640 1 -- 172.21.15.172:0/3159335920 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048030880 con 0x7f6058154ad0 2024-03-16T19:31:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.929+0000 7f6055d9b640 1 -- 172.21.15.172:0/3159335920 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60480309e0 con 0x7f6058154ad0 2024-03-16T19:31:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.929+0000 7f6055d9b640 1 -- 172.21.15.172:0/3159335920 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048030d70 con 0x7f6058154ad0 2024-03-16T19:31:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.930+0000 7f605cb48640 1 -- 172.21.15.172:0/3159335920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6058154ad0 msgr2=0x7f605806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.930+0000 7f605cb48640 1 --2- 172.21.15.172:0/3159335920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6058154ad0 0x7f605806afb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f60480097d0 tx=0x7f604802b9c0 comp rx=0 tx=0).stop 2024-03-16T19:31:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.930+0000 7f605cb48640 1 -- 172.21.15.172:0/3159335920 shutdown_connections 2024-03-16T19:31:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.930+0000 7f605cb48640 1 --2- 172.21.15.172:0/3159335920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6058154ad0 0x7f605806afb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.930+0000 7f605cb48640 1 -- 172.21.15.172:0/3159335920 >> 172.21.15.172:0/3159335920 conn(0x7f6058071be0 msgr2=0x7f6058071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.930+0000 7f605cb48640 1 -- 172.21.15.172:0/3159335920 shutdown_connections 2024-03-16T19:31:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.930+0000 7f605cb48640 1 -- 172.21.15.172:0/3159335920 wait complete. 2024-03-16T19:31:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.931+0000 7f605cb48640 1 Processor -- start 2024-03-16T19:31:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.931+0000 7f605cb48640 1 -- start start 2024-03-16T19:31:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f605cb48640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60580c7b60 0x7f605814cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f605cb48640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60580c7fa0 0x7f605814b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f605cb48640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6058154ad0 0x7f6058151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f605cb48640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605814f3f0 con 0x7f6058154ad0 2024-03-16T19:31:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f605cb48640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605814ee10 con 0x7f60580c7b60 2024-03-16T19:31:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f605cb48640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60581505f0 con 0x7f60580c7fa0 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f6056d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6058154ad0 0x7f6058151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f6056d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6058154ad0 0x7f6058151750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52622/0 (socket says 172.21.15.172:52622) 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f6056d9d640 1 -- 172.21.15.172:0/1522778815 learned_addr learned my addr 172.21.15.172:0/1522778815 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f605659c640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60580c7b60 0x7f605814cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f604ffff640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60580c7fa0 0x7f605814b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f6056d9d640 1 -- 172.21.15.172:0/1522778815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60580c7fa0 msgr2=0x7f605814b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f6056d9d640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60580c7fa0 0x7f605814b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f6056d9d640 1 -- 172.21.15.172:0/1522778815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60580c7b60 msgr2=0x7f605814cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f6056d9d640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60580c7b60 0x7f605814cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f6056d9d640 1 -- 172.21.15.172:0/1522778815 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6048009400 con 0x7f6058154ad0 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f6056d9d640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6058154ad0 0x7f6058151750 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f605000caa0 tx=0x7f605000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.932+0000 7f604effd640 1 -- 172.21.15.172:0/1522778815 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605000ac00 con 0x7f6058154ad0 2024-03-16T19:31:52.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.933+0000 7f604effd640 1 -- 172.21.15.172:0/1522778815 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605000ad60 con 0x7f6058154ad0 2024-03-16T19:31:52.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.933+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60581508d0 con 0x7f6058154ad0 2024-03-16T19:31:52.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.933+0000 7f604effd640 1 -- 172.21.15.172:0/1522778815 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050012620 con 0x7f6058154ad0 2024-03-16T19:31:52.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.933+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60580cd3a0 con 0x7f6058154ad0 2024-03-16T19:31:52.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.934+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605814f5a0 con 0x7f6058154ad0 2024-03-16T19:31:52.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.934+0000 7f604effd640 1 -- 172.21.15.172:0/1522778815 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6050007070 con 0x7f6058154ad0 2024-03-16T19:31:52.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.934+0000 7f604effd640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f603806b3e0 0x7f603806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:52.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.935+0000 7f604effd640 1 -- 172.21.15.172:0/1522778815 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f6050089ad0 con 0x7f6058154ad0 2024-03-16T19:31:52.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.935+0000 7f605659c640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f603806b3e0 0x7f603806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:52.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.935+0000 7f605659c640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f603806b3e0 0x7f603806d8a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6048036000 tx=0x7f6048008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:52.939+0000 7f604effd640 1 -- 172.21.15.172:0/1522778815 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f605814f5a0 con 0x7f6058154ad0 2024-03-16T19:31:53.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.122+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f60580681c0 con 0x7f6058154ad0 2024-03-16T19:31:53.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.980+0000 7f604effd640 1 -- 172.21.15.172:0/1522778815 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v93) v1 ==== 113+0+0 (secure 0 0 0) 0x7f6050017020 con 0x7f6058154ad0 2024-03-16T19:31:53.960 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.3 to 1 (10000) 2024-03-16T19:31:53.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f603806b3e0 msgr2=0x7f603806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:53.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f603806b3e0 0x7f603806d8a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6048036000 tx=0x7f6048008030 comp rx=0 tx=0).stop 2024-03-16T19:31:53.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6058154ad0 msgr2=0x7f6058151750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:53.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6058154ad0 0x7f6058151750 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f605000caa0 tx=0x7f605000cf60 comp rx=0 tx=0).stop 2024-03-16T19:31:53.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 shutdown_connections 2024-03-16T19:31:53.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6058154ad0 0x7f6058151750 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:53.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f603806b3e0 0x7f603806d8a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:53.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60580c7fa0 0x7f605814b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:53.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 --2- 172.21.15.172:0/1522778815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60580c7b60 0x7f605814cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:53.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 >> 172.21.15.172:0/1522778815 conn(0x7f6058071be0 msgr2=0x7f605806dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:53.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 shutdown_connections 2024-03-16T19:31:53.964 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:53.983+0000 7f605cb48640 1 -- 172.21.15.172:0/1522778815 wait complete. 2024-03-16T19:31:53.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-16T19:31:54.080 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.101+0000 7f8eafe6b640 1 Processor -- start 2024-03-16T19:31:54.081 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.101+0000 7f8eafe6b640 1 -- start start 2024-03-16T19:31:54.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eafe6b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8ea80ca8e0 0x7f8ea815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:54.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eafe6b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea80c7b60 0x7f8ea815c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:54.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eafe6b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea806b2e0 con 0x7f8ea8152760 2024-03-16T19:31:54.082 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eafe6b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea806bfe0 con 0x7f8ea80ca8e0 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eafe6b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea806cb70 con 0x7f8ea80c7b60 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eadc07640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8ea8152760 0x7f8ea806dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:58174/0 (socket says 172.21.15.172:58174) 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eadc07640 1 -- 172.21.15.172:0/1110020382 learned_addr learned my addr 172.21.15.172:0/1110020382 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eae408640 1 --2- 172.21.15.172:0/1110020382 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea80c7b60 0x7f8ea815c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8ead406640 1 --2- 172.21.15.172:0/1110020382 >> v2:172.21.15.174:3300/0 conn(0x7f8ea80ca8e0 0x7f8ea815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eacc05640 1 -- 172.21.15.172:0/1110020382 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758173920 0 0) 0x7f8ea806b2e0 con 0x7f8ea8152760 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eacc05640 1 -- 172.21.15.172:0/1110020382 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e94003520 con 0x7f8ea8152760 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8ead406640 1 -- 172.21.15.172:0/1110020382 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea80c7b60 msgr2=0x7f8ea815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8ead406640 1 --2- 172.21.15.172:0/1110020382 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea80c7b60 0x7f8ea815c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8ead406640 1 -- 172.21.15.172:0/1110020382 >> v1:172.21.15.172:6789/0 conn(0x7f8ea8152760 legacy=0x7f8ea806dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8ead406640 1 -- 172.21.15.172:0/1110020382 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea8159580 con 0x7f8ea80ca8e0 2024-03-16T19:31:54.083 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.102+0000 7f8eae408640 1 --2- 172.21.15.172:0/1110020382 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea80c7b60 0x7f8ea815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.103+0000 7f8ead406640 1 --2- 172.21.15.172:0/1110020382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 0x7f8ea815bd40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8e980097a0 tx=0x7f8e9802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13f0ca72fb33b459 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.103+0000 7f8eacc05640 1 -- 172.21.15.172:0/1110020382 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98031830 con 0x7f8ea80ca8e0 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.103+0000 7f8eacc05640 1 -- 172.21.15.172:0/1110020382 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98031990 con 0x7f8ea80ca8e0 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.103+0000 7f8eacc05640 1 -- 172.21.15.172:0/1110020382 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98031d20 con 0x7f8ea80ca8e0 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.104+0000 7f8eafe6b640 1 -- 172.21.15.172:0/1110020382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 msgr2=0x7f8ea815bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.104+0000 7f8eafe6b640 1 --2- 172.21.15.172:0/1110020382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 0x7f8ea815bd40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8e980097a0 tx=0x7f8e9802b8d0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.104+0000 7f8eafe6b640 1 -- 172.21.15.172:0/1110020382 shutdown_connections 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.104+0000 7f8eafe6b640 1 --2- 172.21.15.172:0/1110020382 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea80c7b60 0x7f8ea815c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.104+0000 7f8eafe6b640 1 --2- 172.21.15.172:0/1110020382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 0x7f8ea815bd40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.104+0000 7f8eafe6b640 1 -- 172.21.15.172:0/1110020382 >> 172.21.15.172:0/1110020382 conn(0x7f8ea8071be0 msgr2=0x7f8ea8071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.104+0000 7f8eafe6b640 1 -- 172.21.15.172:0/1110020382 shutdown_connections 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.104+0000 7f8eafe6b640 1 -- 172.21.15.172:0/1110020382 wait complete. 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.105+0000 7f8eafe6b640 1 Processor -- start 2024-03-16T19:31:54.085 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.105+0000 7f8eafe6b640 1 -- start start 2024-03-16T19:31:54.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eafe6b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80c7b60 0x7f8ea814e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eafe6b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 0x7f8ea8157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eafe6b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea8152760 0x7f8ea8157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eafe6b640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea814ef30 con 0x7f8ea80c7b60 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eafe6b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea81570d0 con 0x7f8ea80ca8e0 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eafe6b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea814f630 con 0x7f8ea8152760 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eadc07640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80c7b60 0x7f8ea814e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eadc07640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80c7b60 0x7f8ea814e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52624/0 (socket says 172.21.15.172:52624) 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eadc07640 1 -- 172.21.15.172:0/3401004135 learned_addr learned my addr 172.21.15.172:0/3401004135 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8ead406640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 0x7f8ea8157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eae408640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea8152760 0x7f8ea8157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eadc07640 1 -- 172.21.15.172:0/3401004135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea8152760 msgr2=0x7f8ea8157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eadc07640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea8152760 0x7f8ea8157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eadc07640 1 -- 172.21.15.172:0/3401004135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 msgr2=0x7f8ea8157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eadc07640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 0x7f8ea8157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eadc07640 1 -- 172.21.15.172:0/3401004135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea0009a70 con 0x7f8ea80c7b60 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8ead406640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 0x7f8ea8157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eadc07640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80c7b60 0x7f8ea814e9f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400c3c0 tx=0x7f8ea400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8e9e7fc640 1 -- 172.21.15.172:0/3401004135 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4005ce0 con 0x7f8ea80c7b60 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8e9e7fc640 1 -- 172.21.15.172:0/3401004135 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea4005e40 con 0x7f8ea80c7b60 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e98009400 con 0x7f8ea80c7b60 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8e9e7fc640 1 -- 172.21.15.172:0/3401004135 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea40072f0 con 0x7f8ea80c7b60 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.106+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea8156d60 con 0x7f8ea80c7b60 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.107+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea814fb50 con 0x7f8ea80c7b60 2024-03-16T19:31:54.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.108+0000 7f8e9e7fc640 1 -- 172.21.15.172:0/3401004135 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8ea4007450 con 0x7f8ea80c7b60 2024-03-16T19:31:54.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.108+0000 7f8e9e7fc640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8e8c06b3b0 0x7f8e8c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:54.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.108+0000 7f8ead406640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8e8c06b3b0 0x7f8e8c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:54.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.108+0000 7f8e9e7fc640 1 -- 172.21.15.172:0/3401004135 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f8ea40892b0 con 0x7f8ea80c7b60 2024-03-16T19:31:54.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.109+0000 7f8ead406640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8e8c06b3b0 0x7f8e8c06d870 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8e98036000 tx=0x7f8e9803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:54.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.113+0000 7f8e9e7fc640 1 -- 172.21.15.172:0/3401004135 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ea4053e50 con 0x7f8ea80c7b60 2024-03-16T19:31:54.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.297+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f8ea80cb8c0 con 0x7f8ea80c7b60 2024-03-16T19:31:54.965 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.985+0000 7f8e9e7fc640 1 -- 172.21.15.172:0/3401004135 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v94) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8ea40170a0 con 0x7f8ea80c7b60 2024-03-16T19:31:54.965 INFO:teuthology.orchestra.run.smithi172.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-16T19:31:54.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8e8c06b3b0 msgr2=0x7f8e8c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:54.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8e8c06b3b0 0x7f8e8c06d870 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8e98036000 tx=0x7f8e9803c040 comp rx=0 tx=0).stop 2024-03-16T19:31:54.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80c7b60 msgr2=0x7f8ea814e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:54.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80c7b60 0x7f8ea814e9f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400c3c0 tx=0x7f8ea400c880 comp rx=0 tx=0).stop 2024-03-16T19:31:54.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 shutdown_connections 2024-03-16T19:31:54.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8e8c06b3b0 0x7f8e8c06d870 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ea8152760 0x7f8ea8157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ea80ca8e0 0x7f8ea8157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 --2- 172.21.15.172:0/3401004135 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ea80c7b60 0x7f8ea814e9f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:54.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 >> 172.21.15.172:0/3401004135 conn(0x7f8ea8071be0 msgr2=0x7f8ea80cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:54.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 shutdown_connections 2024-03-16T19:31:54.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:54.988+0000 7f8eafe6b640 1 -- 172.21.15.172:0/3401004135 wait complete. 2024-03-16T19:31:54.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-16T19:31:55.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.106+0000 7f9cac283640 1 Processor -- start 2024-03-16T19:31:55.086 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.106+0000 7f9cac283640 1 -- start start 2024-03-16T19:31:55.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9cac283640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9ca40ccc80 0x7f9ca406e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:55.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9cac283640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca406eb40 0x7f9ca4071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:55.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9cac283640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40c9950 con 0x7f9ca41517e0 2024-03-16T19:31:55.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9cac283640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca4157820 con 0x7f9ca40ccc80 2024-03-16T19:31:55.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9cac283640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca41585a0 con 0x7f9ca406eb40 2024-03-16T19:31:55.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9ca981e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9ca40ccc80 0x7f9ca406e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:55.087 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9caa01f640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9ca41517e0 0x7f9ca40c8ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:58188/0 (socket says 172.21.15.172:58188) 2024-03-16T19:31:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9caa01f640 1 -- 172.21.15.172:0/104863439 learned_addr learned my addr 172.21.15.172:0/104863439 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9caa820640 1 --2- 172.21.15.172:0/104863439 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca406eb40 0x7f9ca4071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9ca981e640 1 -- 172.21.15.172:0/104863439 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca406eb40 msgr2=0x7f9ca4071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9ca981e640 1 --2- 172.21.15.172:0/104863439 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca406eb40 0x7f9ca4071c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9ca981e640 1 -- 172.21.15.172:0/104863439 >> v1:172.21.15.172:6789/0 conn(0x7f9ca41517e0 legacy=0x7f9ca40c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9ca981e640 1 -- 172.21.15.172:0/104863439 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40c9670 con 0x7f9ca40ccc80 2024-03-16T19:31:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.107+0000 7f9ca901d640 1 -- 172.21.15.172:0/104863439 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397352590 0 0) 0x7f9ca40c9950 con 0x7f9ca41517e0 2024-03-16T19:31:55.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.108+0000 7f9ca981e640 1 --2- 172.21.15.172:0/104863439 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca40ccc80 0x7f9ca406e600 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9c940097a0 tx=0x7f9c9402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1211b705c9c74dec server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:55.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.108+0000 7f9ca901d640 1 -- 172.21.15.172:0/104863439 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94030610 con 0x7f9ca40ccc80 2024-03-16T19:31:55.089 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.108+0000 7f9ca901d640 1 -- 172.21.15.172:0/104863439 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c94030770 con 0x7f9ca40ccc80 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.108+0000 7f9ca901d640 1 -- 172.21.15.172:0/104863439 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94030b00 con 0x7f9ca40ccc80 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.109+0000 7f9cac283640 1 -- 172.21.15.172:0/104863439 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca40ccc80 msgr2=0x7f9ca406e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.109+0000 7f9cac283640 1 --2- 172.21.15.172:0/104863439 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca40ccc80 0x7f9ca406e600 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9c940097a0 tx=0x7f9c9402b960 comp rx=0 tx=0).stop 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.109+0000 7f9cac283640 1 -- 172.21.15.172:0/104863439 shutdown_connections 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.109+0000 7f9cac283640 1 --2- 172.21.15.172:0/104863439 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca406eb40 0x7f9ca4071c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.109+0000 7f9cac283640 1 --2- 172.21.15.172:0/104863439 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca40ccc80 0x7f9ca406e600 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.109+0000 7f9cac283640 1 -- 172.21.15.172:0/104863439 >> 172.21.15.172:0/104863439 conn(0x7f9ca4066c20 msgr2=0x7f9ca4067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.109+0000 7f9cac283640 1 -- 172.21.15.172:0/104863439 shutdown_connections 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.109+0000 7f9cac283640 1 -- 172.21.15.172:0/104863439 wait complete. 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.110+0000 7f9cac283640 1 Processor -- start 2024-03-16T19:31:55.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.110+0000 7f9cac283640 1 -- start start 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.110+0000 7f9cac283640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca406eb40 0x7f9ca4146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.110+0000 7f9cac283640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca40ccc80 0x7f9ca41f14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.110+0000 7f9cac283640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ca41517e0 0x7f9ca41428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.110+0000 7f9cac283640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca4151ff0 con 0x7f9ca406eb40 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.110+0000 7f9cac283640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca41542b0 con 0x7f9ca41517e0 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.110+0000 7f9cac283640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca4152240 con 0x7f9ca40ccc80 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa01f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca406eb40 0x7f9ca4146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa01f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca406eb40 0x7f9ca4146be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52648/0 (socket says 172.21.15.172:52648) 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa01f640 1 -- 172.21.15.172:0/81031739 learned_addr learned my addr 172.21.15.172:0/81031739 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9ca981e640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca40ccc80 0x7f9ca41f14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa820640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ca41517e0 0x7f9ca41428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:55.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa01f640 1 -- 172.21.15.172:0/81031739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca40ccc80 msgr2=0x7f9ca41f14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa01f640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca40ccc80 0x7f9ca41f14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa01f640 1 -- 172.21.15.172:0/81031739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ca41517e0 msgr2=0x7f9ca41428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa01f640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ca41517e0 0x7f9ca41428f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa01f640 1 -- 172.21.15.172:0/81031739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c94009400 con 0x7f9ca406eb40 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9caa01f640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca406eb40 0x7f9ca4146be0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000c460 tx=0x7f9ca000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9c9affd640 1 -- 172.21.15.172:0/81031739 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca00074e0 con 0x7f9ca406eb40 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9c9affd640 1 -- 172.21.15.172:0/81031739 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca0007660 con 0x7f9ca406eb40 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca4152520 con 0x7f9ca406eb40 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9c9affd640 1 -- 172.21.15.172:0/81031739 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0004a20 con 0x7f9ca406eb40 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.111+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca41547e0 con 0x7f9ca406eb40 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.112+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca40681c0 con 0x7f9ca406eb40 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.113+0000 7f9c9affd640 1 -- 172.21.15.172:0/81031739 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9ca0004b80 con 0x7f9ca406eb40 2024-03-16T19:31:55.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.113+0000 7f9c9affd640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9c8806b3b0 0x7f9c8806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:55.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.113+0000 7f9c9affd640 1 -- 172.21.15.172:0/81031739 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f9ca0089140 con 0x7f9ca406eb40 2024-03-16T19:31:55.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.113+0000 7f9ca981e640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9c8806b3b0 0x7f9c8806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:55.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.114+0000 7f9ca981e640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9c8806b3b0 0x7f9c8806d870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9c94036000 tx=0x7f9c94008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:55.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.117+0000 7f9c9affd640 1 -- 172.21.15.172:0/81031739 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ca0053d60 con 0x7f9ca406eb40 2024-03-16T19:31:55.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.302+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f9ca40670a0 con 0x7f9ca406eb40 2024-03-16T19:31:55.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.990+0000 7f9c9affd640 1 -- 172.21.15.172:0/81031739 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v95) v1 ==== 113+0+0 (secure 0 0 0) 0x7f9ca0018090 con 0x7f9ca406eb40 2024-03-16T19:31:55.970 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.4 to 1 (10000) 2024-03-16T19:31:55.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9c8806b3b0 msgr2=0x7f9c8806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:55.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9c8806b3b0 0x7f9c8806d870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9c94036000 tx=0x7f9c94008030 comp rx=0 tx=0).stop 2024-03-16T19:31:55.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca406eb40 msgr2=0x7f9ca4146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:55.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca406eb40 0x7f9ca4146be0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000c460 tx=0x7f9ca000c920 comp rx=0 tx=0).stop 2024-03-16T19:31:55.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 shutdown_connections 2024-03-16T19:31:55.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ca41517e0 0x7f9ca41428f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:55.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9c8806b3b0 0x7f9c8806d870 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:55.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ca40ccc80 0x7f9ca41f14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:55.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 --2- 172.21.15.172:0/81031739 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ca406eb40 0x7f9ca4146be0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:55.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 >> 172.21.15.172:0/81031739 conn(0x7f9ca4066c20 msgr2=0x7f9ca40c5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:55.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 shutdown_connections 2024-03-16T19:31:55.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:55.993+0000 7f9cac283640 1 -- 172.21.15.172:0/81031739 wait complete. 2024-03-16T19:31:55.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-16T19:31:56.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.110+0000 7f9dcf876640 1 Processor -- start 2024-03-16T19:31:56.090 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.110+0000 7f9dcf876640 1 -- start start 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcf876640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9dc8154e00 0x7f9dc806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcf876640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc80b5f10 0x7f9dc80caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcf876640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9dc806b500 0x7f9dc80cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcf876640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc815b570 con 0x7f9dc806b500 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcf876640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc81551e0 con 0x7f9dc8154e00 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcf876640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc80cc370 con 0x7f9dc80b5f10 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcde13640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9dc806b500 0x7f9dc80cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcde13640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9dc806b500 0x7f9dc80cb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52664/0 (socket says 172.21.15.172:52664) 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcde13640 1 -- 172.21.15.172:0/1712498902 learned_addr learned my addr 172.21.15.172:0/1712498902 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcce11640 1 --2- 172.21.15.172:0/1712498902 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc80b5f10 0x7f9dc80caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.111+0000 7f9dcd612640 1 --2- 172.21.15.172:0/1712498902 >> v2:172.21.15.174:3300/0 conn(0x7f9dc8154e00 0x7f9dc806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.112+0000 7f9dcde13640 1 -- 172.21.15.172:0/1712498902 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc80b5f10 msgr2=0x7f9dc80caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:56.092 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.112+0000 7f9dcde13640 1 --2- 172.21.15.172:0/1712498902 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc80b5f10 0x7f9dc80caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.112+0000 7f9dcde13640 1 -- 172.21.15.172:0/1712498902 >> v2:172.21.15.174:3300/0 conn(0x7f9dc8154e00 msgr2=0x7f9dc806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:56.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.112+0000 7f9dcde13640 1 --2- 172.21.15.172:0/1712498902 >> v2:172.21.15.174:3300/0 conn(0x7f9dc8154e00 0x7f9dc806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.112+0000 7f9dcde13640 1 -- 172.21.15.172:0/1712498902 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc815b990 con 0x7f9dc806b500 2024-03-16T19:31:56.093 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.112+0000 7f9dcd612640 1 --2- 172.21.15.172:0/1712498902 >> v2:172.21.15.174:3300/0 conn(0x7f9dc8154e00 0x7f9dc806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:56.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.112+0000 7f9dcde13640 1 --2- 172.21.15.172:0/1712498902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc806b500 0x7f9dc80cb410 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80cd910 tx=0x7f9dc002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=349a0128831cc876 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:56.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dbffff640 1 -- 172.21.15.172:0/1712498902 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0030840 con 0x7f9dc806b500 2024-03-16T19:31:56.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dbffff640 1 -- 172.21.15.172:0/1712498902 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc00309a0 con 0x7f9dc806b500 2024-03-16T19:31:56.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dbffff640 1 -- 172.21.15.172:0/1712498902 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0030d30 con 0x7f9dc806b500 2024-03-16T19:31:56.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dcf876640 1 -- 172.21.15.172:0/1712498902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc806b500 msgr2=0x7f9dc80cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:56.094 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1712498902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc806b500 0x7f9dc80cb410 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80cd910 tx=0x7f9dc002b980 comp rx=0 tx=0).stop 2024-03-16T19:31:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dcf876640 1 -- 172.21.15.172:0/1712498902 shutdown_connections 2024-03-16T19:31:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1712498902 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc806b500 0x7f9dc80cb410 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1712498902 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc80b5f10 0x7f9dc80caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1712498902 >> v2:172.21.15.174:3300/0 conn(0x7f9dc8154e00 0x7f9dc806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dcf876640 1 -- 172.21.15.172:0/1712498902 >> 172.21.15.172:0/1712498902 conn(0x7f9dc8066c20 msgr2=0x7f9dc8067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dcf876640 1 -- 172.21.15.172:0/1712498902 shutdown_connections 2024-03-16T19:31:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.113+0000 7f9dcf876640 1 -- 172.21.15.172:0/1712498902 wait complete. 2024-03-16T19:31:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.114+0000 7f9dcf876640 1 Processor -- start 2024-03-16T19:31:56.095 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.114+0000 7f9dcf876640 1 -- start start 2024-03-16T19:31:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcf876640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc806b500 0x7f9dc8147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcf876640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dc80b5f10 0x7f9dc814c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcf876640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc8154e00 0x7f9dc8148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcf876640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc81579d0 con 0x7f9dc8154e00 2024-03-16T19:31:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcf876640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc8157f90 con 0x7f9dc80b5f10 2024-03-16T19:31:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcf876640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc8147d40 con 0x7f9dc806b500 2024-03-16T19:31:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcde13640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc8154e00 0x7f9dc8148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcd612640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc806b500 0x7f9dc8147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:56.097 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcde13640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc8154e00 0x7f9dc8148250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52676/0 (socket says 172.21.15.172:52676) 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcd612640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc806b500 0x7f9dc8147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:50720/0 (socket says 172.21.15.172:50720) 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcde13640 1 -- 172.21.15.172:0/1636899530 learned_addr learned my addr 172.21.15.172:0/1636899530 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcce11640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dc80b5f10 0x7f9dc814c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcde13640 1 -- 172.21.15.172:0/1636899530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc806b500 msgr2=0x7f9dc8147800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcde13640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc806b500 0x7f9dc8147800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcde13640 1 -- 172.21.15.172:0/1636899530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dc80b5f10 msgr2=0x7f9dc814c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcde13640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dc80b5f10 0x7f9dc814c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.115+0000 7f9dcde13640 1 -- 172.21.15.172:0/1636899530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc4009a70 con 0x7f9dc8154e00 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.116+0000 7f9dcde13640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc8154e00 0x7f9dc8148250 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9dc002b9c0 tx=0x7f9dc0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.116+0000 7f9dbdffb640 1 -- 172.21.15.172:0/1636899530 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0039c90 con 0x7f9dc8154e00 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.116+0000 7f9dbdffb640 1 -- 172.21.15.172:0/1636899530 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc0039e10 con 0x7f9dc8154e00 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.116+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc00094d0 con 0x7f9dc8154e00 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.116+0000 7f9dbdffb640 1 -- 172.21.15.172:0/1636899530 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc00446c0 con 0x7f9dc8154e00 2024-03-16T19:31:56.098 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.116+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc81466f0 con 0x7f9dc8154e00 2024-03-16T19:31:56.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.117+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc8067c30 con 0x7f9dc8154e00 2024-03-16T19:31:56.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.117+0000 7f9dbdffb640 1 -- 172.21.15.172:0/1636899530 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9dc002f390 con 0x7f9dc8154e00 2024-03-16T19:31:56.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.118+0000 7f9dbdffb640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9dac06b3b0 0x7f9dac06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:56.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.118+0000 7f9dbdffb640 1 -- 172.21.15.172:0/1636899530 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f9dc003e070 con 0x7f9dc8154e00 2024-03-16T19:31:56.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.118+0000 7f9dcd612640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9dac06b3b0 0x7f9dac06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:56.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.118+0000 7f9dcd612640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9dac06b3b0 0x7f9dac06d870 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40114e0 tx=0x7f9dc400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:56.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.122+0000 7f9dbdffb640 1 -- 172.21.15.172:0/1636899530 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dc00774b0 con 0x7f9dc8154e00 2024-03-16T19:31:56.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.303+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f9dc80cd940 con 0x7f9dc8154e00 2024-03-16T19:31:56.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:31:56.322 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:31:56.980 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:56.999+0000 7f9dbdffb640 1 -- 172.21.15.172:0/1636899530 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v96) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9dc007ab70 con 0x7f9dc8154e00 2024-03-16T19:31:56.980 INFO:teuthology.orchestra.run.smithi172.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-16T19:31:56.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9dac06b3b0 msgr2=0x7f9dac06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:56.983 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9dac06b3b0 0x7f9dac06d870 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40114e0 tx=0x7f9dc400b040 comp rx=0 tx=0).stop 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc8154e00 msgr2=0x7f9dc8148250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc8154e00 0x7f9dc8148250 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9dc002b9c0 tx=0x7f9dc0004730 comp rx=0 tx=0).stop 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 shutdown_connections 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dc8154e00 0x7f9dc8148250 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dc80b5f10 0x7f9dc814c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9dac06b3b0 0x7f9dac06d870 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 --2- 172.21.15.172:0/1636899530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dc806b500 0x7f9dc8147800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 >> 172.21.15.172:0/1636899530 conn(0x7f9dc8066c20 msgr2=0x7f9dc8153580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 shutdown_connections 2024-03-16T19:31:56.984 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.002+0000 7f9dcf876640 1 -- 172.21.15.172:0/1636899530 wait complete. 2024-03-16T19:31:56.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-16T19:31:57.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.124+0000 7fc482d9a640 1 Processor -- start 2024-03-16T19:31:57.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.124+0000 7fc482d9a640 1 -- start start 2024-03-16T19:31:57.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc482d9a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c0ca590 0x7fc47c15be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:57.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc482d9a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc47c066fb0 0x7fc47c15c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:57.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc482d9a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47c159220 con 0x7fc47c152aa0 2024-03-16T19:31:57.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc482d9a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47c0c9da0 con 0x7fc47c066fb0 2024-03-16T19:31:57.106 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc482d9a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c159730 con 0x7fc47c0ca590 2024-03-16T19:31:57.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc481337640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc47c066fb0 0x7fc47c15c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:57.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc480b36640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc47c152aa0 0x7fc47c0c5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:35496/0 (socket says 172.21.15.172:35496) 2024-03-16T19:31:57.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc481337640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc47c066fb0 0x7fc47c15c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:53990/0 (socket says 172.21.15.172:53990) 2024-03-16T19:31:57.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc480b36640 1 -- 172.21.15.172:0/706348923 learned_addr learned my addr 172.21.15.172:0/706348923 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.125+0000 7fc47bfff640 1 --2- 172.21.15.172:0/706348923 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c0ca590 0x7fc47c15be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.126+0000 7fc47b7fe640 1 -- 172.21.15.172:0/706348923 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19133110 0 0) 0x7fc47c159220 con 0x7fc47c152aa0 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.126+0000 7fc47b7fe640 1 -- 172.21.15.172:0/706348923 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc464003520 con 0x7fc47c152aa0 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.126+0000 7fc47bfff640 1 -- 172.21.15.172:0/706348923 >> v2:172.21.15.174:3300/0 conn(0x7fc47c066fb0 msgr2=0x7fc47c15c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.126+0000 7fc47bfff640 1 --2- 172.21.15.172:0/706348923 >> v2:172.21.15.174:3300/0 conn(0x7fc47c066fb0 0x7fc47c15c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.126+0000 7fc47bfff640 1 -- 172.21.15.172:0/706348923 >> v1:172.21.15.174:6789/0 conn(0x7fc47c152aa0 legacy=0x7fc47c0c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.126+0000 7fc47bfff640 1 -- 172.21.15.172:0/706348923 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c0c8e00 con 0x7fc47c0ca590 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.126+0000 7fc481337640 1 --2- 172.21.15.172:0/706348923 >> v2:172.21.15.174:3300/0 conn(0x7fc47c066fb0 0x7fc47c15c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.126+0000 7fc47bfff640 1 --2- 172.21.15.172:0/706348923 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c0ca590 0x7fc47c15be90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc4700097a0 tx=0x7fc47002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a9e5018b4f5e7c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc47b7fe640 1 -- 172.21.15.172:0/706348923 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470031830 con 0x7fc47c0ca590 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc47b7fe640 1 -- 172.21.15.172:0/706348923 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc470031990 con 0x7fc47c0ca590 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc47b7fe640 1 -- 172.21.15.172:0/706348923 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470031d20 con 0x7fc47c0ca590 2024-03-16T19:31:57.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc482d9a640 1 -- 172.21.15.172:0/706348923 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c0ca590 msgr2=0x7fc47c15be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:57.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc482d9a640 1 --2- 172.21.15.172:0/706348923 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c0ca590 0x7fc47c15be90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc4700097a0 tx=0x7fc47002b8d0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc482d9a640 1 -- 172.21.15.172:0/706348923 shutdown_connections 2024-03-16T19:31:57.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc482d9a640 1 --2- 172.21.15.172:0/706348923 >> v2:172.21.15.174:3300/0 conn(0x7fc47c066fb0 0x7fc47c15c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc482d9a640 1 --2- 172.21.15.172:0/706348923 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c0ca590 0x7fc47c15be90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc482d9a640 1 -- 172.21.15.172:0/706348923 >> 172.21.15.172:0/706348923 conn(0x7fc47c06efd0 msgr2=0x7fc47c06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:57.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc482d9a640 1 -- 172.21.15.172:0/706348923 shutdown_connections 2024-03-16T19:31:57.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.127+0000 7fc482d9a640 1 -- 172.21.15.172:0/706348923 wait complete. 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.128+0000 7fc482d9a640 1 Processor -- start 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.128+0000 7fc482d9a640 1 -- start start 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc482d9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c066fb0 0x7fc47c1f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc482d9a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0ca590 0x7fc47c0726a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc482d9a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c152aa0 0x7fc47c149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc482d9a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c148e00 con 0x7fc47c0ca590 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc482d9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c1496a0 con 0x7fc47c066fb0 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc482d9a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c1f2100 con 0x7fc47c152aa0 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc47bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0ca590 0x7fc47c0726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc47bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0ca590 0x7fc47c0726a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52682/0 (socket says 172.21.15.172:52682) 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc47bfff640 1 -- 172.21.15.172:0/1551901672 learned_addr learned my addr 172.21.15.172:0/1551901672 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc481337640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c152aa0 0x7fc47c149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.129+0000 7fc480b36640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c066fb0 0x7fc47c1f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:57.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc47bfff640 1 -- 172.21.15.172:0/1551901672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c152aa0 msgr2=0x7fc47c149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc47bfff640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c152aa0 0x7fc47c149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc47bfff640 1 -- 172.21.15.172:0/1551901672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c066fb0 msgr2=0x7fc47c1f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc47bfff640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c066fb0 0x7fc47c1f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc47bfff640 1 -- 172.21.15.172:0/1551901672 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc474009a70 con 0x7fc47c0ca590 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc481337640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c152aa0 0x7fc47c149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc47bfff640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0ca590 0x7fc47c0726a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc470009890 tx=0x7fc4700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc4797fa640 1 -- 172.21.15.172:0/1551901672 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47003a830 con 0x7fc47c0ca590 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc4797fa640 1 -- 172.21.15.172:0/1551901672 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47003a9b0 con 0x7fc47c0ca590 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc470009400 con 0x7fc47c0ca590 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc4797fa640 1 -- 172.21.15.172:0/1551901672 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47003ad60 con 0x7fc47c0ca590 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.130+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47c14e6a0 con 0x7fc47c0ca590 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.131+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47c0681c0 con 0x7fc47c0ca590 2024-03-16T19:31:57.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.131+0000 7fc4797fa640 1 -- 172.21.15.172:0/1551901672 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc4700433e0 con 0x7fc47c0ca590 2024-03-16T19:31:57.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.132+0000 7fc4797fa640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc46806b3b0 0x7fc46806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:57.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.132+0000 7fc4797fa640 1 -- 172.21.15.172:0/1551901672 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fc4700ab550 con 0x7fc47c0ca590 2024-03-16T19:31:57.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.132+0000 7fc480b36640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc46806b3b0 0x7fc46806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:57.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.132+0000 7fc480b36640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc46806b3b0 0x7fc46806d870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc46c005880 tx=0x7fc46c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:57.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.136+0000 7fc4797fa640 1 -- 172.21.15.172:0/1551901672 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4700760f0 con 0x7fc47c0ca590 2024-03-16T19:31:57.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:57.317+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7fc47c0c98e0 con 0x7fc47c0ca590 2024-03-16T19:31:57.988 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.007+0000 7fc4797fa640 1 -- 172.21.15.172:0/1551901672 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v97) v1 ==== 113+0+0 (secure 0 0 0) 0x7fc4700797b0 con 0x7fc47c0ca590 2024-03-16T19:31:57.988 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.5 to 1 (10000) 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc46806b3b0 msgr2=0x7fc46806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc46806b3b0 0x7fc46806d870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc46c005880 tx=0x7fc46c009040 comp rx=0 tx=0).stop 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0ca590 msgr2=0x7fc47c0726a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0ca590 0x7fc47c0726a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc470009890 tx=0x7fc4700046c0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 shutdown_connections 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc46806b3b0 0x7fc46806d870 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc47c152aa0 0x7fc47c149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc47c0ca590 0x7fc47c0726a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 --2- 172.21.15.172:0/1551901672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c066fb0 0x7fc47c1f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.009+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 >> 172.21.15.172:0/1551901672 conn(0x7fc47c06efd0 msgr2=0x7fc47c06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.010+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 shutdown_connections 2024-03-16T19:31:57.991 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.010+0000 7fc482d9a640 1 -- 172.21.15.172:0/1551901672 wait complete. 2024-03-16T19:31:58.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-16T19:31:58.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.127+0000 7f78aee84640 1 Processor -- start 2024-03-16T19:31:58.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78aee84640 1 -- start start 2024-03-16T19:31:58.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78aee84640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f78a814d600 0x7f78a80c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:58.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78aee84640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f78a80cbee0 0x7f78a8070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:58.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78aee84640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a80cc2f0 0x7f78a80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:58.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78aee84640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a80c9930 con 0x7f78a814d600 2024-03-16T19:31:58.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78aee84640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a80c6a70 con 0x7f78a80cbee0 2024-03-16T19:31:58.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78aee84640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80cc770 con 0x7f78a80cc2f0 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78acc20640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f78a814d600 0x7f78a80c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78acc20640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f78a814d600 0x7f78a80c93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52690/0 (socket says 172.21.15.172:52690) 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.128+0000 7f78acc20640 1 -- 172.21.15.172:0/3151841604 learned_addr learned my addr 172.21.15.172:0/3151841604 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.129+0000 7f78a7fff640 1 --2- 172.21.15.172:0/3151841604 >> v2:172.21.15.174:3300/0 conn(0x7f78a80cbee0 0x7f78a8070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.129+0000 7f78ad421640 1 --2- 172.21.15.172:0/3151841604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a80cc2f0 0x7f78a80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.129+0000 7f78acc20640 1 -- 172.21.15.172:0/3151841604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a80cc2f0 msgr2=0x7f78a80710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.129+0000 7f78acc20640 1 --2- 172.21.15.172:0/3151841604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a80cc2f0 0x7f78a80710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.129+0000 7f78acc20640 1 -- 172.21.15.172:0/3151841604 >> v2:172.21.15.174:3300/0 conn(0x7f78a80cbee0 msgr2=0x7f78a8070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.129+0000 7f78acc20640 1 --2- 172.21.15.172:0/3151841604 >> v2:172.21.15.174:3300/0 conn(0x7f78a80cbee0 0x7f78a8070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.129+0000 7f78acc20640 1 -- 172.21.15.172:0/3151841604 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a8156e20 con 0x7f78a814d600 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.129+0000 7f78a7fff640 1 --2- 172.21.15.172:0/3151841604 >> v2:172.21.15.174:3300/0 conn(0x7f78a80cbee0 0x7f78a8070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78acc20640 1 --2- 172.21.15.172:0/3151841604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a814d600 0x7f78a80c93f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f78a8156d10 tx=0x7f78a002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eab1d53ad7939dd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78a77fe640 1 -- 172.21.15.172:0/3151841604 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a000b0d0 con 0x7f78a814d600 2024-03-16T19:31:58.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78a77fe640 1 -- 172.21.15.172:0/3151841604 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a000b230 con 0x7f78a814d600 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78a77fe640 1 -- 172.21.15.172:0/3151841604 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0009600 con 0x7f78a814d600 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78aee84640 1 -- 172.21.15.172:0/3151841604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a814d600 msgr2=0x7f78a80c93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78aee84640 1 --2- 172.21.15.172:0/3151841604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a814d600 0x7f78a80c93f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f78a8156d10 tx=0x7f78a002e170 comp rx=0 tx=0).stop 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78aee84640 1 -- 172.21.15.172:0/3151841604 shutdown_connections 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78aee84640 1 --2- 172.21.15.172:0/3151841604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a80cc2f0 0x7f78a80710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78aee84640 1 --2- 172.21.15.172:0/3151841604 >> v2:172.21.15.174:3300/0 conn(0x7f78a80cbee0 0x7f78a8070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78aee84640 1 --2- 172.21.15.172:0/3151841604 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a814d600 0x7f78a80c93f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78aee84640 1 -- 172.21.15.172:0/3151841604 >> 172.21.15.172:0/3151841604 conn(0x7f78a8066c20 msgr2=0x7f78a8067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78aee84640 1 -- 172.21.15.172:0/3151841604 shutdown_connections 2024-03-16T19:31:58.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.130+0000 7f78aee84640 1 -- 172.21.15.172:0/3151841604 wait complete. 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.131+0000 7f78aee84640 1 Processor -- start 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.131+0000 7f78aee84640 1 -- start start 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78aee84640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a80cbee0 0x7f78a8152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78aee84640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78a80cc2f0 0x7f78a8153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78aee84640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a814d600 0x7f78a814e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78aee84640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a8152ca0 con 0x7f78a80cbee0 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78aee84640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a814de00 con 0x7f78a80cc2f0 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78aee84640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a81f0fd0 con 0x7f78a814d600 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78acc20640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a80cbee0 0x7f78a8152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:58.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78acc20640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a80cbee0 0x7f78a8152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:52692/0 (socket says 172.21.15.172:52692) 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78acc20640 1 -- 172.21.15.172:0/1306626072 learned_addr learned my addr 172.21.15.172:0/1306626072 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78a7fff640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78a80cc2f0 0x7f78a8153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.132+0000 7f78ad421640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a814d600 0x7f78a814e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78acc20640 1 -- 172.21.15.172:0/1306626072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a814d600 msgr2=0x7f78a814e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78acc20640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a814d600 0x7f78a814e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78acc20640 1 -- 172.21.15.172:0/1306626072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78a80cc2f0 msgr2=0x7f78a8153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78acc20640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78a80cc2f0 0x7f78a8153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78acc20640 1 -- 172.21.15.172:0/1306626072 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7898009400 con 0x7f78a80cbee0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78acc20640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a80cbee0 0x7f78a8152760 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f78a0038040 tx=0x7f78a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78a57fa640 1 -- 172.21.15.172:0/1306626072 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0040620 con 0x7f78a80cbee0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78a57fa640 1 -- 172.21.15.172:0/1306626072 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a00407a0 con 0x7f78a80cbee0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78a57fa640 1 -- 172.21.15.172:0/1306626072 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0040b50 con 0x7f78a80cbee0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a000bd40 con 0x7f78a80cbee0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.133+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a8151600 con 0x7f78a80cbee0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.134+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a80c35e0 con 0x7f78a80cbee0 2024-03-16T19:31:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.134+0000 7f78a57fa640 1 -- 172.21.15.172:0/1306626072 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f78a0040cb0 con 0x7f78a80cbee0 2024-03-16T19:31:58.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.135+0000 7f78a57fa640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f788806b3b0 0x7f788806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:58.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.135+0000 7f78a57fa640 1 -- 172.21.15.172:0/1306626072 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f78a00acdb0 con 0x7f78a80cbee0 2024-03-16T19:31:58.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.135+0000 7f78a7fff640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f788806b3b0 0x7f788806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:58.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.135+0000 7f78a7fff640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f788806b3b0 0x7f788806d870 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f78a8156930 tx=0x7f7898009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:58.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.139+0000 7f78a57fa640 1 -- 172.21.15.172:0/1306626072 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78a0077a60 con 0x7f78a80cbee0 2024-03-16T19:31:58.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:58.325+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f78a80681c0 con 0x7f78a80cbee0 2024-03-16T19:31:58.998 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.017+0000 7f78a57fa640 1 -- 172.21.15.172:0/1306626072 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v98) v1 ==== 133+0+0 (secure 0 0 0) 0x7f78a007b120 con 0x7f78a80cbee0 2024-03-16T19:31:58.999 INFO:teuthology.orchestra.run.smithi172.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.020+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f788806b3b0 msgr2=0x7f788806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.020+0000 7f78aee84640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f788806b3b0 0x7f788806d870 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f78a8156930 tx=0x7f7898009190 comp rx=0 tx=0).stop 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.020+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a80cbee0 msgr2=0x7f78a8152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.020+0000 7f78aee84640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a80cbee0 0x7f78a8152760 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f78a0038040 tx=0x7f78a0004a40 comp rx=0 tx=0).stop 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.021+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 shutdown_connections 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.021+0000 7f78aee84640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f788806b3b0 0x7f788806d870 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.021+0000 7f78aee84640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78a814d600 0x7f78a814e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.021+0000 7f78aee84640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78a80cc2f0 0x7f78a8153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.021+0000 7f78aee84640 1 --2- 172.21.15.172:0/1306626072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78a80cbee0 0x7f78a8152760 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.021+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 >> 172.21.15.172:0/1306626072 conn(0x7f78a8066c20 msgr2=0x7f78a81553b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:59.002 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.021+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 shutdown_connections 2024-03-16T19:31:59.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.021+0000 7f78aee84640 1 -- 172.21.15.172:0/1306626072 wait complete. 2024-03-16T19:31:59.014 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-16T19:31:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.137+0000 7f399af3e640 1 Processor -- start 2024-03-16T19:31:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.137+0000 7f399af3e640 1 -- start start 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f399af3e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f399415a370 0x7f399415a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f399af3e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39940c6a70 con 0x7f399414d600 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f399af3e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399415ad40 con 0x7f39940c93f0 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f399af3e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39940cc770 con 0x7f399415a370 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f3998cda640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f399414d600 0x7f39940cbee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:37962/0 (socket says 172.21.15.172:37962) 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f3998cda640 1 -- 172.21.15.172:0/2387940863 learned_addr learned my addr 172.21.15.172:0/2387940863 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f39994db640 1 --2- 172.21.15.172:0/2387940863 >> v2:172.21.15.174:3300/0 conn(0x7f399415a370 0x7f399415a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f39937fe640 1 -- 172.21.15.172:0/2387940863 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798205467 0 0) 0x7f39940c6a70 con 0x7f399414d600 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f39937fe640 1 -- 172.21.15.172:0/2387940863 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3984003440 con 0x7f399414d600 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f39994db640 1 -- 172.21.15.172:0/2387940863 >> v1:172.21.15.174:6789/0 conn(0x7f39940c93f0 legacy=0x7f3994070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f39994db640 1 -- 172.21.15.172:0/2387940863 >> v1:172.21.15.172:6789/0 conn(0x7f399414d600 legacy=0x7f39940cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:59.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.138+0000 7f39994db640 1 -- 172.21.15.172:0/2387940863 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994156e20 con 0x7f399415a370 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.139+0000 7f39994db640 1 --2- 172.21.15.172:0/2387940863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 0x7f399415a800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f398c009d90 tx=0x7f398c02bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75b5c1e108c1ef06 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.139+0000 7f39937fe640 1 -- 172.21.15.172:0/2387940863 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c02cd80 con 0x7f399415a370 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.139+0000 7f39937fe640 1 -- 172.21.15.172:0/2387940863 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c034440 con 0x7f399415a370 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.139+0000 7f39937fe640 1 -- 172.21.15.172:0/2387940863 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c0347b0 con 0x7f399415a370 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.140+0000 7f399af3e640 1 -- 172.21.15.172:0/2387940863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 msgr2=0x7f399415a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.140+0000 7f399af3e640 1 --2- 172.21.15.172:0/2387940863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 0x7f399415a800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f398c009d90 tx=0x7f398c02bfd0 comp rx=0 tx=0).stop 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.140+0000 7f399af3e640 1 -- 172.21.15.172:0/2387940863 shutdown_connections 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.140+0000 7f399af3e640 1 --2- 172.21.15.172:0/2387940863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 0x7f399415a800 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.140+0000 7f399af3e640 1 -- 172.21.15.172:0/2387940863 >> 172.21.15.172:0/2387940863 conn(0x7f3994066c20 msgr2=0x7f3994067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.140+0000 7f399af3e640 1 -- 172.21.15.172:0/2387940863 shutdown_connections 2024-03-16T19:31:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.140+0000 7f399af3e640 1 -- 172.21.15.172:0/2387940863 wait complete. 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.141+0000 7f399af3e640 1 Processor -- start 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.141+0000 7f399af3e640 1 -- start start 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f399af3e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39940c93f0 0x7f39941503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f399af3e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f399414d600 0x7f39940cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f399af3e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 0x7f39941ec520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f399af3e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399414a990 con 0x7f39940c93f0 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f399af3e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39940cbf50 con 0x7f399415a370 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f399af3e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39941efaf0 con 0x7f399414d600 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3998cda640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39940c93f0 0x7f39941503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3998cda640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39940c93f0 0x7f39941503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38956/0 (socket says 172.21.15.172:38956) 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3998cda640 1 -- 172.21.15.172:0/4086628596 learned_addr learned my addr 172.21.15.172:0/4086628596 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:31:59.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f39994db640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 0x7f39941ec520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3993fff640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f399414d600 0x7f39940cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3998cda640 1 -- 172.21.15.172:0/4086628596 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f399414d600 msgr2=0x7f39940cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3998cda640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f399414d600 0x7f39940cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3998cda640 1 -- 172.21.15.172:0/4086628596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 msgr2=0x7f39941ec520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3998cda640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 0x7f39941ec520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3998cda640 1 -- 172.21.15.172:0/4086628596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c009a70 con 0x7f39940c93f0 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f39994db640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 0x7f39941ec520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f3998cda640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39940c93f0 0x7f39941503a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f398800c500 tx=0x7f398800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f39917fa640 1 -- 172.21.15.172:0/4086628596 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988005ce0 con 0x7f39940c93f0 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f39917fa640 1 -- 172.21.15.172:0/4086628596 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3988005e40 con 0x7f39940c93f0 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f39917fa640 1 -- 172.21.15.172:0/4086628596 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988007330 con 0x7f39940c93f0 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39940cc1f0 con 0x7f39940c93f0 2024-03-16T19:31:59.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.142+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39941f0070 con 0x7f39940c93f0 2024-03-16T19:31:59.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.143+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39940c35e0 con 0x7f39940c93f0 2024-03-16T19:31:59.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.144+0000 7f39917fa640 1 -- 172.21.15.172:0/4086628596 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3988007490 con 0x7f39940c93f0 2024-03-16T19:31:59.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.144+0000 7f39917fa640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f397806b3e0 0x7f397806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:31:59.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.144+0000 7f3993fff640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f397806b3e0 0x7f397806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:31:59.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.144+0000 7f39917fa640 1 -- 172.21.15.172:0/4086628596 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f3988089200 con 0x7f39940c93f0 2024-03-16T19:31:59.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.145+0000 7f3993fff640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f397806b3e0 0x7f397806d8a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f39940c5cc0 tx=0x7f3980009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:31:59.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.149+0000 7f39917fa640 1 -- 172.21.15.172:0/4086628596 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3988054000 con 0x7f39940c93f0 2024-03-16T19:31:59.312 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:31:59.332+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f39940c5700 con 0x7f39940c93f0 2024-03-16T19:32:00.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.019+0000 7f39917fa640 1 -- 172.21.15.172:0/4086628596 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v99) v1 ==== 113+0+0 (secure 0 0 0) 0x7f39880576c0 con 0x7f39940c93f0 2024-03-16T19:32:00.000 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.6 to 1 (10000) 2024-03-16T19:32:00.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f397806b3e0 msgr2=0x7f397806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:00.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f397806b3e0 0x7f397806d8a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f39940c5cc0 tx=0x7f3980009040 comp rx=0 tx=0).stop 2024-03-16T19:32:00.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39940c93f0 msgr2=0x7f39941503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:00.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39940c93f0 0x7f39941503a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f398800c500 tx=0x7f398800c9c0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 shutdown_connections 2024-03-16T19:32:00.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f399415a370 0x7f39941ec520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f397806b3e0 0x7f397806d8a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.003 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f399414d600 0x7f39940cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 --2- 172.21.15.172:0/4086628596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39940c93f0 0x7f39941503a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 >> 172.21.15.172:0/4086628596 conn(0x7f3994066c20 msgr2=0x7f39940c44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:00.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 shutdown_connections 2024-03-16T19:32:00.004 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.022+0000 7f399af3e640 1 -- 172.21.15.172:0/4086628596 wait complete. 2024-03-16T19:32:00.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-16T19:32:00.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.143+0000 7effadb99640 1 Processor -- start 2024-03-16T19:32:00.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.143+0000 7effadb99640 1 -- start start 2024-03-16T19:32:00.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effadb99640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7effa80b0e90 0x7effa806db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:00.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effadb99640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 0x7effa815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:00.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effadb99640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa806bfe0 con 0x7effa8158190 2024-03-16T19:32:00.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effadb99640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa8070300 con 0x7effa80b0e90 2024-03-16T19:32:00.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effadb99640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa80ce970 con 0x7effa806ea50 2024-03-16T19:32:00.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effa6ffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7effa80b0e90 0x7effa806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:00.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effa77fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7effa8158190 0x7effa806f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:37964/0 (socket says 172.21.15.172:37964) 2024-03-16T19:32:00.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effa77fe640 1 -- 172.21.15.172:0/3572431703 learned_addr learned my addr 172.21.15.172:0/3572431703 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:00.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effa7fff640 1 --2- 172.21.15.172:0/3572431703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 0x7effa815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:00.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effa6ffd640 1 -- 172.21.15.172:0/3572431703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 msgr2=0x7effa815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:00.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effa6ffd640 1 --2- 172.21.15.172:0/3572431703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 0x7effa815abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effa6ffd640 1 -- 172.21.15.172:0/3572431703 >> v1:172.21.15.172:6789/0 conn(0x7effa8158190 legacy=0x7effa806f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:00.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.144+0000 7effa6ffd640 1 -- 172.21.15.172:0/3572431703 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa80ce360 con 0x7effa80b0e90 2024-03-16T19:32:00.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.145+0000 7effa6ffd640 1 --2- 172.21.15.172:0/3572431703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 0x7effa806db90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0097a0 tx=0x7eff9c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61650e9bddb17798 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:00.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.145+0000 7effa67fc640 1 -- 172.21.15.172:0/3572431703 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c030610 con 0x7effa80b0e90 2024-03-16T19:32:00.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.145+0000 7effa67fc640 1 -- 172.21.15.172:0/3572431703 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c030770 con 0x7effa80b0e90 2024-03-16T19:32:00.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.145+0000 7effa67fc640 1 -- 172.21.15.172:0/3572431703 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c030b00 con 0x7effa80b0e90 2024-03-16T19:32:00.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.146+0000 7effadb99640 1 -- 172.21.15.172:0/3572431703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 msgr2=0x7effa806db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:00.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.146+0000 7effadb99640 1 --2- 172.21.15.172:0/3572431703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 0x7effa806db90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0097a0 tx=0x7eff9c02b960 comp rx=0 tx=0).stop 2024-03-16T19:32:00.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.146+0000 7effadb99640 1 -- 172.21.15.172:0/3572431703 shutdown_connections 2024-03-16T19:32:00.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.146+0000 7effadb99640 1 --2- 172.21.15.172:0/3572431703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 0x7effa815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.146+0000 7effadb99640 1 --2- 172.21.15.172:0/3572431703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 0x7effa806db90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.146+0000 7effadb99640 1 -- 172.21.15.172:0/3572431703 >> 172.21.15.172:0/3572431703 conn(0x7effa8066c20 msgr2=0x7effa8067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:00.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.146+0000 7effadb99640 1 -- 172.21.15.172:0/3572431703 shutdown_connections 2024-03-16T19:32:00.128 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.146+0000 7effadb99640 1 -- 172.21.15.172:0/3572431703 wait complete. 2024-03-16T19:32:00.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.147+0000 7effadb99640 1 Processor -- start 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.147+0000 7effadb99640 1 -- start start 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effadb99640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 0x7effa8158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effadb99640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 0x7effa815a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effadb99640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa8158190 0x7effa81595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effadb99640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa8154f50 con 0x7effa80b0e90 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effadb99640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa81e7890 con 0x7effa8158190 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effadb99640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa81e7dc0 con 0x7effa806ea50 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa6ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 0x7effa815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa77fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 0x7effa8158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa6ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 0x7effa815a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38974/0 (socket says 172.21.15.172:38974) 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa6ffd640 1 -- 172.21.15.172:0/3502516663 learned_addr learned my addr 172.21.15.172:0/3502516663 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa7fff640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa8158190 0x7effa81595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa77fe640 1 -- 172.21.15.172:0/3502516663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa8158190 msgr2=0x7effa81595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:00.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa77fe640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa8158190 0x7effa81595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa77fe640 1 -- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 msgr2=0x7effa815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa77fe640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 0x7effa815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa77fe640 1 -- 172.21.15.172:0/3502516663 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c009400 con 0x7effa806ea50 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa6ffd640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 0x7effa815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.148+0000 7effa77fe640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 0x7effa8158a50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7eff98000f60 tx=0x7eff9800c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.149+0000 7eff8ffff640 1 -- 172.21.15.172:0/3502516663 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98005ce0 con 0x7effa806ea50 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.149+0000 7eff8ffff640 1 -- 172.21.15.172:0/3502516663 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff98005e40 con 0x7effa806ea50 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.149+0000 7eff8ffff640 1 -- 172.21.15.172:0/3502516663 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98007310 con 0x7effa806ea50 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.149+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa815b580 con 0x7effa806ea50 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.149+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa815b890 con 0x7effa806ea50 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.150+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa80681c0 con 0x7effa806ea50 2024-03-16T19:32:00.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.150+0000 7eff8ffff640 1 -- 172.21.15.172:0/3502516663 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7eff98007470 con 0x7effa806ea50 2024-03-16T19:32:00.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.150+0000 7eff8ffff640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7eff9406b3b0 0x7eff9406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:00.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.150+0000 7eff8ffff640 1 -- 172.21.15.172:0/3502516663 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5799+0+0 (secure 0 0 0) 0x7eff980890a0 con 0x7effa806ea50 2024-03-16T19:32:00.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.150+0000 7effa6ffd640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7eff9406b3b0 0x7eff9406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:00.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.151+0000 7effa6ffd640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7eff9406b3b0 0x7eff9406d870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7eff9c02b9a0 tx=0x7eff9c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:00.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.155+0000 7eff8ffff640 1 -- 172.21.15.172:0/3502516663 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff98053d30 con 0x7effa806ea50 2024-03-16T19:32:00.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:00.336+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) v1 -- 0x7effa806fda0 con 0x7effa806ea50 2024-03-16T19:32:01.010 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.029+0000 7eff8ffff640 1 -- 172.21.15.172:0/3502516663 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v100) v1 ==== 133+0+0 (secure 0 0 0) 0x7eff980170a0 con 0x7effa806ea50 2024-03-16T19:32:01.010 INFO:teuthology.orchestra.run.smithi172.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.031+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7eff9406b3b0 msgr2=0x7eff9406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.031+0000 7effadb99640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7eff9406b3b0 0x7eff9406d870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7eff9c02b9a0 tx=0x7eff9c008030 comp rx=0 tx=0).stop 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.031+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 msgr2=0x7effa8158a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.031+0000 7effadb99640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 0x7effa8158a50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7eff98000f60 tx=0x7eff9800c760 comp rx=0 tx=0).stop 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.032+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 shutdown_connections 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.032+0000 7effadb99640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa8158190 0x7effa81595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.032+0000 7effadb99640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7effa80b0e90 0x7effa815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.032+0000 7effadb99640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7eff9406b3b0 0x7eff9406d870 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.032+0000 7effadb99640 1 --2- 172.21.15.172:0/3502516663 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effa806ea50 0x7effa8158a50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.032+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 >> 172.21.15.172:0/3502516663 conn(0x7effa8066c20 msgr2=0x7effa80cc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.032+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 shutdown_connections 2024-03-16T19:32:01.013 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.032+0000 7effadb99640 1 -- 172.21.15.172:0/3502516663 wait complete. 2024-03-16T19:32:01.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-16T19:32:01.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.149+0000 7f9db3edf640 1 Processor -- start 2024-03-16T19:32:01.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.149+0000 7f9db3edf640 1 -- start start 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.150+0000 7f9db3edf640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9dac0cc570 0x7f9dac066ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.150+0000 7f9db3edf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9dac0c6970 0x7f9dac0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.150+0000 7f9db3edf640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac15ee90 con 0x7f9dac0cae20 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.150+0000 7f9db3edf640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dac15e9a0 con 0x7f9dac0cc570 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.150+0000 7f9db3edf640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dac0cce90 con 0x7f9dac0c6970 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.150+0000 7f9db1c7b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9dac0cc570 0x7f9dac066ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.150+0000 7f9db1c7b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9dac0cc570 0x7f9dac066ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38978/0 (socket says 172.21.15.172:38978) 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.150+0000 7f9db1c7b640 1 -- 172.21.15.172:0/3099899250 learned_addr learned my addr 172.21.15.172:0/3099899250 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.150+0000 7f9db147a640 1 --2- 172.21.15.172:0/3099899250 >> v2:172.21.15.174:3300/0 conn(0x7f9dac0c6970 0x7f9dac0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.151+0000 7f9db1c7b640 1 -- 172.21.15.172:0/3099899250 >> v2:172.21.15.174:3300/0 conn(0x7f9dac0c6970 msgr2=0x7f9dac0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.151+0000 7f9db1c7b640 1 --2- 172.21.15.172:0/3099899250 >> v2:172.21.15.174:3300/0 conn(0x7f9dac0c6970 0x7f9dac0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.151+0000 7f9db1c7b640 1 -- 172.21.15.172:0/3099899250 >> v1:172.21.15.174:6789/0 conn(0x7f9dac0cae20 legacy=0x7f9dac15a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.151+0000 7f9db1c7b640 1 -- 172.21.15.172:0/3099899250 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac0ca590 con 0x7f9dac0cc570 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.151+0000 7f9db1c7b640 1 --2- 172.21.15.172:0/3099899250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cc570 0x7f9dac066ed0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9dac06cc40 tx=0x7f9da802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56472a5f5b770f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.151+0000 7f9db0c79640 1 -- 172.21.15.172:0/3099899250 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da802ca80 con 0x7f9dac0cc570 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.151+0000 7f9db0c79640 1 -- 172.21.15.172:0/3099899250 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da802cbe0 con 0x7f9dac0cc570 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.151+0000 7f9db0c79640 1 -- 172.21.15.172:0/3099899250 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da80343f0 con 0x7f9dac0cc570 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.152+0000 7f9db3edf640 1 -- 172.21.15.172:0/3099899250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cc570 msgr2=0x7f9dac066ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.152+0000 7f9db3edf640 1 --2- 172.21.15.172:0/3099899250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cc570 0x7f9dac066ed0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9dac06cc40 tx=0x7f9da802bee0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.152+0000 7f9db3edf640 1 -- 172.21.15.172:0/3099899250 shutdown_connections 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.152+0000 7f9db3edf640 1 --2- 172.21.15.172:0/3099899250 >> v2:172.21.15.174:3300/0 conn(0x7f9dac0c6970 0x7f9dac0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.152+0000 7f9db3edf640 1 --2- 172.21.15.172:0/3099899250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cc570 0x7f9dac066ed0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.152+0000 7f9db3edf640 1 -- 172.21.15.172:0/3099899250 >> 172.21.15.172:0/3099899250 conn(0x7f9dac06efd0 msgr2=0x7f9dac06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:01.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.152+0000 7f9db3edf640 1 -- 172.21.15.172:0/3099899250 shutdown_connections 2024-03-16T19:32:01.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.152+0000 7f9db3edf640 1 -- 172.21.15.172:0/3099899250 wait complete. 2024-03-16T19:32:01.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.153+0000 7f9db3edf640 1 Processor -- start 2024-03-16T19:32:01.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.153+0000 7f9db3edf640 1 -- start start 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db3edf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dac0c6970 0x7f9dac1e45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db3edf640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cae20 0x7f9dac0c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db3edf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dac0cc570 0x7f9dac1f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db3edf640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac15cf30 con 0x7f9dac0cae20 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db3edf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac14dec0 con 0x7f9dac0c6970 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db3edf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac1e4c90 con 0x7f9dac0cc570 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db147a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cae20 0x7f9dac0c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db1c7b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dac0c6970 0x7f9dac1e45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db147a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cae20 0x7f9dac0c4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38994/0 (socket says 172.21.15.172:38994) 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db147a640 1 -- 172.21.15.172:0/3319946524 learned_addr learned my addr 172.21.15.172:0/3319946524 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db247c640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dac0cc570 0x7f9dac1f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db147a640 1 -- 172.21.15.172:0/3319946524 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dac0cc570 msgr2=0x7f9dac1f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db147a640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dac0cc570 0x7f9dac1f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db147a640 1 -- 172.21.15.172:0/3319946524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dac0c6970 msgr2=0x7f9dac1e45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db147a640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dac0c6970 0x7f9dac1e45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.154+0000 7f9db147a640 1 -- 172.21.15.172:0/3319946524 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da8009b30 con 0x7f9dac0cae20 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.155+0000 7f9db147a640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cae20 0x7f9dac0c4700 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00e870 tx=0x7f9d9c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.155+0000 7f9da27fc640 1 -- 172.21.15.172:0/3319946524 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c0094f0 con 0x7f9dac0cae20 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.155+0000 7f9da27fc640 1 -- 172.21.15.172:0/3319946524 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c00c940 con 0x7f9dac0cae20 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.155+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dac14e110 con 0x7f9dac0cae20 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.155+0000 7f9da27fc640 1 -- 172.21.15.172:0/3319946524 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c00ccf0 con 0x7f9dac0cae20 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.155+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dac14e6d0 con 0x7f9dac0cae20 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.156+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dac0cb8e0 con 0x7f9dac0cae20 2024-03-16T19:32:01.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.156+0000 7f9da27fc640 1 -- 172.21.15.172:0/3319946524 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9d9c013470 con 0x7f9dac0cae20 2024-03-16T19:32:01.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.156+0000 7f9da27fc640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9006b3b0 0x7f9d9006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:01.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.157+0000 7f9db1c7b640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9006b3b0 0x7f9d9006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:01.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.157+0000 7f9da27fc640 1 -- 172.21.15.172:0/3319946524 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f9d9c089ce0 con 0x7f9dac0cae20 2024-03-16T19:32:01.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.157+0000 7f9db1c7b640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9006b3b0 0x7f9d9006d870 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9da8035000 tx=0x7f9da8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:01.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.161+0000 7f9da27fc640 1 -- 172.21.15.172:0/3319946524 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d9c054900 con 0x7f9dac0cae20 2024-03-16T19:32:01.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:01.344+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f9dac0681c0 con 0x7f9dac0cae20 2024-03-16T19:32:02.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.031+0000 7f9da27fc640 1 -- 172.21.15.172:0/3319946524 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v101) v1 ==== 113+0+0 (secure 0 0 0) 0x7f9d9c021020 con 0x7f9dac0cae20 2024-03-16T19:32:02.012 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.7 to 1 (10000) 2024-03-16T19:32:02.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9006b3b0 msgr2=0x7f9d9006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:02.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9006b3b0 0x7f9d9006d870 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9da8035000 tx=0x7f9da8030000 comp rx=0 tx=0).stop 2024-03-16T19:32:02.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cae20 msgr2=0x7f9dac0c4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:02.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cae20 0x7f9dac0c4700 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00e870 tx=0x7f9d9c00ed30 comp rx=0 tx=0).stop 2024-03-16T19:32:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 shutdown_connections 2024-03-16T19:32:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9006b3b0 0x7f9d9006d870 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9dac0cc570 0x7f9dac1f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9dac0cae20 0x7f9dac0c4700 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 --2- 172.21.15.172:0/3319946524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9dac0c6970 0x7f9dac1e45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 >> 172.21.15.172:0/3319946524 conn(0x7f9dac06efd0 msgr2=0x7f9dac0c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 shutdown_connections 2024-03-16T19:32:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.034+0000 7f9db3edf640 1 -- 172.21.15.172:0/3319946524 wait complete. 2024-03-16T19:32:02.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-16T19:32:02.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.153+0000 7fac4ccc3640 1 Processor -- start 2024-03-16T19:32:02.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.153+0000 7fac4ccc3640 1 -- start start 2024-03-16T19:32:02.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.153+0000 7fac4ccc3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fac480c5910 0x7fac4806f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:02.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.153+0000 7fac4ccc3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5cf0 0x7fac480cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:02.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.153+0000 7fac4ccc3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac480c8360 con 0x7fac4815c860 2024-03-16T19:32:02.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.153+0000 7fac4ccc3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac480c88b0 con 0x7fac480c5910 2024-03-16T19:32:02.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.153+0000 7fac4ccc3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4806f550 con 0x7fac480c5cf0 2024-03-16T19:32:02.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.154+0000 7fac46d9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fac4815c860 0x7fac481585f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:37968/0 (socket says 172.21.15.172:37968) 2024-03-16T19:32:02.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.154+0000 7fac46d9d640 1 -- 172.21.15.172:0/1343536315 learned_addr learned my addr 172.21.15.172:0/1343536315 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:02.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.154+0000 7fac4659c640 1 --2- 172.21.15.172:0/1343536315 >> v2:172.21.15.174:3300/0 conn(0x7fac480c5910 0x7fac4806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:02.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.154+0000 7fac3dd9b640 1 --2- 172.21.15.172:0/1343536315 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5cf0 0x7fac480cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:02.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.154+0000 7fac45d9b640 1 -- 172.21.15.172:0/1343536315 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556895216 0 0) 0x7fac480c8360 con 0x7fac4815c860 2024-03-16T19:32:02.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.154+0000 7fac4659c640 1 -- 172.21.15.172:0/1343536315 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5cf0 msgr2=0x7fac480cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:02.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.154+0000 7fac4659c640 1 --2- 172.21.15.172:0/1343536315 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5cf0 0x7fac480cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:02.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.154+0000 7fac4659c640 1 -- 172.21.15.172:0/1343536315 >> v1:172.21.15.172:6789/0 conn(0x7fac4815c860 legacy=0x7fac481585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:02.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.154+0000 7fac4659c640 1 -- 172.21.15.172:0/1343536315 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480cceb0 con 0x7fac480c5910 2024-03-16T19:32:02.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac4659c640 1 --2- 172.21.15.172:0/1343536315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac480c5910 0x7fac4806f010 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fac380097a0 tx=0x7fac3802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d79abf7682b86966 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:02.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac45d9b640 1 -- 172.21.15.172:0/1343536315 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38030610 con 0x7fac480c5910 2024-03-16T19:32:02.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac45d9b640 1 -- 172.21.15.172:0/1343536315 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac38030770 con 0x7fac480c5910 2024-03-16T19:32:02.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac45d9b640 1 -- 172.21.15.172:0/1343536315 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38030b00 con 0x7fac480c5910 2024-03-16T19:32:02.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1343536315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac480c5910 msgr2=0x7fac4806f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:02.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac4ccc3640 1 --2- 172.21.15.172:0/1343536315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac480c5910 0x7fac4806f010 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fac380097a0 tx=0x7fac3802b960 comp rx=0 tx=0).stop 2024-03-16T19:32:02.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1343536315 shutdown_connections 2024-03-16T19:32:02.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac4ccc3640 1 --2- 172.21.15.172:0/1343536315 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5cf0 0x7fac480cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:02.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac4ccc3640 1 --2- 172.21.15.172:0/1343536315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac480c5910 0x7fac4806f010 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:02.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1343536315 >> 172.21.15.172:0/1343536315 conn(0x7fac48066c20 msgr2=0x7fac48067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:02.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1343536315 shutdown_connections 2024-03-16T19:32:02.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.155+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1343536315 wait complete. 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.157+0000 7fac4ccc3640 1 Processor -- start 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.157+0000 7fac4ccc3640 1 -- start start 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac4ccc3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5910 0x7fac480c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac4ccc3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac480c5cf0 0x7fac4814e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac4ccc3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac4815c860 0x7fac481f06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac4ccc3640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac480cbf10 con 0x7fac480c5cf0 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac4ccc3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac481f4800 con 0x7fac4815c860 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac4ccc3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac480c4da0 con 0x7fac480c5910 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3dd9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac480c5cf0 0x7fac4814e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3dd9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac480c5cf0 0x7fac4814e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39000/0 (socket says 172.21.15.172:39000) 2024-03-16T19:32:02.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3dd9b640 1 -- 172.21.15.172:0/1428274216 learned_addr learned my addr 172.21.15.172:0/1428274216 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac4659c640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5910 0x7fac480c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac46d9d640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac4815c860 0x7fac481f06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3dd9b640 1 -- 172.21.15.172:0/1428274216 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5910 msgr2=0x7fac480c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3dd9b640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5910 0x7fac480c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3dd9b640 1 -- 172.21.15.172:0/1428274216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac4815c860 msgr2=0x7fac481f06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3dd9b640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac4815c860 0x7fac481f06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3dd9b640 1 -- 172.21.15.172:0/1428274216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38009400 con 0x7fac480c5cf0 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac46d9d640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac4815c860 0x7fac481f06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3dd9b640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac480c5cf0 0x7fac4814e160 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fac3000e9b0 tx=0x7fac3000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.158+0000 7fac3f7fe640 1 -- 172.21.15.172:0/1428274216 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30009670 con 0x7fac480c5cf0 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.159+0000 7fac3f7fe640 1 -- 172.21.15.172:0/1428274216 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3000c9a0 con 0x7fac480c5cf0 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.159+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac48155f50 con 0x7fac480c5cf0 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.159+0000 7fac3f7fe640 1 -- 172.21.15.172:0/1428274216 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3000cd50 con 0x7fac480c5cf0 2024-03-16T19:32:02.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.159+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac481f3bf0 con 0x7fac480c5cf0 2024-03-16T19:32:02.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.161+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c005290 con 0x7fac480c5cf0 2024-03-16T19:32:02.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.161+0000 7fac3f7fe640 1 -- 172.21.15.172:0/1428274216 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fac30013470 con 0x7fac480c5cf0 2024-03-16T19:32:02.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.161+0000 7fac3f7fe640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fac2806b3e0 0x7fac2806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:02.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.162+0000 7fac3f7fe640 1 -- 172.21.15.172:0/1428274216 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fac30089de0 con 0x7fac480c5cf0 2024-03-16T19:32:02.142 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.162+0000 7fac4659c640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fac2806b3e0 0x7fac2806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:02.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.162+0000 7fac4659c640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fac2806b3e0 0x7fac2806d8a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fac38036000 tx=0x7fac38008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:02.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.166+0000 7fac3f7fe640 1 -- 172.21.15.172:0/1428274216 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac30054a80 con 0x7fac480c5cf0 2024-03-16T19:32:02.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:02.351+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7fac2c005ac0 con 0x7fac480c5cf0 2024-03-16T19:32:03.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.041+0000 7fac3f7fe640 1 -- 172.21.15.172:0/1428274216 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v102) v1 ==== 133+0+0 (secure 0 0 0) 0x7fac300170a0 con 0x7fac480c5cf0 2024-03-16T19:32:03.022 INFO:teuthology.orchestra.run.smithi172.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-16T19:32:03.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fac2806b3e0 msgr2=0x7fac2806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:03.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fac2806b3e0 0x7fac2806d8a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fac38036000 tx=0x7fac38008030 comp rx=0 tx=0).stop 2024-03-16T19:32:03.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac480c5cf0 msgr2=0x7fac4814e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:03.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac480c5cf0 0x7fac4814e160 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fac3000e9b0 tx=0x7fac3000ee70 comp rx=0 tx=0).stop 2024-03-16T19:32:03.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 shutdown_connections 2024-03-16T19:32:03.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac4815c860 0x7fac481f06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac480c5cf0 0x7fac4814e160 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fac2806b3e0 0x7fac2806d8a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 --2- 172.21.15.172:0/1428274216 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fac480c5910 0x7fac480c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 >> 172.21.15.172:0/1428274216 conn(0x7fac48066c20 msgr2=0x7fac4814fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:03.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 shutdown_connections 2024-03-16T19:32:03.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.044+0000 7fac4ccc3640 1 -- 172.21.15.172:0/1428274216 wait complete. 2024-03-16T19:32:03.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-16T19:32:03.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.166+0000 7fbbaca64640 1 Processor -- start 2024-03-16T19:32:03.147 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.166+0000 7fbbaca64640 1 -- start start 2024-03-16T19:32:03.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbbaca64640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbba8154a90 0x7fbba80c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:03.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbbaca64640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbba8154e70 0x7fbba806b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbbaca64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba80cc280 0x7fbba806be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbbaca64640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba815df50 con 0x7fbba8154a90 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbbaca64640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba815e1f0 con 0x7fbba8154e70 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbbaca64640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba815e4d0 con 0x7fbba80cc280 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba659c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbba8154a90 0x7fbba80c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba659c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbba8154a90 0x7fbba80c50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39008/0 (socket says 172.21.15.172:39008) 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba659c640 1 -- 172.21.15.172:0/3038565999 learned_addr learned my addr 172.21.15.172:0/3038565999 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba5d9b640 1 --2- 172.21.15.172:0/3038565999 >> v2:172.21.15.174:3300/0 conn(0x7fbba8154e70 0x7fbba806b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba6d9d640 1 --2- 172.21.15.172:0/3038565999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba80cc280 0x7fbba806be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba659c640 1 -- 172.21.15.172:0/3038565999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba80cc280 msgr2=0x7fbba806be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba659c640 1 --2- 172.21.15.172:0/3038565999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba80cc280 0x7fbba806be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba659c640 1 -- 172.21.15.172:0/3038565999 >> v2:172.21.15.174:3300/0 conn(0x7fbba8154e70 msgr2=0x7fbba806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba659c640 1 --2- 172.21.15.172:0/3038565999 >> v2:172.21.15.174:3300/0 conn(0x7fbba8154e70 0x7fbba806b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba659c640 1 -- 172.21.15.172:0/3038565999 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba815e7b0 con 0x7fbba8154a90 2024-03-16T19:32:03.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.167+0000 7fbba5d9b640 1 --2- 172.21.15.172:0/3038565999 >> v2:172.21.15.174:3300/0 conn(0x7fbba8154e70 0x7fbba806b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.168+0000 7fbba659c640 1 --2- 172.21.15.172:0/3038565999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154a90 0x7fbba80c50b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fbba808db80 tx=0x7fbb9802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77d754a7d4c7490 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.168+0000 7fbba559a640 1 -- 172.21.15.172:0/3038565999 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9802cb00 con 0x7fbba8154a90 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.168+0000 7fbba559a640 1 -- 172.21.15.172:0/3038565999 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9802cc60 con 0x7fbba8154a90 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.168+0000 7fbba559a640 1 -- 172.21.15.172:0/3038565999 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb980343f0 con 0x7fbba8154a90 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.169+0000 7fbbaca64640 1 -- 172.21.15.172:0/3038565999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154a90 msgr2=0x7fbba80c50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.169+0000 7fbbaca64640 1 --2- 172.21.15.172:0/3038565999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154a90 0x7fbba80c50b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fbba808db80 tx=0x7fbb9802bf60 comp rx=0 tx=0).stop 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.169+0000 7fbbaca64640 1 -- 172.21.15.172:0/3038565999 shutdown_connections 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.169+0000 7fbbaca64640 1 --2- 172.21.15.172:0/3038565999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba80cc280 0x7fbba806be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.169+0000 7fbbaca64640 1 --2- 172.21.15.172:0/3038565999 >> v2:172.21.15.174:3300/0 conn(0x7fbba8154e70 0x7fbba806b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.169+0000 7fbbaca64640 1 --2- 172.21.15.172:0/3038565999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154a90 0x7fbba80c50b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.169+0000 7fbbaca64640 1 -- 172.21.15.172:0/3038565999 >> 172.21.15.172:0/3038565999 conn(0x7fbba8067250 msgr2=0x7fbba80d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:03.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.169+0000 7fbbaca64640 1 -- 172.21.15.172:0/3038565999 shutdown_connections 2024-03-16T19:32:03.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.169+0000 7fbbaca64640 1 -- 172.21.15.172:0/3038565999 wait complete. 2024-03-16T19:32:03.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.170+0000 7fbbaca64640 1 Processor -- start 2024-03-16T19:32:03.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.170+0000 7fbbaca64640 1 -- start start 2024-03-16T19:32:03.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbbaca64640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba80cc280 0x7fbba80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbbaca64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba8154a90 0x7fbba8146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbbaca64640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154e70 0x7fbba8153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbbaca64640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba8151e50 con 0x7fbba8154e70 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbbaca64640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba80b5750 con 0x7fbba80cc280 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbbaca64640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba81f4390 con 0x7fbba8154a90 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba6d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154e70 0x7fbba8153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba5d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba8154a90 0x7fbba8146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba6d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154e70 0x7fbba8153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39010/0 (socket says 172.21.15.172:39010) 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba80cc280 0x7fbba80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba6d9d640 1 -- 172.21.15.172:0/2970993017 learned_addr learned my addr 172.21.15.172:0/2970993017 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba6d9d640 1 -- 172.21.15.172:0/2970993017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba8154a90 msgr2=0x7fbba8146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba6d9d640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba8154a90 0x7fbba8146180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba6d9d640 1 -- 172.21.15.172:0/2970993017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba80cc280 msgr2=0x7fbba80b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:03.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba6d9d640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba80cc280 0x7fbba80b4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:03.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.171+0000 7fbba6d9d640 1 -- 172.21.15.172:0/2970993017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c009400 con 0x7fbba8154e70 2024-03-16T19:32:03.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.172+0000 7fbba6d9d640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154e70 0x7fbba8153200 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbba000e880 tx=0x7fbba000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:03.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.172+0000 7fbb96ffd640 1 -- 172.21.15.172:0/2970993017 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba0009630 con 0x7fbba8154e70 2024-03-16T19:32:03.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.172+0000 7fbb96ffd640 1 -- 172.21.15.172:0/2970993017 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba000c940 con 0x7fbba8154e70 2024-03-16T19:32:03.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.172+0000 7fbb96ffd640 1 -- 172.21.15.172:0/2970993017 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba000ccf0 con 0x7fbba8154e70 2024-03-16T19:32:03.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.172+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb98009b30 con 0x7fbba8154e70 2024-03-16T19:32:03.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.172+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba80b5470 con 0x7fbba8154e70 2024-03-16T19:32:03.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.173+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba80681c0 con 0x7fbba8154e70 2024-03-16T19:32:03.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.173+0000 7fbb96ffd640 1 -- 172.21.15.172:0/2970993017 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbba0013470 con 0x7fbba8154e70 2024-03-16T19:32:03.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.174+0000 7fbb96ffd640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbb8806b3b0 0x7fbb8806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:03.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.174+0000 7fbba659c640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbb8806b3b0 0x7fbb8806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:03.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.174+0000 7fbb96ffd640 1 -- 172.21.15.172:0/2970993017 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fbba008a080 con 0x7fbba8154e70 2024-03-16T19:32:03.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.174+0000 7fbba659c640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbb8806b3b0 0x7fbb8806d870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbb98035000 tx=0x7fbb98030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:03.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.178+0000 7fbb96ffd640 1 -- 172.21.15.172:0/2970993017 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbba0054dc0 con 0x7fbba8154e70 2024-03-16T19:32:03.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:03.360+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fbba81f44d0 con 0x7fbba8154e70 2024-03-16T19:32:04.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.049+0000 7fbb96ffd640 1 -- 172.21.15.172:0/2970993017 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v103) v1 ==== 113+0+0 (secure 0 0 0) 0x7fbba0058480 con 0x7fbba8154e70 2024-03-16T19:32:04.030 INFO:teuthology.orchestra.run.smithi172.stderr:reweighted osd.2 to 1 (10000) 2024-03-16T19:32:04.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbb8806b3b0 msgr2=0x7fbb8806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:04.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbb8806b3b0 0x7fbb8806d870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbb98035000 tx=0x7fbb98030000 comp rx=0 tx=0).stop 2024-03-16T19:32:04.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154e70 msgr2=0x7fbba8153200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:04.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154e70 0x7fbba8153200 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbba000e880 tx=0x7fbba000ed40 comp rx=0 tx=0).stop 2024-03-16T19:32:04.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 shutdown_connections 2024-03-16T19:32:04.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbba8154e70 0x7fbba8153200 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:04.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbb8806b3b0 0x7fbb8806d870 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:04.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbba8154a90 0x7fbba8146180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:04.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 --2- 172.21.15.172:0/2970993017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbba80cc280 0x7fbba80b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:04.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 >> 172.21.15.172:0/2970993017 conn(0x7fbba8067250 msgr2=0x7fbba80cad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:04.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 shutdown_connections 2024-03-16T19:32:04.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.052+0000 7fbbaca64640 1 -- 172.21.15.172:0/2970993017 wait complete. 2024-03-16T19:32:04.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-16T19:32:04.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.170+0000 7f4aeb316640 1 Processor -- start 2024-03-16T19:32:04.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.170+0000 7f4aeb316640 1 -- start start 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.171+0000 7f4aeb316640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae4158150 0x7f4ae406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.171+0000 7f4aeb316640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4ae406c500 0x7f4ae40ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.171+0000 7f4aeb316640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae406e010 con 0x7f4ae406f4f0 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.171+0000 7f4aeb316640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae40c8f20 con 0x7f4ae406c500 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.171+0000 7f4aeb316640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae415b580 con 0x7f4ae4158150 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.171+0000 7f4ae98b3640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4ae406f4f0 0x7f4ae415df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:37980/0 (socket says 172.21.15.172:37980) 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.171+0000 7f4ae98b3640 1 -- 172.21.15.172:0/4030512942 learned_addr learned my addr 172.21.15.172:0/4030512942 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.171+0000 7f4ae90b2640 1 --2- 172.21.15.172:0/4030512942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae4158150 0x7f4ae406e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.171+0000 7f4ae88b1640 1 --2- 172.21.15.172:0/4030512942 >> v2:172.21.15.174:3300/0 conn(0x7f4ae406c500 0x7f4ae40ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.172+0000 7f4adbfff640 1 -- 172.21.15.172:0/4030512942 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137786530 0 0) 0x7f4ae406e010 con 0x7f4ae406f4f0 2024-03-16T19:32:04.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.172+0000 7f4adbfff640 1 -- 172.21.15.172:0/4030512942 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad0003520 con 0x7f4ae406f4f0 2024-03-16T19:32:04.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.172+0000 7f4ae90b2640 1 -- 172.21.15.172:0/4030512942 >> v2:172.21.15.174:3300/0 conn(0x7f4ae406c500 msgr2=0x7f4ae40ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:04.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.172+0000 7f4ae90b2640 1 --2- 172.21.15.172:0/4030512942 >> v2:172.21.15.174:3300/0 conn(0x7f4ae406c500 0x7f4ae40ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:04.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.172+0000 7f4ae90b2640 1 -- 172.21.15.172:0/4030512942 >> v1:172.21.15.172:6789/0 conn(0x7f4ae406f4f0 legacy=0x7f4ae415df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:04.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.172+0000 7f4ae90b2640 1 -- 172.21.15.172:0/4030512942 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae406d810 con 0x7f4ae4158150 2024-03-16T19:32:04.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.172+0000 7f4ae88b1640 1 --2- 172.21.15.172:0/4030512942 >> v2:172.21.15.174:3300/0 conn(0x7f4ae406c500 0x7f4ae40ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:04.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.172+0000 7f4ae90b2640 1 --2- 172.21.15.172:0/4030512942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae4158150 0x7f4ae406e9e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009d90 tx=0x7f4ae002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b310ceb95cbf4c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:04.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4adbfff640 1 -- 172.21.15.172:0/4030512942 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae002cbc0 con 0x7f4ae4158150 2024-03-16T19:32:04.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4adbfff640 1 -- 172.21.15.172:0/4030512942 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae002cd20 con 0x7f4ae4158150 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4adbfff640 1 -- 172.21.15.172:0/4030512942 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0034660 con 0x7f4ae4158150 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4aeb316640 1 -- 172.21.15.172:0/4030512942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae4158150 msgr2=0x7f4ae406e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4aeb316640 1 --2- 172.21.15.172:0/4030512942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae4158150 0x7f4ae406e9e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009d90 tx=0x7f4ae002be10 comp rx=0 tx=0).stop 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4aeb316640 1 -- 172.21.15.172:0/4030512942 shutdown_connections 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4aeb316640 1 --2- 172.21.15.172:0/4030512942 >> v2:172.21.15.174:3300/0 conn(0x7f4ae406c500 0x7f4ae40ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4aeb316640 1 --2- 172.21.15.172:0/4030512942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae4158150 0x7f4ae406e9e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4aeb316640 1 -- 172.21.15.172:0/4030512942 >> 172.21.15.172:0/4030512942 conn(0x7f4ae4066c20 msgr2=0x7f4ae4067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4aeb316640 1 -- 172.21.15.172:0/4030512942 shutdown_connections 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.173+0000 7f4aeb316640 1 -- 172.21.15.172:0/4030512942 wait complete. 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.174+0000 7f4aeb316640 1 Processor -- start 2024-03-16T19:32:04.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.174+0000 7f4aeb316640 1 -- start start 2024-03-16T19:32:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4aeb316640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae406c500 0x7f4ae41f1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4aeb316640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ae406f4f0 0x7f4ae41e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4aeb316640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4158150 0x7f4ae4153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4aeb316640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae41529b0 con 0x7f4ae4158150 2024-03-16T19:32:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4aeb316640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae41f1840 con 0x7f4ae406f4f0 2024-03-16T19:32:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4aeb316640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae4153fd0 con 0x7f4ae406c500 2024-03-16T19:32:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae98b3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4158150 0x7f4ae4153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae90b2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae406c500 0x7f4ae41f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae98b3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4158150 0x7f4ae4153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39012/0 (socket says 172.21.15.172:39012) 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae98b3640 1 -- 172.21.15.172:0/1850967730 learned_addr learned my addr 172.21.15.172:0/1850967730 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae88b1640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ae406f4f0 0x7f4ae41e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae90b2640 1 -- 172.21.15.172:0/1850967730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ae406f4f0 msgr2=0x7f4ae41e6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae90b2640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ae406f4f0 0x7f4ae41e6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae90b2640 1 -- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4158150 msgr2=0x7f4ae4153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae90b2640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4158150 0x7f4ae4153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae90b2640 1 -- 172.21.15.172:0/1850967730 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4009400 con 0x7f4ae406c500 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.175+0000 7f4ae98b3640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4158150 0x7f4ae4153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.176+0000 7f4ae90b2640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae406c500 0x7f4ae41f1300 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0033000 tx=0x7f4ae0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.176+0000 7f4ad9ffb640 1 -- 172.21.15.172:0/1850967730 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0040650 con 0x7f4ae406c500 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.176+0000 7f4ad9ffb640 1 -- 172.21.15.172:0/1850967730 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae002ce90 con 0x7f4ae406c500 2024-03-16T19:32:04.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.176+0000 7f4ad9ffb640 1 -- 172.21.15.172:0/1850967730 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0040a10 con 0x7f4ae406c500 2024-03-16T19:32:04.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.176+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae0009a70 con 0x7f4ae406c500 2024-03-16T19:32:04.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.176+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae4154590 con 0x7f4ae406c500 2024-03-16T19:32:04.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.177+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae4158d20 con 0x7f4ae406c500 2024-03-16T19:32:04.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.177+0000 7f4ad9ffb640 1 -- 172.21.15.172:0/1850967730 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4ae0040b70 con 0x7f4ae406c500 2024-03-16T19:32:04.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.178+0000 7f4ad9ffb640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ac806b3b0 0x7f4ac806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:04.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.178+0000 7f4ad9ffb640 1 -- 172.21.15.172:0/1850967730 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f4ae00ac6c0 con 0x7f4ae406c500 2024-03-16T19:32:04.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.178+0000 7f4ae88b1640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ac806b3b0 0x7f4ac806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:04.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.178+0000 7f4ae88b1640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ac806b3b0 0x7f4ac806d870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40cba10 tx=0x7f4ad4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:04.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.187+0000 7f4ad9ffb640 1 -- 172.21.15.172:0/1850967730 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ae0077310 con 0x7f4ae406c500 2024-03-16T19:32:04.350 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:04.369+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f4ae4075aa0 con 0x7f4ae406c500 2024-03-16T19:32:05.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.058+0000 7f4ad9ffb640 1 -- 172.21.15.172:0/1850967730 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v104) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4ae0009450 con 0x7f4ae406c500 2024-03-16T19:32:05.039 INFO:teuthology.orchestra.run.smithi172.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ac806b3b0 msgr2=0x7f4ac806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ac806b3b0 0x7f4ac806d870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40cba10 tx=0x7f4ad4009190 comp rx=0 tx=0).stop 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae406c500 msgr2=0x7f4ae41f1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae406c500 0x7f4ae41f1300 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0033000 tx=0x7f4ae0004a40 comp rx=0 tx=0).stop 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 shutdown_connections 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4158150 0x7f4ae4153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ae406f4f0 0x7f4ae41e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4ac806b3b0 0x7f4ac806d870 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 --2- 172.21.15.172:0/1850967730 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ae406c500 0x7f4ae41f1300 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 >> 172.21.15.172:0/1850967730 conn(0x7f4ae4066c20 msgr2=0x7f4ae40c41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 shutdown_connections 2024-03-16T19:32:05.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.061+0000 7f4aeb316640 1 -- 172.21.15.172:0/1850967730 wait complete. 2024-03-16T19:32:05.053 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-16T19:32:05.053 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-16T19:32:05.054 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-16T19:32:05.054 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-16T19:32:05.054 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-16T19:32:05.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:32:05.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.175+0000 7f8be092c640 1 Processor -- start 2024-03-16T19:32:05.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.176+0000 7f8be092c640 1 -- start start 2024-03-16T19:32:05.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.176+0000 7f8be092c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8bdc06bb10 0x7f8bdc0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.176+0000 7f8be092c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 0x7f8bdc0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.176+0000 7f8be092c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bdc0cc720 con 0x7f8bdc0cc260 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.176+0000 7f8be092c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bdc0cc9c0 con 0x7f8bdc06bb10 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.176+0000 7f8be092c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc15de30 con 0x7f8bdc06bef0 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.176+0000 7f8bda59c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8bdc06bb10 0x7f8bdc0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.176+0000 7f8bdad9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8bdc0cc260 0x7f8bdc154660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:37996/0 (socket says 172.21.15.172:37996) 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.176+0000 7f8bdad9d640 1 -- 172.21.15.172:0/28175962 learned_addr learned my addr 172.21.15.172:0/28175962 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.177+0000 7f8bd3fff640 1 --2- 172.21.15.172:0/28175962 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 0x7f8bdc0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.177+0000 7f8bda59c640 1 -- 172.21.15.172:0/28175962 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 msgr2=0x7f8bdc0cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.177+0000 7f8bda59c640 1 --2- 172.21.15.172:0/28175962 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 0x7f8bdc0cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.177+0000 7f8bda59c640 1 -- 172.21.15.172:0/28175962 >> v1:172.21.15.172:6789/0 conn(0x7f8bdc0cc260 legacy=0x7f8bdc154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.177+0000 7f8bd9d9b640 1 -- 172.21.15.172:0/28175962 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948724710 0 0) 0x7f8bdc0cc720 con 0x7f8bdc0cc260 2024-03-16T19:32:05.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.177+0000 7f8bda59c640 1 -- 172.21.15.172:0/28175962 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc15e110 con 0x7f8bdc06bb10 2024-03-16T19:32:05.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.178+0000 7f8bda59c640 1 --2- 172.21.15.172:0/28175962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 0x7f8bdc0cae50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc0097a0 tx=0x7f8bcc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4457172c520a8a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.178+0000 7f8bd9d9b640 1 -- 172.21.15.172:0/28175962 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc030610 con 0x7f8bdc06bb10 2024-03-16T19:32:05.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.178+0000 7f8bd9d9b640 1 -- 172.21.15.172:0/28175962 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc030770 con 0x7f8bdc06bb10 2024-03-16T19:32:05.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.178+0000 7f8bd9d9b640 1 -- 172.21.15.172:0/28175962 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc030b00 con 0x7f8bdc06bb10 2024-03-16T19:32:05.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.179+0000 7f8be092c640 1 -- 172.21.15.172:0/28175962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 msgr2=0x7f8bdc0cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.179+0000 7f8be092c640 1 --2- 172.21.15.172:0/28175962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 0x7f8bdc0cae50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc0097a0 tx=0x7f8bcc02b960 comp rx=0 tx=0).stop 2024-03-16T19:32:05.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.179+0000 7f8be092c640 1 -- 172.21.15.172:0/28175962 shutdown_connections 2024-03-16T19:32:05.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.179+0000 7f8be092c640 1 --2- 172.21.15.172:0/28175962 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 0x7f8bdc0cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.179+0000 7f8be092c640 1 --2- 172.21.15.172:0/28175962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 0x7f8bdc0cae50 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.179+0000 7f8be092c640 1 -- 172.21.15.172:0/28175962 >> 172.21.15.172:0/28175962 conn(0x7f8bdc067250 msgr2=0x7f8bdc0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.179+0000 7f8be092c640 1 -- 172.21.15.172:0/28175962 shutdown_connections 2024-03-16T19:32:05.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.179+0000 7f8be092c640 1 -- 172.21.15.172:0/28175962 wait complete. 2024-03-16T19:32:05.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.181+0000 7f8be092c640 1 Processor -- start 2024-03-16T19:32:05.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.181+0000 7f8be092c640 1 -- start start 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8be092c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 0x7f8bdc157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8be092c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 0x7f8bdc0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8be092c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bdc0cc260 0x7f8bdc1f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8be092c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc1f4380 con 0x7f8bdc06bb10 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8be092c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc1581e0 con 0x7f8bdc0cc260 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8be092c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc1f1070 con 0x7f8bdc06bef0 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bda59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 0x7f8bdc157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bda59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 0x7f8bdc157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39030/0 (socket says 172.21.15.172:39030) 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd3fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 0x7f8bdc0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bda59c640 1 -- 172.21.15.172:0/2816495848 learned_addr learned my addr 172.21.15.172:0/2816495848 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bdad9d640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bdc0cc260 0x7f8bdc1f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd3fff640 1 -- 172.21.15.172:0/2816495848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bdc0cc260 msgr2=0x7f8bdc1f4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd3fff640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bdc0cc260 0x7f8bdc1f4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd3fff640 1 -- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 msgr2=0x7f8bdc157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd3fff640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 0x7f8bdc157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd3fff640 1 -- 172.21.15.172:0/2816495848 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc009400 con 0x7f8bdc06bef0 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bda59c640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 0x7f8bdc157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd3fff640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 0x7f8bdc0d06e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400e820 tx=0x7f8bc400ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd2ffd640 1 -- 172.21.15.172:0/2816495848 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc40095b0 con 0x7f8bdc06bef0 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd2ffd640 1 -- 172.21.15.172:0/2816495848 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc400c940 con 0x7f8bdc06bef0 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.182+0000 7f8bd2ffd640 1 -- 172.21.15.172:0/2816495848 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc400ccf0 con 0x7f8bdc06bef0 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.183+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bdc1f1350 con 0x7f8bdc06bef0 2024-03-16T19:32:05.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.183+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bdc1f1910 con 0x7f8bdc06bef0 2024-03-16T19:32:05.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.183+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bdc0681c0 con 0x7f8bdc06bef0 2024-03-16T19:32:05.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.184+0000 7f8bd2ffd640 1 -- 172.21.15.172:0/2816495848 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8bc4013470 con 0x7f8bdc06bef0 2024-03-16T19:32:05.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.184+0000 7f8bd2ffd640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bbc06b3b0 0x7f8bbc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.184+0000 7f8bd2ffd640 1 -- 172.21.15.172:0/2816495848 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f8bc4089db0 con 0x7f8bdc06bef0 2024-03-16T19:32:05.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.184+0000 7f8bda59c640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bbc06b3b0 0x7f8bbc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.185+0000 7f8bda59c640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bbc06b3b0 0x7f8bbc06d870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc02b9a0 tx=0x7f8bcc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.188+0000 7f8bd2ffd640 1 -- 172.21.15.172:0/2816495848 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bc40549d0 con 0x7f8bdc06bef0 2024-03-16T19:32:05.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.363+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bdc1f44f0 con 0x7f8bdc06bef0 2024-03-16T19:32:05.345 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.365+0000 7f8bd2ffd640 1 -- 172.21.15.172:0/2816495848 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16790 (secure 0 0 0) 0x7f8bdc1f44f0 con 0x7f8bdc06bef0 2024-03-16T19:32:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:05.345 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":104,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:32:05.055348+0000","last_up_change":"2024-03-16T19:31:48.956759+0000","last_in_change":"2024-03-16T19:31:48.956759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":58,"up_from":88,"up_thru":89,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6825","nonce":3572014952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6827","nonce":3572014952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6831","nonce":3572014952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6829","nonce":3572014952}]},"public_addr":"172.21.15.172:6825/3572014952","cluster_addr":"172.21.15.172:6827/3572014952","heartbeat_back_addr":"172.21.15.172:6831/3572014952","heartbeat_front_addr":"172.21.15.172:6829/3572014952","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bbc06b3b0 msgr2=0x7f8bbc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bbc06b3b0 0x7f8bbc06d870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc02b9a0 tx=0x7f8bcc008030 comp rx=0 tx=0).stop 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 msgr2=0x7f8bdc0d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 0x7f8bdc0d06e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400e820 tx=0x7f8bc400ece0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 shutdown_connections 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bdc0cc260 0x7f8bdc1f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8bbc06b3b0 0x7f8bbc06d870 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bdc06bef0 0x7f8bdc0d06e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 --2- 172.21.15.172:0/2816495848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8bdc06bb10 0x7f8bdc157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 >> 172.21.15.172:0/2816495848 conn(0x7f8bdc067250 msgr2=0x7f8bdc1520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 shutdown_connections 2024-03-16T19:32:05.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.366+0000 7f8be092c640 1 -- 172.21.15.172:0/2816495848 wait complete. 2024-03-16T19:32:05.358 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-16T19:32:05.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:32:05.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.478+0000 7f0371f5c640 1 Processor -- start 2024-03-16T19:32:05.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.478+0000 7f0371f5c640 1 -- start start 2024-03-16T19:32:05.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f0371f5c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f036c154cb0 0x7f036c06dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f0371f5c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f036c06fe10 0x7f036c15de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f0371f5c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036c15e340 con 0x7f036c0c8ba0 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f0371f5c640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036c0cbba0 con 0x7f036c06fe10 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f0371f5c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036c0c91f0 con 0x7f036c154cb0 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f036affd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f036c06fe10 0x7f036c15de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f036bfff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f036c0c8ba0 0x7f036c0cabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:38000/0 (socket says 172.21.15.172:38000) 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f036affd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f036c06fe10 0x7f036c15de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39038/0 (socket says 172.21.15.172:39038) 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f036bfff640 1 -- 172.21.15.172:0/4140300291 learned_addr learned my addr 172.21.15.172:0/4140300291 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f036b7fe640 1 --2- 172.21.15.172:0/4140300291 >> v2:172.21.15.174:3300/0 conn(0x7f036c154cb0 0x7f036c06dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.479+0000 7f036affd640 1 -- 172.21.15.172:0/4140300291 >> v2:172.21.15.174:3300/0 conn(0x7f036c154cb0 msgr2=0x7f036c06dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.480+0000 7f036affd640 1 --2- 172.21.15.172:0/4140300291 >> v2:172.21.15.174:3300/0 conn(0x7f036c154cb0 0x7f036c06dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.480+0000 7f036affd640 1 -- 172.21.15.172:0/4140300291 >> v1:172.21.15.172:6789/0 conn(0x7f036c0c8ba0 legacy=0x7f036c0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.480+0000 7f036a7fc640 1 -- 172.21.15.172:0/4140300291 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842691160 0 0) 0x7f036c15e340 con 0x7f036c0c8ba0 2024-03-16T19:32:05.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.480+0000 7f036affd640 1 -- 172.21.15.172:0/4140300291 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c06c180 con 0x7f036c06fe10 2024-03-16T19:32:05.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.480+0000 7f036affd640 1 --2- 172.21.15.172:0/4140300291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 0x7f036c15de00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f036c0c8860 tx=0x7f036002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fbc6e0758c66958 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.480+0000 7f036a7fc640 1 -- 172.21.15.172:0/4140300291 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360031710 con 0x7f036c06fe10 2024-03-16T19:32:05.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.480+0000 7f036a7fc640 1 -- 172.21.15.172:0/4140300291 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036002f050 con 0x7f036c06fe10 2024-03-16T19:32:05.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.480+0000 7f036a7fc640 1 -- 172.21.15.172:0/4140300291 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360031a90 con 0x7f036c06fe10 2024-03-16T19:32:05.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.481+0000 7f0371f5c640 1 -- 172.21.15.172:0/4140300291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 msgr2=0x7f036c15de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.481+0000 7f0371f5c640 1 --2- 172.21.15.172:0/4140300291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 0x7f036c15de00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f036c0c8860 tx=0x7f036002b7b0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.481+0000 7f0371f5c640 1 -- 172.21.15.172:0/4140300291 shutdown_connections 2024-03-16T19:32:05.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.481+0000 7f0371f5c640 1 --2- 172.21.15.172:0/4140300291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 0x7f036c15de00 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.481+0000 7f0371f5c640 1 --2- 172.21.15.172:0/4140300291 >> v2:172.21.15.174:3300/0 conn(0x7f036c154cb0 0x7f036c06dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.481+0000 7f0371f5c640 1 -- 172.21.15.172:0/4140300291 >> 172.21.15.172:0/4140300291 conn(0x7f036c070860 msgr2=0x7f036c070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.481+0000 7f0371f5c640 1 -- 172.21.15.172:0/4140300291 shutdown_connections 2024-03-16T19:32:05.463 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.481+0000 7f0371f5c640 1 -- 172.21.15.172:0/4140300291 wait complete. 2024-03-16T19:32:05.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.482+0000 7f0371f5c640 1 Processor -- start 2024-03-16T19:32:05.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.482+0000 7f0371f5c640 1 -- start start 2024-03-16T19:32:05.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f0371f5c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 0x7f036c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f0371f5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f036c0c8ba0 0x7f036c155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f0371f5c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c154cb0 0x7f036c155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f0371f5c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c0cc840 con 0x7f036c06fe10 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f0371f5c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c15b980 con 0x7f036c154cb0 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f0371f5c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c14cc90 con 0x7f036c0c8ba0 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 0x7f036c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 0x7f036c1578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39040/0 (socket says 172.21.15.172:39040) 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036b7fe640 1 -- 172.21.15.172:0/2214409981 learned_addr learned my addr 172.21.15.172:0/2214409981 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036affd640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f036c0c8ba0 0x7f036c155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036bfff640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c154cb0 0x7f036c155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036b7fe640 1 -- 172.21.15.172:0/2214409981 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f036c0c8ba0 msgr2=0x7f036c155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036b7fe640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f036c0c8ba0 0x7f036c155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036b7fe640 1 -- 172.21.15.172:0/2214409981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c154cb0 msgr2=0x7f036c155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036b7fe640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c154cb0 0x7f036c155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036b7fe640 1 -- 172.21.15.172:0/2214409981 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360009400 con 0x7f036c06fe10 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.483+0000 7f036affd640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f036c0c8ba0 0x7f036c155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:05.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.484+0000 7f036b7fe640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 0x7f036c1578a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f035c00e870 tx=0x7f035c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.484+0000 7f0353fff640 1 -- 172.21.15.172:0/2214409981 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c0094f0 con 0x7f036c06fe10 2024-03-16T19:32:05.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.484+0000 7f0353fff640 1 -- 172.21.15.172:0/2214409981 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c00c940 con 0x7f036c06fe10 2024-03-16T19:32:05.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.484+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036c14cf70 con 0x7f036c06fe10 2024-03-16T19:32:05.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.484+0000 7f0353fff640 1 -- 172.21.15.172:0/2214409981 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c00ccf0 con 0x7f036c06fe10 2024-03-16T19:32:05.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.484+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c14b6d0 con 0x7f036c06fe10 2024-03-16T19:32:05.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.485+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036c070530 con 0x7f036c06fe10 2024-03-16T19:32:05.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.485+0000 7f0353fff640 1 -- 172.21.15.172:0/2214409981 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f035c013470 con 0x7f036c06fe10 2024-03-16T19:32:05.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.486+0000 7f0353fff640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f035806b3e0 0x7f035806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.486+0000 7f0353fff640 1 -- 172.21.15.172:0/2214409981 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f035c089c70 con 0x7f036c06fe10 2024-03-16T19:32:05.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.486+0000 7f036affd640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f035806b3e0 0x7f035806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.486+0000 7f036affd640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f035806b3e0 0x7f035806d8a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0360035000 tx=0x7f0360008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.490+0000 7f0353fff640 1 -- 172.21.15.172:0/2214409981 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f035c054920 con 0x7f036c06fe10 2024-03-16T19:32:05.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.661+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f036c1581b0 con 0x7f036c06fe10 2024-03-16T19:32:05.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.662+0000 7f0353fff640 1 -- 172.21.15.172:0/2214409981 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16790 (secure 0 0 0) 0x7f035c057fe0 con 0x7f036c06fe10 2024-03-16T19:32:05.643 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:05.643 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":104,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:32:05.055348+0000","last_up_change":"2024-03-16T19:31:48.956759+0000","last_in_change":"2024-03-16T19:31:48.956759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":58,"up_from":88,"up_thru":89,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6825","nonce":3572014952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6827","nonce":3572014952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6831","nonce":3572014952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6829","nonce":3572014952}]},"public_addr":"172.21.15.172:6825/3572014952","cluster_addr":"172.21.15.172:6827/3572014952","heartbeat_back_addr":"172.21.15.172:6831/3572014952","heartbeat_front_addr":"172.21.15.172:6829/3572014952","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f035806b3e0 msgr2=0x7f035806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f035806b3e0 0x7f035806d8a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0360035000 tx=0x7f0360008030 comp rx=0 tx=0).stop 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 msgr2=0x7f036c1578a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 0x7f036c1578a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f035c00e870 tx=0x7f035c00ed30 comp rx=0 tx=0).stop 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 shutdown_connections 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c154cb0 0x7f036c155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f035806b3e0 0x7f035806d8a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f036c0c8ba0 0x7f036c155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 --2- 172.21.15.172:0/2214409981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f036c06fe10 0x7f036c1578a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 >> 172.21.15.172:0/2214409981 conn(0x7f036c070860 msgr2=0x7f036c152a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 shutdown_connections 2024-03-16T19:32:05.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.664+0000 7f0371f5c640 1 -- 172.21.15.172:0/2214409981 wait complete. 2024-03-16T19:32:05.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-16T19:32:05.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-16T19:32:05.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-16T19:32:05.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-16T19:32:05.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-16T19:32:05.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-16T19:32:05.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-16T19:32:05.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-16T19:32:05.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.849+0000 7f538b59e640 1 Processor -- start 2024-03-16T19:32:05.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.849+0000 7f538b59e640 1 -- start start 2024-03-16T19:32:05.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.850+0000 7f5fabb5c640 1 Processor -- start 2024-03-16T19:32:05.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.851+0000 7f538b59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 0x7f538c15a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.851+0000 7f5fabb5c640 1 -- start start 2024-03-16T19:32:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.851+0000 7ff29d014640 1 Processor -- start 2024-03-16T19:32:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.851+0000 7f538b59e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c0c6f20 con 0x7f538c153ce0 2024-03-16T19:32:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.851+0000 7f538b59e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c0c8290 con 0x7f538c06d5c0 2024-03-16T19:32:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.851+0000 7f538b59e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c0c8080 con 0x7f538c06ebf0 2024-03-16T19:32:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7f5389d9b640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f538c06d5c0 0x7f538c071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:56594/0 (socket says 172.21.15.172:56594) 2024-03-16T19:32:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7f5389d9b640 1 -- 172.21.15.172:0/2389758207 learned_addr learned my addr 172.21.15.172:0/2389758207 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7f538a59c640 1 --2- 172.21.15.172:0/2389758207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 0x7f538c15a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7f538a59c640 1 -- 172.21.15.172:0/2389758207 >> v1:172.21.15.174:6789/0 conn(0x7f538c06d5c0 legacy=0x7f538c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7f538a59c640 1 -- 172.21.15.172:0/2389758207 >> v1:172.21.15.172:6789/0 conn(0x7f538c153ce0 legacy=0x7f538c0c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7f538a59c640 1 -- 172.21.15.172:0/2389758207 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c0c6bb0 con 0x7f538c06ebf0 2024-03-16T19:32:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.851+0000 7ff29d014640 1 -- start start 2024-03-16T19:32:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7ff3181a1640 1 Processor -- start 2024-03-16T19:32:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.851+0000 7fe5ab73e640 1 Processor -- start 2024-03-16T19:32:05.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.851+0000 7fe5ab73e640 1 -- start start 2024-03-16T19:32:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7f31d27d7640 1 Processor -- start 2024-03-16T19:32:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7fb2ee402640 1 Processor -- start 2024-03-16T19:32:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7f31d27d7640 1 -- start start 2024-03-16T19:32:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31d27d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc158d20 0x7f31cc0c1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31d27d7640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f31cc06d410 0x7f31cc06c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31d27d7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f31cc159100 0x7f31cc06cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31d27d7640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31cc0c7db0 con 0x7f31cc06d410 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31d27d7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31cc0c8050 con 0x7f31cc159100 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31d27d7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0c82f0 con 0x7f31cc158d20 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31cb7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f31cc06d410 0x7f31cc06c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff29d014640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff290006f30 0x7ff2900e33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff29d014640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff29004c170 0x7ff29004c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff29d014640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29004bce0 con 0x7ff2900e6920 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff29d014640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29004cb40 con 0x7ff29004c170 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff29d014640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29004b9b0 con 0x7ff290006f30 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff2977fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7ff2900e6920 0x7ff290007310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:38026/0 (socket says 172.21.15.172:38026) 2024-03-16T19:32:05.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff2977fe640 1 -- 172.21.15.172:0/3870179966 learned_addr learned my addr 172.21.15.172:0/3870179966 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff297fff640 1 --2- 172.21.15.172:0/3870179966 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff290006f30 0x7ff2900e33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff29c813640 1 --2- 172.21.15.172:0/3870179966 >> v2:172.21.15.174:3300/0 conn(0x7ff29004c170 0x7ff29004c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff297fff640 1 -- 172.21.15.172:0/3870179966 >> v2:172.21.15.174:3300/0 conn(0x7ff29004c170 msgr2=0x7ff29004c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff297fff640 1 --2- 172.21.15.172:0/3870179966 >> v2:172.21.15.174:3300/0 conn(0x7ff29004c170 0x7ff29004c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff297fff640 1 -- 172.21.15.172:0/3870179966 >> v1:172.21.15.172:6789/0 conn(0x7ff2900e6920 legacy=0x7ff290007310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff297fff640 1 -- 172.21.15.172:0/3870179966 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff290007a20 con 0x7ff290006f30 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7f5fabb5c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5fa4150cc0 0x7f5fa40c8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7f5fabb5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa40c6a30 0x7f5fa40c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7f5faa0f9640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5fa406aed0 0x7f5fa4157050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:38024/0 (socket says 172.21.15.172:38024) 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7f5faa0f9640 1 -- 172.21.15.172:0/3901768637 learned_addr learned my addr 172.21.15.172:0/3901768637 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7f5fa90f7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa40c6a30 0x7f5fa40c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3901768637 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa4156bc0 con 0x7f5fa406aed0 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3901768637 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa406c070 con 0x7f5fa4150cc0 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3901768637 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa41510a0 con 0x7f5fa40c6a30 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f5fa88f6640 1 -- 172.21.15.172:0/3901768637 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394180972 0 0) 0x7f5fa4156bc0 con 0x7f5fa406aed0 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f5fa90f7640 1 -- 172.21.15.172:0/3901768637 >> v2:172.21.15.172:3300/0 conn(0x7f5fa4150cc0 msgr2=0x7f5fa40c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f5fa90f7640 1 --2- 172.21.15.172:0/3901768637 >> v2:172.21.15.172:3300/0 conn(0x7f5fa4150cc0 0x7f5fa40c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f5fa90f7640 1 -- 172.21.15.172:0/3901768637 >> v1:172.21.15.172:6789/0 conn(0x7f5fa406aed0 legacy=0x7f5fa4157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f5fa90f7640 1 -- 172.21.15.172:0/3901768637 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa415ab30 con 0x7f5fa40c6a30 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.852+0000 7ff3181a1640 1 -- start start 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff3181a1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff310159110 0x7ff310162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff3181a1640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7ff31015ed80 0x7ff31006ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff3181a1640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31006b7d0 con 0x7ff310073180 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff3181a1640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3100c2c10 con 0x7ff31015ed80 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff3181a1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31015fdc0 con 0x7ff310159110 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff315f3d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff310159110 0x7ff310162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7ff315f3d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff310159110 0x7ff310162ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40554/0 (socket says 172.21.15.172:40554) 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff315f3d640 1 -- 172.21.15.172:0/3779586394 learned_addr learned my addr 172.21.15.172:0/3779586394 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff31573c640 1 --2- 172.21.15.172:0/3779586394 >> v2:172.21.15.172:3300/0 conn(0x7ff31015ed80 0x7ff31006ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7fe5ab73e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe59c04c1e0 0x7fe59c0dfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7fe5ab73e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59c009030 con 0x7fe59c04d400 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7fe5ab73e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59c0d61e0 con 0x7fe59c0d5e00 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7fe5ab73e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59c046b10 con 0x7fe59c04c1e0 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7fe5aa4dc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fe59c04d400 0x7fe59c006ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:38018/0 (socket says 172.21.15.172:38018) 2024-03-16T19:32:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7fe5aa4dc640 1 -- 172.21.15.172:0/3292466757 learned_addr learned my addr 172.21.15.172:0/3292466757 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:05.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7fe5a8cd9640 1 --2- 172.21.15.172:0/3292466757 >> v2:172.21.15.174:3300/0 conn(0x7fe59c04c1e0 0x7fe59c0dfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7fe59bfff640 1 -- 172.21.15.172:0/3292466757 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935145367 0 0) 0x7fe59c0d61e0 con 0x7fe59c0d5e00 2024-03-16T19:32:05.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7fe59bfff640 1 -- 172.21.15.172:0/3292466757 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe590003520 con 0x7fe59c0d5e00 2024-03-16T19:32:05.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.853+0000 7fb2ee402640 1 -- start start 2024-03-16T19:32:05.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7fb2ee402640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb2e00dd500 0x7fb2e0046260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31cb7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f31cc06d410 0x7f31cc06c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39064/0 (socket says 172.21.15.172:39064) 2024-03-16T19:32:05.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31cb7fe640 1 -- 172.21.15.172:0/291179467 learned_addr learned my addr 172.21.15.172:0/291179467 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f31d0d74640 1 --2- 172.21.15.172:0/291179467 >> v2:172.21.15.174:3300/0 conn(0x7f31cc159100 0x7f31cc06cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7fe59bfff640 1 -- 172.21.15.172:0/3292466757 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104139933 0 0) 0x7fe59c009030 con 0x7fe59c04d400 2024-03-16T19:32:05.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7fe59bfff640 1 -- 172.21.15.172:0/3292466757 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe59c0d61e0 con 0x7fe59c04d400 2024-03-16T19:32:05.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2ee402640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb2e00473b0 0x7fb2e00e33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7f538a59c640 1 --2- 172.21.15.172:0/2389758207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 0x7f538c15a7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5384009dc0 tx=0x7f538402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44808d1b955f2950 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f31cb7fe640 1 -- 172.21.15.172:0/291179467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc158d20 msgr2=0x7f31cc0c1cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:32:05.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7ff29c813640 1 --2- 172.21.15.172:0/3870179966 >> v2:172.21.15.174:3300/0 conn(0x7ff29004c170 0x7ff29004c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:05.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2ee402640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e00e69f0 con 0x7fb2e004d8d0 2024-03-16T19:32:05.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2ee402640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e0036dc0 con 0x7fb2e00dd500 2024-03-16T19:32:05.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2ee402640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e00e6cd0 con 0x7fb2e00473b0 2024-03-16T19:32:05.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2ecbff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fb2e004d8d0 0x7fb2e004df70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:56612/0 (socket says 172.21.15.172:56612) 2024-03-16T19:32:05.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2ecbff640 1 -- 172.21.15.172:0/4197228985 learned_addr learned my addr 172.21.15.172:0/4197228985 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:05.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2ed400640 1 --2- 172.21.15.172:0/4197228985 >> v2:172.21.15.172:3300/0 conn(0x7fb2e00dd500 0x7fb2e0046260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7ff297fff640 1 --2- 172.21.15.172:0/3870179966 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff290006f30 0x7ff2900e33b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0097a0 tx=0x7ff28c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c2e1666fac9efcd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f31cb7fe640 1 --2- 172.21.15.172:0/291179467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc158d20 0x7f31cc0c1cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f31cb7fe640 1 -- 172.21.15.172:0/291179467 >> v2:172.21.15.174:3300/0 conn(0x7f31cc159100 msgr2=0x7f31cc06cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f538959a640 1 -- 172.21.15.172:0/2389758207 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538402cc40 con 0x7f538c06ebf0 2024-03-16T19:32:05.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f31cb7fe640 1 --2- 172.21.15.172:0/291179467 >> v2:172.21.15.174:3300/0 conn(0x7f31cc159100 0x7f31cc06cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322444575 0 0) 0x7fb2e00e69f0 con 0x7fb2e004d8d0 2024-03-16T19:32:05.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d4003520 con 0x7fb2e004d8d0 2024-03-16T19:32:05.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f31cb7fe640 1 -- 172.21.15.172:0/291179467 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc15eed0 con 0x7f31cc06d410 2024-03-16T19:32:05.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f538959a640 1 -- 172.21.15.172:0/2389758207 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538402cda0 con 0x7f538c06ebf0 2024-03-16T19:32:05.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f31d0d74640 1 --2- 172.21.15.172:0/291179467 >> v2:172.21.15.174:3300/0 conn(0x7f31cc159100 0x7f31cc06cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:05.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2edc01640 1 --2- 172.21.15.172:0/4197228985 >> v2:172.21.15.174:3300/0 conn(0x7fb2e00473b0 0x7fb2e00e33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f538959a640 1 -- 172.21.15.172:0/2389758207 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384034660 con 0x7f538c06ebf0 2024-03-16T19:32:05.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fe5a8cd9640 1 -- 172.21.15.172:0/3292466757 >> v1:172.21.15.174:6789/0 conn(0x7fe59c0d5e00 legacy=0x7fe59c046400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fe5a8cd9640 1 -- 172.21.15.172:0/3292466757 >> v1:172.21.15.172:6789/0 conn(0x7fe59c04d400 legacy=0x7fe59c006ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fe5a8cd9640 1 -- 172.21.15.172:0/3292466757 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c0dff50 con 0x7fe59c04c1e0 2024-03-16T19:32:05.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff31573c640 1 -- 172.21.15.172:0/3779586394 >> v2:172.21.15.174:3300/0 conn(0x7ff310159110 msgr2=0x7ff310162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff31573c640 1 --2- 172.21.15.172:0/3779586394 >> v2:172.21.15.174:3300/0 conn(0x7ff310159110 0x7ff310162ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff31573c640 1 -- 172.21.15.172:0/3779586394 >> v1:172.21.15.172:6789/0 conn(0x7ff310073180 legacy=0x7ff3100c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.854+0000 7ff31573c640 1 -- 172.21.15.172:0/3779586394 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3101594f0 con 0x7ff31015ed80 2024-03-16T19:32:05.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7ff296ffd640 1 -- 172.21.15.172:0/3870179966 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c030840 con 0x7ff290006f30 2024-03-16T19:32:05.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158829160 0 0) 0x7fb2d4003520 con 0x7fb2e004d8d0 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 >> v2:172.21.15.174:3300/0 conn(0x7fb2e00473b0 msgr2=0x7fb2e00e33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 --2- 172.21.15.172:0/4197228985 >> v2:172.21.15.174:3300/0 conn(0x7fb2e00473b0 0x7fb2e00e33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 >> v2:172.21.15.172:3300/0 conn(0x7fb2e00dd500 msgr2=0x7fb2e0046260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 --2- 172.21.15.172:0/4197228985 >> v2:172.21.15.172:3300/0 conn(0x7fb2e00dd500 0x7fb2e0046260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e00e6fb0 con 0x7fb2e004d8d0 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7fb2e4001670 con 0x7fb2e004d8d0 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 >> v1:172.21.15.174:6789/0 conn(0x7fb2e004d8d0 legacy=0x7fb2e004df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d4003970 0x7fb2d4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d40047c0 0x7fb2d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d4005180 0x7fb2d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e0036dc0 con 0x7fb2d40047c0 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e00e6cd0 con 0x7fb2d4005180 2024-03-16T19:32:05.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d400c130 con 0x7fb2d4003970 2024-03-16T19:32:05.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7ff296ffd640 1 -- 172.21.15.172:0/3870179966 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28c0309a0 con 0x7ff290006f30 2024-03-16T19:32:05.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7ff296ffd640 1 -- 172.21.15.172:0/3870179966 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c030d10 con 0x7ff290006f30 2024-03-16T19:32:05.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.855+0000 7f5fa90f7640 1 --2- 172.21.15.172:0/3901768637 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa40c6a30 0x7f5fa40c7c80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400b160 tx=0x7f5f9402c1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=760c239255b8012e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7fb2ecbff640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d40047c0 0x7fb2d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7fb2edc01640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d4005180 0x7fb2d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7fb2ed400640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d4003970 0x7fb2d4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7f51dce21640 1 Processor -- start 2024-03-16T19:32:05.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7f51dce21640 1 -- start start 2024-03-16T19:32:05.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7ff31573c640 1 --2- 172.21.15.172:0/3779586394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff31015ed80 0x7ff31006ba10 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff3100c1100 tx=0x7ff30802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a58725f2019b3d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7f538b59e640 1 -- 172.21.15.172:0/2389758207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 msgr2=0x7f538c15a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7f538b59e640 1 --2- 172.21.15.172:0/2389758207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 0x7f538c15a7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5384009dc0 tx=0x7f538402be90 comp rx=0 tx=0).stop 2024-03-16T19:32:05.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7f5fa88f6640 1 -- 172.21.15.172:0/3901768637 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94034440 con 0x7f5fa40c6a30 2024-03-16T19:32:05.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7f5fa88f6640 1 -- 172.21.15.172:0/3901768637 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f940345a0 con 0x7f5fa40c6a30 2024-03-16T19:32:05.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7f5fa88f6640 1 -- 172.21.15.172:0/3901768637 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94034930 con 0x7f5fa40c6a30 2024-03-16T19:32:05.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7f31cb7fe640 1 --2- 172.21.15.172:0/291179467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31cc06d410 0x7f31cc06c610 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009850 tx=0x7f31c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effb0ffcd7c4de6e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f31caffd640 1 -- 172.21.15.172:0/291179467 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c0004230 con 0x7f31cc06d410 2024-03-16T19:32:05.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7fe5a8cd9640 1 --2- 172.21.15.172:0/3292466757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe59c04c1e0 0x7fe59c0dfa10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0e00e0 tx=0x7fe5a40a7bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf48e60938cc7eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff314f3b640 1 -- 172.21.15.172:0/3779586394 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30802cbd0 con 0x7ff31015ed80 2024-03-16T19:32:05.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff314f3b640 1 -- 172.21.15.172:0/3779586394 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30802cd30 con 0x7ff31015ed80 2024-03-16T19:32:05.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff314f3b640 1 -- 172.21.15.172:0/3779586394 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30802cea0 con 0x7ff31015ed80 2024-03-16T19:32:05.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff29d014640 1 -- 172.21.15.172:0/3870179966 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff290006f30 msgr2=0x7ff2900e33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff29d014640 1 --2- 172.21.15.172:0/3870179966 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff290006f30 0x7ff2900e33b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0097a0 tx=0x7ff28c02b980 comp rx=0 tx=0).stop 2024-03-16T19:32:05.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f31caffd640 1 -- 172.21.15.172:0/291179467 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c0004390 con 0x7f31cc06d410 2024-03-16T19:32:05.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff29d014640 1 -- 172.21.15.172:0/3870179966 shutdown_connections 2024-03-16T19:32:05.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff29d014640 1 --2- 172.21.15.172:0/3870179966 >> v2:172.21.15.174:3300/0 conn(0x7ff29004c170 0x7ff29004c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff29d014640 1 --2- 172.21.15.172:0/3870179966 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff290006f30 0x7ff2900e33b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff29d014640 1 -- 172.21.15.172:0/3870179966 >> 172.21.15.172:0/3870179966 conn(0x7ff2900061a0 msgr2=0x7ff2900065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff29d014640 1 -- 172.21.15.172:0/3870179966 shutdown_connections 2024-03-16T19:32:05.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f31caffd640 1 -- 172.21.15.172:0/291179467 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c0003810 con 0x7f31cc06d410 2024-03-16T19:32:05.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7ff29d014640 1 -- 172.21.15.172:0/3870179966 wait complete. 2024-03-16T19:32:05.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7fb2ed400640 1 -- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d4005180 msgr2=0x7fb2d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7fb2ed400640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d4005180 0x7fb2d400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7fb2ed400640 1 -- 172.21.15.172:0/4197228985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d40047c0 msgr2=0x7fb2d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7fb2ed400640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2d40047c0 0x7fb2d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.856+0000 7fb2ed400640 1 -- 172.21.15.172:0/4197228985 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e00e6fb0 con 0x7fb2d4003970 2024-03-16T19:32:05.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f51dce21640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f51d8158fe0 0x7f51d8162ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7fb2edc01640 1 -- 172.21.15.172:0/4197228985 reap_dead start 2024-03-16T19:32:05.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3901768637 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa40c6a30 msgr2=0x7f5fa40c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f538b59e640 1 -- 172.21.15.172:0/2389758207 shutdown_connections 2024-03-16T19:32:05.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f538b59e640 1 --2- 172.21.15.172:0/2389758207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 0x7f538c15a7c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f538b59e640 1 -- 172.21.15.172:0/2389758207 >> 172.21.15.172:0/2389758207 conn(0x7f538c06e0d0 msgr2=0x7f538c06e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f5fabb5c640 1 --2- 172.21.15.172:0/3901768637 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa40c6a30 0x7f5fa40c7c80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400b160 tx=0x7f5f9402c1d0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7fb2ed400640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d4003970 0x7fb2d4003d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8000b60 tx=0x7fb2d40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6275f720d390be22 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f538b59e640 1 -- 172.21.15.172:0/2389758207 shutdown_connections 2024-03-16T19:32:05.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f51dce21640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d80b43b0 con 0x7f51d80d0800 2024-03-16T19:32:05.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7fe59bfff640 1 -- 172.21.15.172:0/3292466757 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40a8a30 con 0x7fe59c04c1e0 2024-03-16T19:32:05.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7fe59bfff640 1 -- 172.21.15.172:0/3292466757 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a40a8b90 con 0x7fe59c04c1e0 2024-03-16T19:32:05.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f51dce21640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d80b4690 con 0x7f51d80cb120 2024-03-16T19:32:05.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f51dce21640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80b4970 con 0x7f51d8158fe0 2024-03-16T19:32:05.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7fe59bfff640 1 -- 172.21.15.172:0/3292466757 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40b0440 con 0x7fe59c04c1e0 2024-03-16T19:32:05.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f31d27d7640 1 -- 172.21.15.172:0/291179467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31cc06d410 msgr2=0x7f31cc06c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f31d27d7640 1 --2- 172.21.15.172:0/291179467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31cc06d410 0x7f31cc06c610 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009850 tx=0x7f31c002b8f0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3901768637 shutdown_connections 2024-03-16T19:32:05.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f5fabb5c640 1 --2- 172.21.15.172:0/3901768637 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa40c6a30 0x7f5fa40c7c80 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400b160 tx=0x7f5f9402c1d0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f5fabb5c640 1 --2- 172.21.15.172:0/3901768637 >> v2:172.21.15.172:3300/0 conn(0x7f5fa4150cc0 0x7f5fa40c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f51d77fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f51d8158fe0 0x7f51d8162ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3901768637 >> 172.21.15.172:0/3901768637 conn(0x7f5fa40cc030 msgr2=0x7f5fa40cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e80a9700 con 0x7fb2d4003970 2024-03-16T19:32:05.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e80a9860 con 0x7fb2d4003970 2024-03-16T19:32:05.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7fb2dffff640 1 -- 172.21.15.172:0/4197228985 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e80a9c10 con 0x7fb2d4003970 2024-03-16T19:32:05.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f51d77fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f51d8158fe0 0x7f51d8162ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39086/0 (socket says 172.21.15.172:39086) 2024-03-16T19:32:05.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f51d77fe640 1 -- 172.21.15.172:0/3748309500 learned_addr learned my addr 172.21.15.172:0/3748309500 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f31d27d7640 1 -- 172.21.15.172:0/291179467 shutdown_connections 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f31d27d7640 1 --2- 172.21.15.172:0/291179467 >> v2:172.21.15.174:3300/0 conn(0x7f31cc159100 0x7f31cc06cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f31d27d7640 1 --2- 172.21.15.172:0/291179467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31cc06d410 0x7f31cc06c610 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f31d27d7640 1 --2- 172.21.15.172:0/291179467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc158d20 0x7f31cc0c1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f31d27d7640 1 -- 172.21.15.172:0/291179467 >> 172.21.15.172:0/291179467 conn(0x7f31cc06e0d0 msgr2=0x7f31cc06e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7ff3181a1640 1 -- 172.21.15.172:0/3779586394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff31015ed80 msgr2=0x7ff31006ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7ff3181a1640 1 --2- 172.21.15.172:0/3779586394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff31015ed80 0x7ff31006ba10 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff3100c1100 tx=0x7ff30802be20 comp rx=0 tx=0).stop 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7ff3181a1640 1 -- 172.21.15.172:0/3779586394 shutdown_connections 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7ff3181a1640 1 --2- 172.21.15.172:0/3779586394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff31015ed80 0x7ff31006ba10 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7ff3181a1640 1 --2- 172.21.15.172:0/3779586394 >> v2:172.21.15.174:3300/0 conn(0x7ff310159110 0x7ff310162ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7ff3181a1640 1 -- 172.21.15.172:0/3779586394 >> 172.21.15.172:0/3779586394 conn(0x7ff3100cc030 msgr2=0x7ff3100cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.857+0000 7f538b59e640 1 -- 172.21.15.172:0/2389758207 wait complete. 2024-03-16T19:32:05.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3292466757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe59c04c1e0 msgr2=0x7fe59c0dfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7fe5ab73e640 1 --2- 172.21.15.172:0/3292466757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe59c04c1e0 0x7fe59c0dfa10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0e00e0 tx=0x7fe5a40a7bc0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f51d67fc640 1 -- 172.21.15.172:0/3748309500 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886159761 0 0) 0x7f51d80b43b0 con 0x7f51d80d0800 2024-03-16T19:32:05.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f51d77fe640 1 -- 172.21.15.172:0/3748309500 >> v1:172.21.15.174:6789/0 conn(0x7f51d80cb120 legacy=0x7f51d80d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:05.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3901768637 shutdown_connections 2024-03-16T19:32:05.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f51d77fe640 1 -- 172.21.15.172:0/3748309500 >> v1:172.21.15.172:6789/0 conn(0x7f51d80d0800 legacy=0x7f51d8169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3901768637 wait complete. 2024-03-16T19:32:05.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.858+0000 7f51d77fe640 1 -- 172.21.15.172:0/3748309500 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d80b4c50 con 0x7f51d8158fe0 2024-03-16T19:32:05.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7ff29d014640 1 Processor -- start 2024-03-16T19:32:05.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7ff3181a1640 1 -- 172.21.15.172:0/3779586394 shutdown_connections 2024-03-16T19:32:05.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7ff3181a1640 1 -- 172.21.15.172:0/3779586394 wait complete. 2024-03-16T19:32:05.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7f31d27d7640 1 -- 172.21.15.172:0/291179467 shutdown_connections 2024-03-16T19:32:05.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3292466757 shutdown_connections 2024-03-16T19:32:05.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fe5ab73e640 1 --2- 172.21.15.172:0/3292466757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe59c04c1e0 0x7fe59c0dfa10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3292466757 >> 172.21.15.172:0/3292466757 conn(0x7fe59c051460 msgr2=0x7fe59c051870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fb2ee402640 1 -- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d4003970 msgr2=0x7fb2d4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fb2ee402640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d4003970 0x7fb2d4003d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8000b60 tx=0x7fb2d40057f0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3292466757 shutdown_connections 2024-03-16T19:32:05.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fb2ee402640 1 -- 172.21.15.172:0/4197228985 shutdown_connections 2024-03-16T19:32:05.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fb2ee402640 1 --2- 172.21.15.172:0/4197228985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2d4003970 0x7fb2d4003d90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fb2ee402640 1 -- 172.21.15.172:0/4197228985 >> 172.21.15.172:0/4197228985 conn(0x7fb2e000d190 msgr2=0x7fb2e000d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7ff29d014640 1 -- start start 2024-03-16T19:32:05.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fb2ee402640 1 -- 172.21.15.172:0/4197228985 shutdown_connections 2024-03-16T19:32:05.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fb2ee402640 1 -- 172.21.15.172:0/4197228985 wait complete. 2024-03-16T19:32:05.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7f538b59e640 1 Processor -- start 2024-03-16T19:32:05.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7f538b59e640 1 -- start start 2024-03-16T19:32:05.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.860+0000 7f31d27d7640 1 -- 172.21.15.172:0/291179467 wait complete. 2024-03-16T19:32:05.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.859+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3292466757 wait complete. 2024-03-16T19:32:05.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.860+0000 7ff29d014640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290006f30 0x7ff2900e6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.860+0000 7ff3181a1640 1 Processor -- start 2024-03-16T19:32:05.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.860+0000 7f5fabb5c640 1 Processor -- start 2024-03-16T19:32:05.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff29d014640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff29004c170 0x7ff2900df6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7f5fabb5c640 1 -- start start 2024-03-16T19:32:05.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff3181a1640 1 -- start start 2024-03-16T19:32:05.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7f538b59e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f538c06d5c0 0x7f538c0c2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff297fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290006f30 0x7ff2900e6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff297fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290006f30 0x7ff2900e6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40598/0 (socket says 172.21.15.172:40598) 2024-03-16T19:32:05.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7fe5ab73e640 1 Processor -- start 2024-03-16T19:32:05.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff297fff640 1 -- 172.21.15.172:0/155708863 learned_addr learned my addr 172.21.15.172:0/155708863 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff29d014640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2900e6920 0x7ff2900de220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff290185530 con 0x7ff29004c170 2024-03-16T19:32:05.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2900ddd90 con 0x7ff290006f30 2024-03-16T19:32:05.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2900e0250 con 0x7ff2900e6920 2024-03-16T19:32:05.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7ff2977fe640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff29004c170 0x7ff2900df6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7fb2ee402640 1 Processor -- start 2024-03-16T19:32:05.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7f31d27d7640 1 Processor -- start 2024-03-16T19:32:05.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7fb2ee402640 1 -- start start 2024-03-16T19:32:05.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.860+0000 7f51d77fe640 1 --2- 172.21.15.172:0/3748309500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d8158fe0 0x7f51d8162ba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009870 tx=0x7f51c802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d6c449a08a18aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.860+0000 7f51d67fc640 1 -- 172.21.15.172:0/3748309500 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c802f840 con 0x7f51d8158fe0 2024-03-16T19:32:05.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7f51d67fc640 1 -- 172.21.15.172:0/3748309500 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c802f9a0 con 0x7f51d8158fe0 2024-03-16T19:32:05.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7f538b59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 0x7f538c0c2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7f538b59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f538c153ce0 0x7f538c13e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f5fabb5c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa406aed0 0x7f5fa41597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff29c813640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2900e6920 0x7ff2900de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7fe5ab73e640 1 -- start start 2024-03-16T19:32:05.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff297fff640 1 -- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2900e6920 msgr2=0x7ff2900de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff297fff640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2900e6920 0x7ff2900de220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7fe5ab73e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe59c04c1e0 0x7fe59c050a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff297fff640 1 -- 172.21.15.172:0/155708863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff29004c170 msgr2=0x7ff2900df6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff297fff640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff29004c170 0x7ff2900df6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff297fff640 1 -- 172.21.15.172:0/155708863 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c009400 con 0x7ff290006f30 2024-03-16T19:32:05.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff2977fe640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff29004c170 0x7ff2900df6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:05.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7fe5ab73e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe59c04d400 0x7fe59c0368f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f5fa98f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa406aed0 0x7f5fa41597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f5fa98f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa406aed0 0x7f5fa41597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40622/0 (socket says 172.21.15.172:40622) 2024-03-16T19:32:05.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f5fabb5c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa4150cc0 0x7f5fa4146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f5fa98f8640 1 -- 172.21.15.172:0/3917106129 learned_addr learned my addr 172.21.15.172:0/3917106129 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7f538b59e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c1557f0 con 0x7f538c06d5c0 2024-03-16T19:32:05.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f538a59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f538c06d5c0 0x7f538c0c2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f538a59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f538c06d5c0 0x7f538c0c2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39096/0 (socket says 172.21.15.172:39096) 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f538a59c640 1 -- 172.21.15.172:0/1410470205 learned_addr learned my addr 172.21.15.172:0/1410470205 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff3181a1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff310073180 0x7ff3100c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff3181a1640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff310159110 0x7ff310148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff3181a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31015ed80 0x7ff310146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff3181a1640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3100cccd0 con 0x7ff310159110 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff3181a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31014b590 con 0x7ff31015ed80 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff3181a1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31014b830 con 0x7ff310073180 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c155b20 con 0x7f538c153ce0 2024-03-16T19:32:05.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c1545f0 con 0x7f538c06ebf0 2024-03-16T19:32:05.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f538ad9d640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f538c153ce0 0x7f538c13e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7fe5ab73e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe59c0d5e00 0x7fe59c036e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f5389d9b640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 0x7f538c0c2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7fe5ab73e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c036460 con 0x7fe59c04d400 2024-03-16T19:32:05.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7fe5ab73e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0dcbd0 con 0x7fe59c0d5e00 2024-03-16T19:32:05.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7fe5ab73e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0d4280 con 0x7fe59c04c1e0 2024-03-16T19:32:05.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f5fabb5c640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fa4156710 0x7f5fa41ebf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f5fa90f7640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa4150cc0 0x7f5fa4146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff31673e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31015ed80 0x7ff310146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff31673e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31015ed80 0x7ff310146860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40630/0 (socket says 172.21.15.172:40630) 2024-03-16T19:32:05.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7ff31673e640 1 -- 172.21.15.172:0/4203680405 learned_addr learned my addr 172.21.15.172:0/4203680405 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7f51dce21640 1 -- 172.21.15.172:0/3748309500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d8158fe0 msgr2=0x7f51d8162ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.861+0000 7f51dce21640 1 --2- 172.21.15.172:0/3748309500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d8158fe0 0x7f51d8162ba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009870 tx=0x7f51c802ba20 comp rx=0 tx=0).stop 2024-03-16T19:32:05.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f51dce21640 1 -- 172.21.15.172:0/3748309500 shutdown_connections 2024-03-16T19:32:05.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f51dce21640 1 --2- 172.21.15.172:0/3748309500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d8158fe0 0x7f51d8162ba0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f51dce21640 1 -- 172.21.15.172:0/3748309500 >> 172.21.15.172:0/3748309500 conn(0x7f51d806ca20 msgr2=0x7f51d8066c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f31d27d7640 1 -- start start 2024-03-16T19:32:05.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.862+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa41472f0 con 0x7f5fa4156710 2024-03-16T19:32:05.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa41525e0 con 0x7f5fa406aed0 2024-03-16T19:32:05.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f5faa0f9640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fa4156710 0x7f5fa41ebf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa40c3430 con 0x7f5fa4150cc0 2024-03-16T19:32:05.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff31573c640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff310159110 0x7ff310148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fe5a8cd9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe59c04d400 0x7fe59c0368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fe5a8cd9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe59c04d400 0x7fe59c0368f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39104/0 (socket says 172.21.15.172:39104) 2024-03-16T19:32:05.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fe5a8cd9640 1 -- 172.21.15.172:0/3017947549 learned_addr learned my addr 172.21.15.172:0/3017947549 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fb2ee402640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d4003970 0x7fb2e00e1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f31d27d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc06d410 0x7f31cc150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f31d27d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31cc158d20 0x7f31cc14dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f31d27d7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31cc14e180 0x7f31cc0cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f31d27d7640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc14d280 con 0x7f31cc14e180 2024-03-16T19:32:05.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f31d27d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc152f00 con 0x7f31cc158d20 2024-03-16T19:32:05.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f31d27d7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc14d7b0 con 0x7f31cc06d410 2024-03-16T19:32:05.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff31673e640 1 -- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff310073180 msgr2=0x7ff3100c5f00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T19:32:05.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff31673e640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff310073180 0x7ff3100c5f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff31673e640 1 -- 172.21.15.172:0/4203680405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff310159110 msgr2=0x7ff310148af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff31673e640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff310159110 0x7ff310148af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff31673e640 1 -- 172.21.15.172:0/4203680405 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff308009a70 con 0x7ff31015ed80 2024-03-16T19:32:05.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff297fff640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290006f30 0x7ff2900e6020 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0098c0 tx=0x7ff28c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff31573c640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff310159110 0x7ff310148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:05.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fb2ee402640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2e00dfca0 0x7fb2e00e02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f51dce21640 1 -- 172.21.15.172:0/3748309500 shutdown_connections 2024-03-16T19:32:05.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff31673e640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31015ed80 0x7ff310146860 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c3c0 tx=0x7ff30000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f51dce21640 1 -- 172.21.15.172:0/3748309500 wait complete. 2024-03-16T19:32:05.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fb2ee402640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2e0170ad0 0x7fb2e0170f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f538a59c640 1 -- 172.21.15.172:0/1410470205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 msgr2=0x7f538c0c2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f538a59c640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538c06ebf0 0x7f538c0c2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f538a59c640 1 -- 172.21.15.172:0/1410470205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f538c153ce0 msgr2=0x7f538c13e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f538a59c640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f538c153ce0 0x7f538c13e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f538a59c640 1 -- 172.21.15.172:0/1410470205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384009a70 con 0x7f538c06d5c0 2024-03-16T19:32:05.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fe5a8cd9640 1 -- 172.21.15.172:0/3017947549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe59c04c1e0 msgr2=0x7fe59c050a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T19:32:05.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fe5a8cd9640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe59c04c1e0 0x7fe59c050a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fe5a8cd9640 1 -- 172.21.15.172:0/3017947549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe59c0d5e00 msgr2=0x7fe59c036e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fe5a8cd9640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe59c0d5e00 0x7fe59c036e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fe5a8cd9640 1 -- 172.21.15.172:0/3017947549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4073d10 con 0x7fe59c04d400 2024-03-16T19:32:05.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fb2ecbff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2e00dfca0 0x7fb2e00e02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff294ff9640 1 -- 172.21.15.172:0/155708863 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c03e070 con 0x7ff290006f30 2024-03-16T19:32:05.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fb2ed400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d4003970 0x7fb2e00e1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fb2ed400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d4003970 0x7fb2e00e1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40662/0 (socket says 172.21.15.172:40662) 2024-03-16T19:32:05.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fb2ecbff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2e00dfca0 0x7fb2e00e02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56504/0 (socket says 172.21.15.172:56504) 2024-03-16T19:32:05.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fb2edc01640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2e0170ad0 0x7fb2e0170f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7fe5a8cd9640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe59c04d400 0x7fe59c0368f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe5a404fde0 tx=0x7fe5a40691b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff290185a20 con 0x7ff290006f30 2024-03-16T19:32:05.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f5fa98f8640 1 -- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa4150cc0 msgr2=0x7f5fa4146db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f5fa98f8640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fa4150cc0 0x7f5fa4146db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2900e06e0 con 0x7ff290006f30 2024-03-16T19:32:05.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f5fa98f8640 1 -- 172.21.15.172:0/3917106129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fa4156710 msgr2=0x7f5fa41ebf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f5fa98f8640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fa4156710 0x7f5fa41ebf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f5fa98f8640 1 -- 172.21.15.172:0/3917106129 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f94009400 con 0x7f5fa406aed0 2024-03-16T19:32:05.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7ff294ff9640 1 -- 172.21.15.172:0/155708863 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28c003820 con 0x7ff290006f30 2024-03-16T19:32:05.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7ff3067fc640 1 -- 172.21.15.172:0/4203680405 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3000052d0 con 0x7ff31015ed80 2024-03-16T19:32:05.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7ff3067fc640 1 -- 172.21.15.172:0/4203680405 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300005430 con 0x7ff31015ed80 2024-03-16T19:32:05.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7ff3067fc640 1 -- 172.21.15.172:0/4203680405 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300007240 con 0x7ff31015ed80 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7fb2edc01640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2e0170ad0 0x7fb2e0170f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39134/0 (socket says 172.21.15.172:39134) 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7fb2ed400640 1 -- 172.21.15.172:0/2631953268 learned_addr learned my addr 172.21.15.172:0/2631953268 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e0174cb0 con 0x7fb2e0170ad0 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e00df160 con 0x7fb2d4003970 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e0173e20 con 0x7fb2e00dfca0 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7fb2ecbff640 1 -- 172.21.15.172:0/2631953268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d4003970 msgr2=0x7fb2e00e1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7fb2ecbff640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2d4003970 0x7fb2e00e1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7fb2ecbff640 1 -- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2e0170ad0 msgr2=0x7fb2e0170f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7fb2ecbff640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2e0170ad0 0x7fb2e0170f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7fb2ecbff640 1 -- 172.21.15.172:0/2631953268 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e01714a0 con 0x7fb2e00dfca0 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7fb2edc01640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb2e0170ad0 0x7fb2e0170f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:05.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f31cbfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc06d410 0x7f31cc150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f31cbfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc06d410 0x7f31cc150ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56490/0 (socket says 172.21.15.172:56490) 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.863+0000 7f31cbfff640 1 -- 172.21.15.172:0/383680491 learned_addr learned my addr 172.21.15.172:0/383680491 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f31cb7fe640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31cc158d20 0x7f31cc14dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f31cbfff640 1 -- 172.21.15.172:0/383680491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31cc158d20 msgr2=0x7f31cc14dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f31cbfff640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31cc158d20 0x7f31cc14dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f31cbfff640 1 -- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31cc14e180 msgr2=0x7f31cc0cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f31cbfff640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f31cc14e180 0x7f31cc0cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f31cbfff640 1 -- 172.21.15.172:0/383680491 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c4009a70 con 0x7f31cc06d410 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7ff294ff9640 1 -- 172.21.15.172:0/155708863 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c003bb0 con 0x7ff290006f30 2024-03-16T19:32:05.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff278001030 con 0x7ff290006f30 2024-03-16T19:32:05.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f5fa98f8640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa406aed0 0x7f5fa41597e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c8c0 tx=0x7f5fa000cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f5f9a7fc640 1 -- 172.21.15.172:0/3917106129 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000db00 con 0x7f5fa406aed0 2024-03-16T19:32:05.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa414e2d0 con 0x7f5fa406aed0 2024-03-16T19:32:05.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa41ec4b0 con 0x7f5fa406aed0 2024-03-16T19:32:05.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7f5f9a7fc640 1 -- 172.21.15.172:0/3917106129 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa000dc60 con 0x7f5fa406aed0 2024-03-16T19:32:05.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7f5f9a7fc640 1 -- 172.21.15.172:0/3917106129 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0015460 con 0x7f5fa406aed0 2024-03-16T19:32:05.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7ff3181a1640 1 -- 172.21.15.172:0/4203680405 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31014bb10 con 0x7ff31015ed80 2024-03-16T19:32:05.891 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7ff3181a1640 1 -- 172.21.15.172:0/4203680405 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff310143630 con 0x7ff31015ed80 2024-03-16T19:32:05.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7fe599ffb640 1 -- 172.21.15.172:0/3017947549 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40ba820 con 0x7fe59c04d400 2024-03-16T19:32:05.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.864+0000 7fe599ffb640 1 -- 172.21.15.172:0/3017947549 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a40ba980 con 0x7fe59c04d400 2024-03-16T19:32:05.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7fe599ffb640 1 -- 172.21.15.172:0/3017947549 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40bacf0 con 0x7fe59c04d400 2024-03-16T19:32:05.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59c035e30 con 0x7fe59c04d400 2024-03-16T19:32:05.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7f51dce21640 1 Processor -- start 2024-03-16T19:32:05.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7f31cbfff640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc06d410 0x7f31cc150ef0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00b450 tx=0x7f31bc00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7f31c8ff9640 1 -- 172.21.15.172:0/383680491 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc011830 con 0x7f31cc06d410 2024-03-16T19:32:05.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59c0dd110 con 0x7fe59c04d400 2024-03-16T19:32:05.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7f31c8ff9640 1 -- 172.21.15.172:0/383680491 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31bc0119b0 con 0x7f31cc06d410 2024-03-16T19:32:05.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7f31c8ff9640 1 -- 172.21.15.172:0/383680491 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc011d60 con 0x7f31cc06d410 2024-03-16T19:32:05.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7f51dce21640 1 -- start start 2024-03-16T19:32:05.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f8c001030 con 0x7f5fa406aed0 2024-03-16T19:32:05.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7fb2ecbff640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2e00dfca0 0x7fb2e00e02c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4007360 tx=0x7fb2e4007820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.865+0000 7f538a59c640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f538c06d5c0 0x7f538c0c2510 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5384009ee0 tx=0x7f5384004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.894 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7ff3181a1640 1 -- 172.21.15.172:0/4203680405 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff31006b4f0 con 0x7ff31015ed80 2024-03-16T19:32:05.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe59c008420 con 0x7fe59c04d400 2024-03-16T19:32:05.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7fb2ddffb640 1 -- 172.21.15.172:0/2631953268 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e400f8e0 con 0x7fb2e00dfca0 2024-03-16T19:32:05.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7fb2ddffb640 1 -- 172.21.15.172:0/2631953268 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e400fa40 con 0x7fb2e00dfca0 2024-03-16T19:32:05.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7fb2ddffb640 1 -- 172.21.15.172:0/2631953268 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e400f8e0 con 0x7fb2e00dfca0 2024-03-16T19:32:05.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e004a580 con 0x7fb2e00dfca0 2024-03-16T19:32:05.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7f31d27d7640 1 -- 172.21.15.172:0/383680491 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c0009460 con 0x7f31cc06d410 2024-03-16T19:32:05.895 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7f31d27d7640 1 -- 172.21.15.172:0/383680491 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc14e630 con 0x7f31cc06d410 2024-03-16T19:32:05.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f5f9a7fc640 1 -- 172.21.15.172:0/3917106129 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5fa001e3b0 con 0x7f5fa406aed0 2024-03-16T19:32:05.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f51dce21640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51d80cb120 0x7f51d81f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f5f9a7fc640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5f8806b4b0 0x7f5f8806d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7f537affd640 1 -- 172.21.15.172:0/1410470205 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538403f420 con 0x7f538c06d5c0 2024-03-16T19:32:05.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.866+0000 7f537affd640 1 -- 172.21.15.172:0/1410470205 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538403f580 con 0x7f538c06d5c0 2024-03-16T19:32:05.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f31b27fc640 1 -- 172.21.15.172:0/383680491 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f31cc15eb80 con 0x7f31cc06d410 2024-03-16T19:32:05.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f5f9a7fc640 1 -- 172.21.15.172:0/3917106129 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f5fa008a6f0 con 0x7f5fa406aed0 2024-03-16T19:32:05.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f5f9a7fc640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f5f88073850 0x7f5f88075cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7ff3067fc640 1 -- 172.21.15.172:0/4203680405 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff3000073a0 con 0x7ff31015ed80 2024-03-16T19:32:05.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f537affd640 1 -- 172.21.15.172:0/1410470205 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538403f910 con 0x7f538c06d5c0 2024-03-16T19:32:05.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538c0c1380 con 0x7f538c06d5c0 2024-03-16T19:32:05.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f51dce21640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51d80d0800 0x7f51d80c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f5f9a7fc640 1 -- 172.21.15.172:0/3917106129 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f88076380 con 0x7f5f88073850 2024-03-16T19:32:05.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f5f9a7fc640 1 -- 172.21.15.172:0/3917106129 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fa008cd10 con 0x7f5fa406aed0 2024-03-16T19:32:05.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f51d77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51d80cb120 0x7f51d81f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538c0c1940 con 0x7f538c06d5c0 2024-03-16T19:32:05.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f51d77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51d80cb120 0x7f51d81f4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40666/0 (socket says 172.21.15.172:40666) 2024-03-16T19:32:05.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f51d77fe640 1 -- 172.21.15.172:0/1624600391 learned_addr learned my addr 172.21.15.172:0/1624600391 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:05.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7f5fa90f7640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5f8806b4b0 0x7f5f8806d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f5faa0f9640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f5f88073850 0x7f5f88075cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f51d6ffd640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51d80d0800 0x7f51d80c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.900 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f51dce21640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d8158fe0 0x7f51d815d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7fe599ffb640 1 -- 172.21.15.172:0/3017947549 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fe5a40b7050 con 0x7fe59c04d400 2024-03-16T19:32:05.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e0174310 con 0x7fb2e00dfca0 2024-03-16T19:32:05.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7f5fa90f7640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5f8806b4b0 0x7f5f8806d970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5f94036000 tx=0x7f5f94030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.867+0000 7ff294ff9640 1 -- 172.21.15.172:0/155708863 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff28c003d10 con 0x7ff290006f30 2024-03-16T19:32:05.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7ff294ff9640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff27c06b480 0x7ff27c06d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7ff2977fe640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff27c06b480 0x7ff27c06d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7ff294ff9640 1 -- 172.21.15.172:0/155708863 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7ff28c0abd80 con 0x7ff290006f30 2024-03-16T19:32:05.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7ff294ff9640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff27c073820 0x7ff27c075c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7ff294ff9640 1 -- 172.21.15.172:0/155708863 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff27c076350 con 0x7ff27c073820 2024-03-16T19:32:05.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7ff294ff9640 1 -- 172.21.15.172:0/155708863 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff28c0ac130 con 0x7ff290006f30 2024-03-16T19:32:05.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7f31c8ff9640 1 -- 172.21.15.172:0/383680491 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f31bc01a400 con 0x7f31cc06d410 2024-03-16T19:32:05.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2e00dd8e0 con 0x7fb2e00dfca0 2024-03-16T19:32:05.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d815c540 con 0x7f51d8158fe0 2024-03-16T19:32:05.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80c6ea0 con 0x7f51d80cb120 2024-03-16T19:32:05.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80c76e0 con 0x7f51d80d0800 2024-03-16T19:32:05.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7ff29c813640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff27c073820 0x7ff27c075c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7ff2977fe640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff27c06b480 0x7ff27c06d940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff29005e070 tx=0x7ff284009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f538c06e4b0 con 0x7f538c06d5c0 2024-03-16T19:32:05.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.868+0000 7fe599ffb640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe58806b400 0x7fe58806d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7fe5a94da640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe58806b400 0x7fe58806d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7fe5a94da640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe58806b400 0x7fe58806d8c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe5940014f0 tx=0x7fe5940078d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7fe599ffb640 1 -- 172.21.15.172:0/3017947549 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fe5a40f63f0 con 0x7fe59c04d400 2024-03-16T19:32:05.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7fe599ffb640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fe588073720 0x7fe588075b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7fe599ffb640 1 -- 172.21.15.172:0/3017947549 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe588076250 con 0x7fe588073720 2024-03-16T19:32:05.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7fe599ffb640 1 -- 172.21.15.172:0/3017947549 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5a40f1f90 con 0x7fe59c04d400 2024-03-16T19:32:05.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.869+0000 7f31c8ff9640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f31b806b480 0x7f31b806d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f31c8ff9640 1 -- 172.21.15.172:0/383680491 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f31bc08a5f0 con 0x7f31cc06d410 2024-03-16T19:32:05.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f31c8ff9640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f31b80737c0 0x7f31b8075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f31c8ff9640 1 -- 172.21.15.172:0/383680491 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31b80762f0 con 0x7f31b80737c0 2024-03-16T19:32:05.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f31c8ff9640 1 -- 172.21.15.172:0/383680491 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31bc08a9a0 con 0x7f31cc06d410 2024-03-16T19:32:05.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7fe5aa4dc640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fe588073720 0x7fe588075b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f51d77fe640 1 -- 172.21.15.172:0/1624600391 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51d80d0800 msgr2=0x7f51d80c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f51d7fff640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d8158fe0 0x7f51d815d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7ff29c813640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff27c073820 0x7ff27c075c80 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f537affd640 1 -- 172.21.15.172:0/1410470205 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f538403fa70 con 0x7f538c06d5c0 2024-03-16T19:32:05.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f51d77fe640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51d80d0800 0x7f51d80c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f51d77fe640 1 -- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d8158fe0 msgr2=0x7f51d815d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7fb2ddffb640 1 -- 172.21.15.172:0/2631953268 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb2e400fbb0 con 0x7fb2e00dfca0 2024-03-16T19:32:05.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f51d77fe640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d8158fe0 0x7f51d815d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f51d77fe640 1 -- 172.21.15.172:0/1624600391 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c80094d0 con 0x7f51d80cb120 2024-03-16T19:32:05.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7fe5aa4dc640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fe588073720 0x7fe588075b80 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f51d7fff640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51d8158fe0 0x7f51d815d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:05.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7ff3067fc640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff2f406b480 0x7ff2f406d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f31cb7fe640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f31b806b480 0x7f31b806d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7f31d0d74640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f31b80737c0 0x7f31b8075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.870+0000 7fb2ddffb640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2cc06b4a0 0x7fb2cc06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f537affd640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f536c06b4b0 0x7f536c06d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f537affd640 1 -- 172.21.15.172:0/1410470205 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f53840ac6f0 con 0x7f538c06d5c0 2024-03-16T19:32:05.909 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f51d77fe640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51d80cb120 0x7f51d81f4380 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009870 tx=0x7f51c80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7ff3067fc640 1 -- 172.21.15.172:0/4203680405 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7ff300088f80 con 0x7ff31015ed80 2024-03-16T19:32:05.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7ff3067fc640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff2f4073740 0x7ff2f4075ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7ff3067fc640 1 -- 172.21.15.172:0/4203680405 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2f4076270 con 0x7ff2f4073740 2024-03-16T19:32:05.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7fb2ddffb640 1 -- 172.21.15.172:0/2631953268 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fb2e4089900 con 0x7fb2e00dfca0 2024-03-16T19:32:05.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7fb2ddffb640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb2cc0737c0 0x7fb2cc075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7fb2ddffb640 1 -- 172.21.15.172:0/2631953268 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2cc0762f0 con 0x7fb2cc0737c0 2024-03-16T19:32:05.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7fb2ddffb640 1 -- 172.21.15.172:0/2631953268 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2e400f8e0 con 0x7fb2e00dfca0 2024-03-16T19:32:05.910 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7ff3067fc640 1 -- 172.21.15.172:0/4203680405 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff300053990 con 0x7ff31015ed80 2024-03-16T19:32:05.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7fb2ed400640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2cc06b4a0 0x7fb2cc06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f5faa0f9640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f5f88073850 0x7f5f88075cb0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7ff315f3d640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff2f406b480 0x7ff2f406d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f5389d9b640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f536c06b4b0 0x7f536c06d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f537affd640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f536c073680 0x7f536c075ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f537affd640 1 -- 172.21.15.172:0/1410470205 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f536c0761b0 con 0x7f536c073680 2024-03-16T19:32:05.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7ff294ff9640 1 -- 172.21.15.172:0/155708863 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff27c076350 con 0x7ff27c073820 2024-03-16T19:32:05.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f537affd640 1 -- 172.21.15.172:0/1410470205 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53840aed10 con 0x7f538c06d5c0 2024-03-16T19:32:05.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7fe599ffb640 1 -- 172.21.15.172:0/3017947549 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe588076250 con 0x7fe588073720 2024-03-16T19:32:05.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f31d0d74640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f31b80737c0 0x7f31b8075c20 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7ff31573c640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff2f4073740 0x7ff2f4075ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f51bffff640 1 -- 172.21.15.172:0/1624600391 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c803b040 con 0x7f51d80cb120 2024-03-16T19:32:05.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f51bffff640 1 -- 172.21.15.172:0/1624600391 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8042680 con 0x7f51d80cb120 2024-03-16T19:32:05.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f538ad9d640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f536c073680 0x7f536c075ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f51bffff640 1 -- 172.21.15.172:0/1624600391 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8042a10 con 0x7f51d80cb120 2024-03-16T19:32:05.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.872+0000 7fb2edc01640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb2cc0737c0 0x7fb2cc075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.872+0000 7f538ad9d640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f536c073680 0x7f536c075ae0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.872+0000 7f5389d9b640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f536c06b4b0 0x7f536c06d970 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f538c0c24a0 tx=0x7f5380007e10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.871+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d80c79c0 con 0x7f51d80cb120 2024-03-16T19:32:05.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.872+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d80c7f80 con 0x7f51d80cb120 2024-03-16T19:32:05.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.872+0000 7ff31573c640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff2f4073740 0x7ff2f4075ba0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.872+0000 7f5f9a7fc640 1 -- 172.21.15.172:0/3917106129 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5f88076380 con 0x7f5f88073850 2024-03-16T19:32:05.915 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.873+0000 7fb2ed400640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2cc06b4a0 0x7fb2cc06d960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8075e70 tx=0x7fb2e80b7040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.873+0000 7ff3067fc640 1 -- 172.21.15.172:0/4203680405 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff2f4076270 con 0x7ff2f4073740 2024-03-16T19:32:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.873+0000 7f537affd640 1 -- 172.21.15.172:0/1410470205 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f536c0761b0 con 0x7f536c073680 2024-03-16T19:32:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.873+0000 7f31cb7fe640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f31b806b480 0x7f31b806d940 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31c0035000 tx=0x7f31c0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.873+0000 7f31c8ff9640 1 -- 172.21.15.172:0/383680491 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f31b80762f0 con 0x7f31b80737c0 2024-03-16T19:32:05.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.873+0000 7fb2edc01640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb2cc0737c0 0x7fb2cc075c20 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.873+0000 7f51bffff640 1 -- 172.21.15.172:0/1624600391 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f51c8042b70 con 0x7f51d80cb120 2024-03-16T19:32:05.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.874+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51d8162680 con 0x7f51d80cb120 2024-03-16T19:32:05.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.874+0000 7f51bffff640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51c406b480 0x7f51c406d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.874+0000 7f51bffff640 1 -- 172.21.15.172:0/1624600391 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f51c80acfb0 con 0x7f51d80cb120 2024-03-16T19:32:05.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.875+0000 7f51bffff640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f51c4073740 0x7f51c4075ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:05.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.875+0000 7fb2ddffb640 1 -- 172.21.15.172:0/2631953268 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2cc0762f0 con 0x7fb2cc0737c0 2024-03-16T19:32:05.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.875+0000 7f51bffff640 1 -- 172.21.15.172:0/1624600391 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51c4076270 con 0x7f51c4073740 2024-03-16T19:32:05.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.875+0000 7f51bffff640 1 -- 172.21.15.172:0/1624600391 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51c802fd30 con 0x7f51d80cb120 2024-03-16T19:32:05.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.875+0000 7f51d6ffd640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51c406b480 0x7f51c406d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.876+0000 7f51d7fff640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f51c4073740 0x7f51c4075ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:05.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.876+0000 7f51d7fff640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f51c4073740 0x7f51c4075ba0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.876+0000 7f51d6ffd640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51c406b480 0x7f51c406d940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f51d0001850 tx=0x7f51d00017e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.874+0000 7ff315f3d640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff2f406b480 0x7ff2f406d940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0096f0 tx=0x7ff30c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:05.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.877+0000 7f51bffff640 1 -- 172.21.15.172:0/1624600391 <== osd.2 v2:172.21.15.172:6823/3572014952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51c4076270 con 0x7f51c4073740 2024-03-16T19:32:05.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.902+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f8c002d30 con 0x7f5f88073850 2024-03-16T19:32:05.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.903+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff278002d30 con 0x7ff27c073820 2024-03-16T19:32:05.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.903+0000 7f5f9a7fc640 1 -- 172.21.15.172:0/3917106129 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5f8c002d30 con 0x7f5f88073850 2024-03-16T19:32:05.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.903+0000 7ff294ff9640 1 -- 172.21.15.172:0/155708863 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff278002d30 con 0x7ff27c073820 2024-03-16T19:32:05.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.903+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe59c04cba0 con 0x7fe588073720 2024-03-16T19:32:05.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.903+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f5f88073850 msgr2=0x7f5f88075cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.921 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.903+0000 7f5fabb5c640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f5f88073850 0x7f5f88075cb0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.903+0000 7fe599ffb640 1 -- 172.21.15.172:0/3017947549 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe5a0001200 con 0x7fe588073720 2024-03-16T19:32:05.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff27c073820 msgr2=0x7ff27c075c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7ff29d014640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7ff27c073820 0x7ff27c075c80 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff27c06b480 msgr2=0x7ff27c06d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.922 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7ff29d014640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff27c06b480 0x7ff27c06d940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff29005e070 tx=0x7ff284009040 comp rx=0 tx=0).stop 2024-03-16T19:32:05.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fe588073720 msgr2=0x7fe588075b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7fe5ab73e640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7fe588073720 0x7fe588075b80 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290006f30 msgr2=0x7ff2900e6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7ff29d014640 1 --2- 172.21.15.172:0/155708863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290006f30 0x7ff2900e6020 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0098c0 tx=0x7ff28c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.923 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5f8806b4b0 msgr2=0x7f5f8806d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7f5fabb5c640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5f8806b4b0 0x7f5f8806d970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5f94036000 tx=0x7f5f94030000 comp rx=0 tx=0).stop 2024-03-16T19:32:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa406aed0 msgr2=0x7f5fa41597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7f5fabb5c640 1 --2- 172.21.15.172:0/3917106129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa406aed0 0x7f5fa41597e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c8c0 tx=0x7f5fa000cd80 comp rx=0 tx=0).stop 2024-03-16T19:32:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7f5faa0f9640 1 -- 172.21.15.172:0/3917106129 reap_dead start 2024-03-16T19:32:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 shutdown_connections 2024-03-16T19:32:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 >> 172.21.15.172:0/3917106129 conn(0x7f5fa40cc030 msgr2=0x7f5fa40cc410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe58806b400 msgr2=0x7fe58806d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7fe5ab73e640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe58806b400 0x7fe58806d8c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe5940014f0 tx=0x7fe5940078d0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7ff29c813640 1 -- 172.21.15.172:0/155708863 reap_dead start 2024-03-16T19:32:05.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 shutdown_connections 2024-03-16T19:32:05.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 >> 172.21.15.172:0/155708863 conn(0x7ff2900061a0 msgr2=0x7ff2900494b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe59c04d400 msgr2=0x7fe59c0368f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.904+0000 7fe5ab73e640 1 --2- 172.21.15.172:0/3017947549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe59c04d400 0x7fe59c0368f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe5a404fde0 tx=0x7fe5a40691b0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.925 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7fe5aa4dc640 1 -- 172.21.15.172:0/3017947549 reap_dead start 2024-03-16T19:32:05.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 shutdown_connections 2024-03-16T19:32:05.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7ff29d014640 1 -- 172.21.15.172:0/155708863 wait complete. 2024-03-16T19:32:05.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 shutdown_connections 2024-03-16T19:32:05.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7f5fabb5c640 1 -- 172.21.15.172:0/3917106129 wait complete. 2024-03-16T19:32:05.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 shutdown_connections 2024-03-16T19:32:05.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 >> 172.21.15.172:0/3017947549 conn(0x7fe59c051460 msgr2=0x7fe59c046400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 shutdown_connections 2024-03-16T19:32:05.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7fe5ab73e640 1 -- 172.21.15.172:0/3017947549 wait complete. 2024-03-16T19:32:05.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7ff3181a1640 1 -- 172.21.15.172:0/4203680405 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff31006af70 con 0x7ff2f4073740 2024-03-16T19:32:05.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f538c071c70 con 0x7f536c073680 2024-03-16T19:32:05.927 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.905+0000 7ff3067fc640 1 -- 172.21.15.172:0/4203680405 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff3080345c0 con 0x7ff2f4073740 2024-03-16T19:32:05.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.906+0000 7f537affd640 1 -- 172.21.15.172:0/1410470205 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f537c0013c0 con 0x7f536c073680 2024-03-16T19:32:05.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.906+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f536c073680 msgr2=0x7f536c075ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.928 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.906+0000 7f538b59e640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f536c073680 0x7f536c075ae0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.906+0000 7ff2f3fff640 1 -- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff2f4073740 msgr2=0x7ff2f4075ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.906+0000 7ff2f3fff640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7ff2f4073740 0x7ff2f4075ba0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.906+0000 7ff2f3fff640 1 -- 172.21.15.172:0/4203680405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff2f406b480 msgr2=0x7ff2f406d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.906+0000 7ff2f3fff640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff2f406b480 0x7ff2f406d940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0096f0 tx=0x7ff30c008040 comp rx=0 tx=0).stop 2024-03-16T19:32:05.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.906+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f536c06b4b0 msgr2=0x7f536c06d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.929 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7f538b59e640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f536c06b4b0 0x7f536c06d970 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f538c0c24a0 tx=0x7f5380007e10 comp rx=0 tx=0).stop 2024-03-16T19:32:05.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7ff2f3fff640 1 -- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31015ed80 msgr2=0x7ff310146860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7ff2f3fff640 1 --2- 172.21.15.172:0/4203680405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31015ed80 0x7ff310146860 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c3c0 tx=0x7ff30000c880 comp rx=0 tx=0).stop 2024-03-16T19:32:05.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb2e00e7570 con 0x7fb2cc0737c0 2024-03-16T19:32:05.930 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7ff31673e640 1 -- 172.21.15.172:0/4203680405 reap_dead start 2024-03-16T19:32:05.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7ff2f3fff640 1 -- 172.21.15.172:0/4203680405 shutdown_connections 2024-03-16T19:32:05.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7ff2f3fff640 1 -- 172.21.15.172:0/4203680405 >> 172.21.15.172:0/4203680405 conn(0x7ff3100cc030 msgr2=0x7ff3100c6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7ff2f3fff640 1 -- 172.21.15.172:0/4203680405 shutdown_connections 2024-03-16T19:32:05.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7ff2f3fff640 1 -- 172.21.15.172:0/4203680405 wait complete. 2024-03-16T19:32:05.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f538c06d5c0 msgr2=0x7f538c0c2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7f538b59e640 1 --2- 172.21.15.172:0/1410470205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f538c06d5c0 0x7f538c0c2510 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5384009ee0 tx=0x7f5384004a40 comp rx=0 tx=0).stop 2024-03-16T19:32:05.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7f538ad9d640 1 -- 172.21.15.172:0/1410470205 reap_dead start 2024-03-16T19:32:05.931 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.907+0000 7fb2ddffb640 1 -- 172.21.15.172:0/2631953268 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb2cc0762f0 con 0x7fb2cc0737c0 2024-03-16T19:32:05.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.908+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 shutdown_connections 2024-03-16T19:32:05.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.908+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 >> 172.21.15.172:0/1410470205 conn(0x7f538c06e0d0 msgr2=0x7f538c1514d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.908+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 shutdown_connections 2024-03-16T19:32:05.932 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.908+0000 7f538b59e640 1 -- 172.21.15.172:0/1410470205 wait complete. 2024-03-16T19:32:05.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.908+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb2cc0737c0 msgr2=0x7fb2cc075c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.908+0000 7fb2ee402640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb2cc0737c0 0x7fb2cc075c20 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.933 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.909+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2cc06b4a0 msgr2=0x7fb2cc06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.909+0000 7fb2ee402640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb2cc06b4a0 0x7fb2cc06d960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8075e70 tx=0x7fb2e80b7040 comp rx=0 tx=0).stop 2024-03-16T19:32:05.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.909+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2e00dfca0 msgr2=0x7fb2e00e02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.909+0000 7fb2ee402640 1 --2- 172.21.15.172:0/2631953268 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2e00dfca0 0x7fb2e00e02c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4007360 tx=0x7fb2e4007820 comp rx=0 tx=0).stop 2024-03-16T19:32:05.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.909+0000 7fb2edc01640 1 -- 172.21.15.172:0/2631953268 reap_dead start 2024-03-16T19:32:05.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.909+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 shutdown_connections 2024-03-16T19:32:05.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.909+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 >> 172.21.15.172:0/2631953268 conn(0x7fb2e000d190 msgr2=0x7fb2e0007460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.911+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 shutdown_connections 2024-03-16T19:32:05.934 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.911+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f51d81623a0 con 0x7f51c4073740 2024-03-16T19:32:05.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.911+0000 7f31b27fc640 1 -- 172.21.15.172:0/383680491 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f31cc15f490 con 0x7f31b80737c0 2024-03-16T19:32:05.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.911+0000 7fb2ee402640 1 -- 172.21.15.172:0/2631953268 wait complete. 2024-03-16T19:32:05.935 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.911+0000 7f51bffff640 1 -- 172.21.15.172:0/1624600391 <== osd.2 v2:172.21.15.172:6823/3572014952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f51c4076270 con 0x7f51c4073740 2024-03-16T19:32:05.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.911+0000 7f31c8ff9640 1 -- 172.21.15.172:0/383680491 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f31c400f040 con 0x7f31b80737c0 2024-03-16T19:32:05.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.912+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f51c4073740 msgr2=0x7f51c4075ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.912+0000 7f51dce21640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f51c4073740 0x7f51c4075ba0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.912+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51c406b480 msgr2=0x7f51c406d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.912+0000 7f51dce21640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f51c406b480 0x7f51c406d940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f51d0001850 tx=0x7f51d00017e0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.912+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51d80cb120 msgr2=0x7f51d81f4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.912+0000 7f51dce21640 1 --2- 172.21.15.172:0/1624600391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51d80cb120 0x7f51d81f4380 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009870 tx=0x7f51c80046c0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.912+0000 7f51d7fff640 1 -- 172.21.15.172:0/1624600391 reap_dead start 2024-03-16T19:32:05.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.913+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 shutdown_connections 2024-03-16T19:32:05.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.913+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 >> 172.21.15.172:0/1624600391 conn(0x7f51d806ca20 msgr2=0x7f51d80ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.913+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 shutdown_connections 2024-03-16T19:32:05.937 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.913+0000 7f51dce21640 1 -- 172.21.15.172:0/1624600391 wait complete. 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.913+0000 7f31b27fc640 1 -- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f31b80737c0 msgr2=0x7f31b8075c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.913+0000 7f31b27fc640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f31b80737c0 0x7f31b8075c20 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.914+0000 7f31b27fc640 1 -- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f31b806b480 msgr2=0x7f31b806d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.914+0000 7f31b27fc640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f31b806b480 0x7f31b806d940 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31c0035000 tx=0x7f31c0008030 comp rx=0 tx=0).stop 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.914+0000 7f31b27fc640 1 -- 172.21.15.172:0/383680491 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc06d410 msgr2=0x7f31cc150ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.914+0000 7f31b27fc640 1 --2- 172.21.15.172:0/383680491 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31cc06d410 0x7f31cc150ef0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00b450 tx=0x7f31bc00b910 comp rx=0 tx=0).stop 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.914+0000 7f31d0d74640 1 -- 172.21.15.172:0/383680491 reap_dead start 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.915+0000 7f31b27fc640 1 -- 172.21.15.172:0/383680491 shutdown_connections 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.915+0000 7f31b27fc640 1 -- 172.21.15.172:0/383680491 >> 172.21.15.172:0/383680491 conn(0x7f31cc06e0d0 msgr2=0x7f31cc15dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.915+0000 7f31b27fc640 1 -- 172.21.15.172:0/383680491 shutdown_connections 2024-03-16T19:32:05.938 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:05.915+0000 7f31b27fc640 1 -- 172.21.15.172:0/383680491 wait complete. 2024-03-16T19:32:05.939 INFO:teuthology.orchestra.run.smithi172.stdout:55834574886 2024-03-16T19:32:05.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-16T19:32:05.940 INFO:teuthology.orchestra.run.smithi172.stdout:55834574885 2024-03-16T19:32:05.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-16T19:32:05.941 INFO:teuthology.orchestra.run.smithi172.stdout:55834574886 2024-03-16T19:32:05.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-16T19:32:05.942 INFO:teuthology.orchestra.run.smithi172.stdout:55834574887 2024-03-16T19:32:05.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-16T19:32:05.943 INFO:teuthology.orchestra.run.smithi172.stdout:55834574888 2024-03-16T19:32:05.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-16T19:32:05.944 INFO:teuthology.orchestra.run.smithi172.stdout:55834574886 2024-03-16T19:32:05.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-16T19:32:05.945 INFO:teuthology.orchestra.run.smithi172.stdout:377957122053 2024-03-16T19:32:05.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-16T19:32:05.945 INFO:teuthology.orchestra.run.smithi172.stdout:55834574887 2024-03-16T19:32:05.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-16T19:32:06.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.170+0000 7fa27e207640 1 Processor -- start 2024-03-16T19:32:06.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.170+0000 7fa27e207640 1 -- start start 2024-03-16T19:32:06.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.171+0000 7fa27e207640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa2780c85b0 0x7fa278156f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.171+0000 7fa27e207640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa278156ad0 con 0x7fa2780c6a30 2024-03-16T19:32:06.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.171+0000 7fa27e207640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2780c1c00 con 0x7fa278150da0 2024-03-16T19:32:06.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.171+0000 7fa27e207640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27806b980 con 0x7fa2780c85b0 2024-03-16T19:32:06.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.171+0000 7fa276ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa2780c6a30 0x7fa2780c7ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:38056/0 (socket says 172.21.15.172:38056) 2024-03-16T19:32:06.152 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.171+0000 7fa276ffd640 1 -- 172.21.15.172:0/2240928684 learned_addr learned my addr 172.21.15.172:0/2240928684 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:06.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.171+0000 7f1da74d4640 1 Processor -- start 2024-03-16T19:32:06.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.171+0000 7fa277fff640 1 --2- 172.21.15.172:0/2240928684 >> v2:172.21.15.172:3300/0 conn(0x7fa2780c85b0 0x7fa278156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.171+0000 7fa2767fc640 1 -- 172.21.15.172:0/2240928684 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220557002 0 0) 0x7fa278156ad0 con 0x7fa2780c6a30 2024-03-16T19:32:06.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.172+0000 7fa2767fc640 1 -- 172.21.15.172:0/2240928684 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa260004630 con 0x7fa2780c6a30 2024-03-16T19:32:06.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.172+0000 7fc1849c1640 1 Processor -- start 2024-03-16T19:32:06.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.172+0000 7fa2767fc640 1 -- 172.21.15.172:0/2240928684 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 752170559 0 0) 0x7fa260004630 con 0x7fa2780c6a30 2024-03-16T19:32:06.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.172+0000 7fa277fff640 1 -- 172.21.15.172:0/2240928684 >> v1:172.21.15.174:6789/0 conn(0x7fa278150da0 legacy=0x7fa27806aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.172+0000 7fa277fff640 1 -- 172.21.15.172:0/2240928684 >> v1:172.21.15.172:6789/0 conn(0x7fa2780c6a30 legacy=0x7fa2780c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.172+0000 7fa277fff640 1 -- 172.21.15.172:0/2240928684 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2780c1ee0 con 0x7fa2780c85b0 2024-03-16T19:32:06.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.172+0000 7fc1849c1640 1 -- start start 2024-03-16T19:32:06.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7fc1849c1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc1780dd3b0 0x7fc178046450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7fc1849c1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc17804ba50 0x7fc17804be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7fc1849c1640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc178045440 con 0x7fc1780e3080 2024-03-16T19:32:06.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7fc1849c1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc178045fc0 con 0x7fc1780dd3b0 2024-03-16T19:32:06.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7fc1849c1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1780e3c20 con 0x7fc17804ba50 2024-03-16T19:32:06.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fc17ffff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fc1780e3080 0x7fc1780e3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:38066/0 (socket says 172.21.15.172:38066) 2024-03-16T19:32:06.155 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fc17ffff640 1 -- 172.21.15.172:0/68817835 learned_addr learned my addr 172.21.15.172:0/68817835 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.172+0000 7f1da74d4640 1 -- start start 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da74d4640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1da0152a20 0x7f1da006f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da74d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da00b0de0 0x7f1da015c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da74d4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1da00ce4e0 0x7f1da00ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da74d4640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da00c9cf0 con 0x7f1da0152a20 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da74d4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da00c9f90 con 0x7f1da00ce4e0 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da74d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da00ca250 con 0x7f1da00b0de0 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da5270640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1da0152a20 0x7f1da006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da5a71640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1da00ce4e0 0x7f1da00ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da5270640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1da0152a20 0x7f1da006f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39168/0 (socket says 172.21.15.172:39168) 2024-03-16T19:32:06.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da5a71640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1da00ce4e0 0x7f1da00ce970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40672/0 (socket says 172.21.15.172:40672) 2024-03-16T19:32:06.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1da5a71640 1 -- 172.21.15.172:0/2938442348 learned_addr learned my addr 172.21.15.172:0/2938442348 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7f1da5270640 1 -- 172.21.15.172:0/2938442348 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da00b0de0 msgr2=0x7f1da015c560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T19:32:06.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7f1da5270640 1 --2- 172.21.15.172:0/2938442348 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da00b0de0 0x7f1da015c560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7f1da5270640 1 -- 172.21.15.172:0/2938442348 >> v2:172.21.15.174:3300/0 conn(0x7f1da00ce4e0 msgr2=0x7f1da00ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7f1da5270640 1 --2- 172.21.15.172:0/2938442348 >> v2:172.21.15.174:3300/0 conn(0x7f1da00ce4e0 0x7f1da00ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7fa277fff640 1 --2- 172.21.15.172:0/2240928684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2780c85b0 0x7fa278156f90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0097a0 tx=0x7fa26c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f38ee9b088fa1aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fa2767fc640 1 -- 172.21.15.172:0/2240928684 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa270005070 con 0x7fa2780c85b0 2024-03-16T19:32:06.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fa2767fc640 1 -- 172.21.15.172:0/2240928684 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26c031590 con 0x7fa2780c85b0 2024-03-16T19:32:06.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fa2767fc640 1 -- 172.21.15.172:0/2240928684 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c031920 con 0x7fa2780c85b0 2024-03-16T19:32:06.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7f1c4a719640 1 Processor -- start 2024-03-16T19:32:06.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7fcf8da84640 1 Processor -- start 2024-03-16T19:32:06.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7f1da5270640 1 -- 172.21.15.172:0/2938442348 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da00ca530 con 0x7f1da0152a20 2024-03-16T19:32:06.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7f1da5a71640 1 --2- 172.21.15.172:0/2938442348 >> v2:172.21.15.174:3300/0 conn(0x7f1da00ce4e0 0x7f1da00ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:06.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.173+0000 7fcf8da84640 1 -- start start 2024-03-16T19:32:06.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fcf8da84640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf88151910 0x7fcf88072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815213086 0 0) 0x7fc178045440 con 0x7fc1780e3080 2024-03-16T19:32:06.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc170003520 con 0x7fc1780e3080 2024-03-16T19:32:06.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fc17f7fe640 1 --2- 172.21.15.172:0/68817835 >> v2:172.21.15.174:3300/0 conn(0x7fc1780dd3b0 0x7fc178046450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fcf8da84640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fcf880c3ea0 0x7fcf880c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.160 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fc17effd640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc17804ba50 0x7fc17804be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fcf877fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf88151910 0x7fcf88072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7fcf877fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf88151910 0x7fcf88072fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56542/0 (socket says 172.21.15.172:56542) 2024-03-16T19:32:06.161 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694211472 0 0) 0x7fc170003520 con 0x7fc1780e3080 2024-03-16T19:32:06.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fcf877fe640 1 -- 172.21.15.172:0/2118207412 learned_addr learned my addr 172.21.15.172:0/2118207412 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fcf7effd640 1 --2- 172.21.15.172:0/2118207412 >> v2:172.21.15.172:3300/0 conn(0x7fcf880c3ea0 0x7fcf880c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc17804ba50 msgr2=0x7fc17804be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc17804ba50 0x7fc17804be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 >> v2:172.21.15.174:3300/0 conn(0x7fc1780dd3b0 msgr2=0x7fc178046450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 --2- 172.21.15.172:0/68817835 >> v2:172.21.15.174:3300/0 conn(0x7fc1780dd3b0 0x7fc178046450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1780dd790 con 0x7fc1780e3080 2024-03-16T19:32:06.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17f7fe640 1 --2- 172.21.15.172:0/68817835 >> v2:172.21.15.174:3300/0 conn(0x7fc1780dd3b0 0x7fc178046450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:32:06.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7fc16c001670 con 0x7fc1780e3080 2024-03-16T19:32:06.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.174+0000 7f1c4a719640 1 -- start start 2024-03-16T19:32:06.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7f1c4a719640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1c4415ed80 0x7f1c4406ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7f1c4a719640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4406b7d0 con 0x7f1c44073180 2024-03-16T19:32:06.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7f1c4a719640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c441594f0 con 0x7f1c44159110 2024-03-16T19:32:06.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7f1c4a719640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c440c2c10 con 0x7f1c4415ed80 2024-03-16T19:32:06.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fcbaa28b640 1 Processor -- start 2024-03-16T19:32:06.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 >> v1:172.21.15.172:6789/0 conn(0x7fc1780e3080 legacy=0x7fc1780e3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc170003970 0x7fc170003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1700047c0 0x7fc170004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc17804ba50 0x7fc170005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc178045fc0 con 0x7fc170003970 2024-03-16T19:32:06.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1780e3c20 con 0x7fc1700047c0 2024-03-16T19:32:06.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17000ba80 con 0x7fc17804ba50 2024-03-16T19:32:06.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17effd640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1700047c0 0x7fc170004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17ffff640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc17804ba50 0x7fc170005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7f1c43fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f1c44159110 0x7f1c44162ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:56652/0 (socket says 172.21.15.172:56652) 2024-03-16T19:32:06.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7f1c43fff640 1 -- 172.21.15.172:0/547133525 learned_addr learned my addr 172.21.15.172:0/547133525 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:06.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fcbaa28b640 1 -- start start 2024-03-16T19:32:06.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7f1c437fe640 1 --2- 172.21.15.172:0/547133525 >> v2:172.21.15.172:3300/0 conn(0x7f1c4415ed80 0x7f1c4406ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fc17f7fe640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc170003970 0x7fc170003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fc17effd640 1 -- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc17804ba50 msgr2=0x7fc170005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7f1c42ffd640 1 -- 172.21.15.172:0/547133525 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324405309 0 0) 0x7f1c441594f0 con 0x7f1c44159110 2024-03-16T19:32:06.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7f1c42ffd640 1 -- 172.21.15.172:0/547133525 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c30003520 con 0x7f1c44159110 2024-03-16T19:32:06.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fcf8da84640 1 -- 172.21.15.172:0/2118207412 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf88072b10 con 0x7fcf880c3930 2024-03-16T19:32:06.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fcf8da84640 1 -- 172.21.15.172:0/2118207412 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf880c2330 con 0x7fcf880c3ea0 2024-03-16T19:32:06.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fcf8da84640 1 -- 172.21.15.172:0/2118207412 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880c7da0 con 0x7fcf88151910 2024-03-16T19:32:06.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fa27e207640 1 -- 172.21.15.172:0/2240928684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2780c85b0 msgr2=0x7fa278156f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7fa27e207640 1 --2- 172.21.15.172:0/2240928684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2780c85b0 0x7fa278156f90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0097a0 tx=0x7fa26c02b8b0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fc17effd640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc17804ba50 0x7fc170005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fc17effd640 1 -- 172.21.15.172:0/68817835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc170003970 msgr2=0x7fc170003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fc17effd640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc170003970 0x7fc170003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fc17effd640 1 -- 172.21.15.172:0/68817835 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc170003520 con 0x7fc1700047c0 2024-03-16T19:32:06.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fa27e207640 1 -- 172.21.15.172:0/2240928684 shutdown_connections 2024-03-16T19:32:06.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fa27e207640 1 --2- 172.21.15.172:0/2240928684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2780c85b0 0x7fa278156f90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fa27e207640 1 -- 172.21.15.172:0/2240928684 >> 172.21.15.172:0/2240928684 conn(0x7fa2780cc030 msgr2=0x7fa2780cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fcf877fe640 1 -- 172.21.15.172:0/2118207412 >> v2:172.21.15.172:3300/0 conn(0x7fcf880c3ea0 msgr2=0x7fcf880c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fcf877fe640 1 --2- 172.21.15.172:0/2118207412 >> v2:172.21.15.172:3300/0 conn(0x7fcf880c3ea0 0x7fcf880c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fcf877fe640 1 -- 172.21.15.172:0/2118207412 >> v1:172.21.15.172:6789/0 conn(0x7fcf880c3930 legacy=0x7fcf880ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fcf86ffd640 1 -- 172.21.15.172:0/2118207412 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203400221 0 0) 0x7fcf88072b10 con 0x7fcf880c3930 2024-03-16T19:32:06.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fcf877fe640 1 -- 172.21.15.172:0/2118207412 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88071b10 con 0x7fcf88151910 2024-03-16T19:32:06.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fa27e207640 1 -- 172.21.15.172:0/2240928684 shutdown_connections 2024-03-16T19:32:06.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7f1c437fe640 1 -- 172.21.15.172:0/547133525 >> v1:172.21.15.174:6789/0 conn(0x7f1c44159110 legacy=0x7f1c44162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7f1c437fe640 1 -- 172.21.15.172:0/547133525 >> v1:172.21.15.172:6789/0 conn(0x7f1c44073180 legacy=0x7f1c440c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7f1c437fe640 1 -- 172.21.15.172:0/547133525 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4406af70 con 0x7f1c4415ed80 2024-03-16T19:32:06.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.176+0000 7fa27e207640 1 -- 172.21.15.172:0/2240928684 wait complete. 2024-03-16T19:32:06.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.175+0000 7f1da5270640 1 --2- 172.21.15.172:0/2938442348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da0152a20 0x7f1da006f070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1da006f000 tx=0x7f1d9002bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cbf18ab58f6025 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcbaa28b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba41518f0 0x7fcba40708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcbaa28b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcba40c7400 0x7fcba4070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcbaa28b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba4151cd0 con 0x7fcba40b1ef0 2024-03-16T19:32:06.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcbaa28b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba4158540 con 0x7fcba40c7400 2024-03-16T19:32:06.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcbaa28b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40cce40 con 0x7fcba41518f0 2024-03-16T19:32:06.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fc17effd640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1700047c0 0x7fc170004c40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc180075f10 tx=0x7fc1700057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca06ac82d03804f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcba9a8a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcba40c7400 0x7fcba4070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcba8a88640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fcba40b1ef0 0x7fcba40c6cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:56666/0 (socket says 172.21.15.172:56666) 2024-03-16T19:32:06.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7faa44a22640 1 Processor -- start 2024-03-16T19:32:06.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1800a9720 con 0x7fc1700047c0 2024-03-16T19:32:06.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1800a98c0 con 0x7fc1700047c0 2024-03-16T19:32:06.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fc17e7fc640 1 -- 172.21.15.172:0/68817835 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1800a9cf0 con 0x7fc1700047c0 2024-03-16T19:32:06.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7f1d97fff640 1 -- 172.21.15.172:0/2938442348 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9002d700 con 0x7f1da0152a20 2024-03-16T19:32:06.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7f1d97fff640 1 -- 172.21.15.172:0/2938442348 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9002d860 con 0x7f1da0152a20 2024-03-16T19:32:06.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7f1d97fff640 1 -- 172.21.15.172:0/2938442348 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9002dbf0 con 0x7f1da0152a20 2024-03-16T19:32:06.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcf877fe640 1 --2- 172.21.15.172:0/2118207412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf88151910 0x7fcf88072fa0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009e50 tx=0x7fcf7802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=155ea5771c0810f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.176 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7faa44a22640 1 -- start start 2024-03-16T19:32:06.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcba8a88640 1 -- 172.21.15.172:0/2129936764 learned_addr learned my addr 172.21.15.172:0/2129936764 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:06.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcba9a8a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcba40c7400 0x7fcba4070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40702/0 (socket says 172.21.15.172:40702) 2024-03-16T19:32:06.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcba9289640 1 --2- 172.21.15.172:0/2129936764 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba41518f0 0x7fcba40708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.177 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7f1c437fe640 1 --2- 172.21.15.172:0/547133525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 0x7f1c4406ba10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c0097a0 tx=0x7f1c2c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c6525ec97d8318 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcf86ffd640 1 -- 172.21.15.172:0/2118207412 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7802cb00 con 0x7fcf88151910 2024-03-16T19:32:06.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcf86ffd640 1 -- 172.21.15.172:0/2118207412 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7802cc60 con 0x7fcf88151910 2024-03-16T19:32:06.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.177+0000 7fcf86ffd640 1 -- 172.21.15.172:0/2118207412 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf780343f0 con 0x7fcf88151910 2024-03-16T19:32:06.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fc1849c1640 1 -- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1700047c0 msgr2=0x7fc170004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fc1849c1640 1 --2- 172.21.15.172:0/68817835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1700047c0 0x7fc170004c40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc180075f10 tx=0x7fc1700057f0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fc17ffff640 1 -- 172.21.15.172:0/68817835 reap_dead start 2024-03-16T19:32:06.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fc1849c1640 1 -- 172.21.15.172:0/68817835 shutdown_connections 2024-03-16T19:32:06.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fc1849c1640 1 -- 172.21.15.172:0/68817835 >> 172.21.15.172:0/68817835 conn(0x7fc178050170 msgr2=0x7fc178050580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fc1849c1640 1 -- 172.21.15.172:0/68817835 shutdown_connections 2024-03-16T19:32:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fc1849c1640 1 -- 172.21.15.172:0/68817835 wait complete. 2024-03-16T19:32:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7faa44a22640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faa380dd4d0 0x7faa38045560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7faa44a22640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7faa38045aa0 0x7faa3804bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7faa44a22640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa380e4050 con 0x7faa3804c3e0 2024-03-16T19:32:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7faa44a22640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa380dd8b0 con 0x7faa38045aa0 2024-03-16T19:32:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7faa44a22640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa380e34d0 con 0x7faa380dd4d0 2024-03-16T19:32:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3f59e640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7faa3804c3e0 0x7faa38046510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:56676/0 (socket says 172.21.15.172:56676) 2024-03-16T19:32:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3f59e640 1 -- 172.21.15.172:0/2081080115 learned_addr learned my addr 172.21.15.172:0/2081080115 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:06.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fcb9bfff640 1 -- 172.21.15.172:0/2129936764 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040306394 0 0) 0x7fcba4151cd0 con 0x7fcba40b1ef0 2024-03-16T19:32:06.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fcb9bfff640 1 -- 172.21.15.172:0/2129936764 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb90003520 con 0x7fcba40b1ef0 2024-03-16T19:32:06.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fcba9a8a640 1 -- 172.21.15.172:0/2129936764 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba41518f0 msgr2=0x7fcba40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fcba9a8a640 1 --2- 172.21.15.172:0/2129936764 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba41518f0 0x7fcba40708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fcba9a8a640 1 -- 172.21.15.172:0/2129936764 >> v1:172.21.15.174:6789/0 conn(0x7fcba40b1ef0 legacy=0x7fcba40c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.180 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fcba9a8a640 1 -- 172.21.15.172:0/2129936764 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba4157650 con 0x7fcba40c7400 2024-03-16T19:32:06.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1da74d4640 1 -- 172.21.15.172:0/2938442348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da0152a20 msgr2=0x7f1da006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1da74d4640 1 --2- 172.21.15.172:0/2938442348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da0152a20 0x7f1da006f070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1da006f000 tx=0x7f1d9002bb50 comp rx=0 tx=0).stop 2024-03-16T19:32:06.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1da74d4640 1 -- 172.21.15.172:0/2938442348 shutdown_connections 2024-03-16T19:32:06.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1da74d4640 1 --2- 172.21.15.172:0/2938442348 >> v2:172.21.15.174:3300/0 conn(0x7f1da00ce4e0 0x7f1da00ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1da74d4640 1 --2- 172.21.15.172:0/2938442348 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da00b0de0 0x7f1da015c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1da74d4640 1 --2- 172.21.15.172:0/2938442348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da0152a20 0x7f1da006f070 secure :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1da006f000 tx=0x7f1d9002bb50 comp rx=0 tx=0).stop 2024-03-16T19:32:06.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1da74d4640 1 -- 172.21.15.172:0/2938442348 >> 172.21.15.172:0/2938442348 conn(0x7f1da0066e00 msgr2=0x7f1da00671e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.181 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1da74d4640 1 -- 172.21.15.172:0/2938442348 shutdown_connections 2024-03-16T19:32:06.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1c42ffd640 1 -- 172.21.15.172:0/547133525 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c031640 con 0x7f1c4415ed80 2024-03-16T19:32:06.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1c42ffd640 1 -- 172.21.15.172:0/547133525 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2c0317a0 con 0x7f1c4415ed80 2024-03-16T19:32:06.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7f1c42ffd640 1 -- 172.21.15.172:0/547133525 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c031b30 con 0x7f1c4415ed80 2024-03-16T19:32:06.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fcf8da84640 1 -- 172.21.15.172:0/2118207412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf88151910 msgr2=0x7fcf88072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.178+0000 7fcf8da84640 1 --2- 172.21.15.172:0/2118207412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf88151910 0x7fcf88072fa0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009e50 tx=0x7fcf7802bf60 comp rx=0 tx=0).stop 2024-03-16T19:32:06.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcf8da84640 1 -- 172.21.15.172:0/2118207412 shutdown_connections 2024-03-16T19:32:06.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcf8da84640 1 --2- 172.21.15.172:0/2118207412 >> v2:172.21.15.172:3300/0 conn(0x7fcf880c3ea0 0x7fcf880c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.182 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcf8da84640 1 --2- 172.21.15.172:0/2118207412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf88151910 0x7fcf88072fa0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcf8da84640 1 -- 172.21.15.172:0/2118207412 >> 172.21.15.172:0/2118207412 conn(0x7fcf8806db70 msgr2=0x7fcf8806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcf8da84640 1 -- 172.21.15.172:0/2118207412 shutdown_connections 2024-03-16T19:32:06.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcf8da84640 1 -- 172.21.15.172:0/2118207412 wait complete. 2024-03-16T19:32:06.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3ed9d640 1 --2- 172.21.15.172:0/2081080115 >> v2:172.21.15.174:3300/0 conn(0x7faa380dd4d0 0x7faa38045560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3e59c640 1 --2- 172.21.15.172:0/2081080115 >> v2:172.21.15.172:3300/0 conn(0x7faa38045aa0 0x7faa3804bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3dd9b640 1 -- 172.21.15.172:0/2081080115 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616281323 0 0) 0x7faa380e4050 con 0x7faa3804c3e0 2024-03-16T19:32:06.183 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3dd9b640 1 -- 172.21.15.172:0/2081080115 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa28003520 con 0x7faa3804c3e0 2024-03-16T19:32:06.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcba9a8a640 1 --2- 172.21.15.172:0/2129936764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40c7400 0x7fcba4070e10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00bfa0 tx=0x7fcb9c02e0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed0cdd161addefa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fa27e207640 1 Processor -- start 2024-03-16T19:32:06.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7f1da74d4640 1 -- 172.21.15.172:0/2938442348 wait complete. 2024-03-16T19:32:06.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcb9bfff640 1 -- 172.21.15.172:0/2129936764 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c02ea60 con 0x7fcba40c7400 2024-03-16T19:32:06.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcb9bfff640 1 -- 172.21.15.172:0/2129936764 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9c02ebc0 con 0x7fcba40c7400 2024-03-16T19:32:06.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fcb9bfff640 1 -- 172.21.15.172:0/2129936764 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c009540 con 0x7fcba40c7400 2024-03-16T19:32:06.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7fa27e207640 1 -- start start 2024-03-16T19:32:06.185 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3ed9d640 1 -- 172.21.15.172:0/2081080115 >> v2:172.21.15.172:3300/0 conn(0x7faa38045aa0 msgr2=0x7faa3804bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3ed9d640 1 --2- 172.21.15.172:0/2081080115 >> v2:172.21.15.172:3300/0 conn(0x7faa38045aa0 0x7faa3804bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3ed9d640 1 -- 172.21.15.172:0/2081080115 >> v1:172.21.15.174:6789/0 conn(0x7faa3804c3e0 legacy=0x7faa38046510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3ed9d640 1 -- 172.21.15.172:0/2081080115 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa380e37b0 con 0x7faa380dd4d0 2024-03-16T19:32:06.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.179+0000 7faa3e59c640 1 --2- 172.21.15.172:0/2081080115 >> v2:172.21.15.172:3300/0 conn(0x7faa38045aa0 0x7faa3804bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:06.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7f1c4a719640 1 -- 172.21.15.172:0/547133525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 msgr2=0x7f1c4406ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.186 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7f1c4a719640 1 --2- 172.21.15.172:0/547133525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 0x7f1c4406ba10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c0097a0 tx=0x7f1c2c02b8f0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7fcbaa28b640 1 -- 172.21.15.172:0/2129936764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40c7400 msgr2=0x7fcba4070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7fcbaa28b640 1 --2- 172.21.15.172:0/2129936764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40c7400 0x7fcba4070e10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00bfa0 tx=0x7fcb9c02e0c0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7f1da74d4640 1 Processor -- start 2024-03-16T19:32:06.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7f1da74d4640 1 -- start start 2024-03-16T19:32:06.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7fcbaa28b640 1 -- 172.21.15.172:0/2129936764 shutdown_connections 2024-03-16T19:32:06.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7fcbaa28b640 1 --2- 172.21.15.172:0/2129936764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40c7400 0x7fcba4070e10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7fcbaa28b640 1 --2- 172.21.15.172:0/2129936764 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba41518f0 0x7fcba40708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7fcbaa28b640 1 -- 172.21.15.172:0/2129936764 >> 172.21.15.172:0/2129936764 conn(0x7fcba40cbfc0 msgr2=0x7fcba40cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7fa27e207640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa2780c6a30 0x7fa2781e5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7faa3ed9d640 1 --2- 172.21.15.172:0/2081080115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380dd4d0 0x7faa38045560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faa38004cf0 tx=0x7faa3402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb3214fdbd3bdf39 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fcbaa28b640 1 -- 172.21.15.172:0/2129936764 shutdown_connections 2024-03-16T19:32:06.189 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7f1c4a719640 1 -- 172.21.15.172:0/547133525 shutdown_connections 2024-03-16T19:32:06.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa27e207640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2780c85b0 0x7fa2781f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa2777fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa2780c6a30 0x7fa2781e5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7f1c4a719640 1 --2- 172.21.15.172:0/547133525 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 0x7f1c4406ba10 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7f1c4a719640 1 -- 172.21.15.172:0/547133525 >> 172.21.15.172:0/547133525 conn(0x7f1c440cc030 msgr2=0x7f1c440cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7fc1849c1640 1 Processor -- start 2024-03-16T19:32:06.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7faa3dd9b640 1 -- 172.21.15.172:0/2081080115 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34030610 con 0x7faa380dd4d0 2024-03-16T19:32:06.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fcf8da84640 1 Processor -- start 2024-03-16T19:32:06.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7faa3dd9b640 1 -- 172.21.15.172:0/2081080115 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa34030770 con 0x7faa380dd4d0 2024-03-16T19:32:06.191 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7faa3dd9b640 1 -- 172.21.15.172:0/2081080115 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34030b00 con 0x7faa380dd4d0 2024-03-16T19:32:06.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.180+0000 7fc1849c1640 1 -- start start 2024-03-16T19:32:06.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa2777fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa2780c6a30 0x7fa2781e5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56564/0 (socket says 172.21.15.172:56564) 2024-03-16T19:32:06.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa27e207640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa278150da0 0x7fa2781e86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa27e207640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2780c6e70 con 0x7fa2780c85b0 2024-03-16T19:32:06.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fcf8da84640 1 -- start start 2024-03-16T19:32:06.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa27e207640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2781e6e80 con 0x7fa278150da0 2024-03-16T19:32:06.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa27e207640 1 -- 172.21.15.172:0/3723432955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2781e7540 con 0x7fa2780c6a30 2024-03-16T19:32:06.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa2777fe640 1 -- 172.21.15.172:0/3723432955 learned_addr learned my addr 172.21.15.172:0/3723432955 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa276ffd640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2780c85b0 0x7fa2781f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fc1849c1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1780dba20 0x7fc1780dbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7f5eb4a81640 1 Processor -- start 2024-03-16T19:32:06.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fa277fff640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa278150da0 0x7fa2781e86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc1849c1640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc178047e40 0x7fc1780486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f1da74d4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da00b0de0 0x7f1da00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fa2777fe640 1 -- 172.21.15.172:0/3723432955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa278150da0 msgr2=0x7fa2781e86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f1c4a719640 1 -- 172.21.15.172:0/547133525 shutdown_connections 2024-03-16T19:32:06.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc1849c1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc178048c00 0x7fc1780d5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc17effd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc178047e40 0x7fc1780486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc1849c1640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc178049a60 con 0x7fc178047e40 2024-03-16T19:32:06.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1780dba20 0x7fc1780dbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1780dba20 0x7fc1780dbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56570/0 (socket says 172.21.15.172:56570) 2024-03-16T19:32:06.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc17f7fe640 1 -- 172.21.15.172:0/3457261894 learned_addr learned my addr 172.21.15.172:0/3457261894 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fa2777fe640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa278150da0 0x7fa2781e86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fa2777fe640 1 -- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2780c85b0 msgr2=0x7fa2781f5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fa2777fe640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2780c85b0 0x7fa2781f5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fa2777fe640 1 -- 172.21.15.172:0/3723432955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c009400 con 0x7fa2780c6a30 2024-03-16T19:32:06.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f5eb4a81640 1 -- start start 2024-03-16T19:32:06.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc178047cc0 con 0x7fc178048c00 2024-03-16T19:32:06.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc178179660 con 0x7fc1780dba20 2024-03-16T19:32:06.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7faa44a22640 1 -- 172.21.15.172:0/2081080115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380dd4d0 msgr2=0x7faa38045560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7faa44a22640 1 --2- 172.21.15.172:0/2081080115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380dd4d0 0x7faa38045560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faa38004cf0 tx=0x7faa3402b960 comp rx=0 tx=0).stop 2024-03-16T19:32:06.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7faa44a22640 1 -- 172.21.15.172:0/2081080115 shutdown_connections 2024-03-16T19:32:06.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7faa44a22640 1 --2- 172.21.15.172:0/2081080115 >> v2:172.21.15.172:3300/0 conn(0x7faa38045aa0 0x7faa3804bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.198 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7faa44a22640 1 --2- 172.21.15.172:0/2081080115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa380dd4d0 0x7faa38045560 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faa38004cf0 tx=0x7faa3402b960 comp rx=0 tx=0).stop 2024-03-16T19:32:06.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7faa44a22640 1 -- 172.21.15.172:0/2081080115 >> 172.21.15.172:0/2081080115 conn(0x7faa380505c0 msgr2=0x7faa380509d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f1da74d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1da00ce4e0 0x7f1da01565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f1da74d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da014efe0 0x7f1da01f0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf8da84640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf880c3930 0x7fcf881de240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f1da74d4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da0155e40 con 0x7f1da00b0de0 2024-03-16T19:32:06.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf8da84640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf880c3ea0 0x7fcf88155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf8da84640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf88151910 0x7fcf881e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.200 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fc17ffff640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc178048c00 0x7fc1780d5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f1da74d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da00b3df0 con 0x7f1da00ce4e0 2024-03-16T19:32:06.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f1da74d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da0070e50 con 0x7f1da014efe0 2024-03-16T19:32:06.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fa2777fe640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa2780c6a30 0x7fa2781e5d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa26800c460 tx=0x7fa26800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.181+0000 7fcbaa28b640 1 -- 172.21.15.172:0/2129936764 wait complete. 2024-03-16T19:32:06.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf8da84640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881dde20 con 0x7fcf88151910 2024-03-16T19:32:06.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf8da84640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881576f0 con 0x7fcf880c3ea0 2024-03-16T19:32:06.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcbaa28b640 1 Processor -- start 2024-03-16T19:32:06.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf8da84640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88154d20 con 0x7fcf880c3930 2024-03-16T19:32:06.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf877fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf880c3930 0x7fcf881de240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf7effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf880c3ea0 0x7fcf88155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7faa44a22640 1 -- 172.21.15.172:0/2081080115 shutdown_connections 2024-03-16T19:32:06.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7faa44a22640 1 -- 172.21.15.172:0/2081080115 wait complete. 2024-03-16T19:32:06.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf7effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf880c3ea0 0x7fcf88155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40740/0 (socket says 172.21.15.172:40740) 2024-03-16T19:32:06.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7fcf7effd640 1 -- 172.21.15.172:0/948528456 learned_addr learned my addr 172.21.15.172:0/948528456 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf87fff640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf88151910 0x7fcf881e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc17f7fe640 1 -- 172.21.15.172:0/3457261894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc178048c00 msgr2=0x7fc1780d5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc17f7fe640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc178048c00 0x7fc1780d5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc17f7fe640 1 -- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc178047e40 msgr2=0x7fc1780486c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc17f7fe640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc178047e40 0x7fc1780486c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc17f7fe640 1 -- 172.21.15.172:0/3457261894 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc178049080 con 0x7fc1780dba20 2024-03-16T19:32:06.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc17effd640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc178047e40 0x7fc1780486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:06.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf7effd640 1 -- 172.21.15.172:0/948528456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf880c3930 msgr2=0x7fcf881de240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf7effd640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf880c3930 0x7fcf881de240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf7effd640 1 -- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf88151910 msgr2=0x7fcf881e2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf7effd640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf88151910 0x7fcf881e2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf7effd640 1 -- 172.21.15.172:0/948528456 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78009b30 con 0x7fcf880c3ea0 2024-03-16T19:32:06.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf7effd640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf880c3ea0 0x7fcf88155cb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcf70011f20 tx=0x7fcf70014550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf84ff9640 1 -- 172.21.15.172:0/948528456 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf700112d0 con 0x7fcf880c3ea0 2024-03-16T19:32:06.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf84ff9640 1 -- 172.21.15.172:0/948528456 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf70017040 con 0x7fcf880c3ea0 2024-03-16T19:32:06.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcf84ff9640 1 -- 172.21.15.172:0/948528456 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70011670 con 0x7fcf880c3ea0 2024-03-16T19:32:06.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fcbaa28b640 1 -- start start 2024-03-16T19:32:06.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcf8da84640 1 -- 172.21.15.172:0/948528456 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf881ecf80 con 0x7fcf880c3ea0 2024-03-16T19:32:06.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc17f7fe640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1780dba20 0x7fc1780dbe00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc174010890 tx=0x7fc174010d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc16bfff640 1 -- 172.21.15.172:0/3457261894 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174010e80 con 0x7fc1780dba20 2024-03-16T19:32:06.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc16bfff640 1 -- 172.21.15.172:0/3457261894 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17400c050 con 0x7fc1780dba20 2024-03-16T19:32:06.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc16bfff640 1 -- 172.21.15.172:0/3457261894 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174004260 con 0x7fc1780dba20 2024-03-16T19:32:06.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc178179890 con 0x7fc1780dba20 2024-03-16T19:32:06.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.183+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc178047950 con 0x7fc1780dba20 2024-03-16T19:32:06.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcf8da84640 1 -- 172.21.15.172:0/948528456 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf881554b0 con 0x7fcf880c3ea0 2024-03-16T19:32:06.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7f5eb4a81640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5eb0073180 0x7f5eb00c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7f5eb4a81640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb006b7d0 con 0x7f5eb015ed80 2024-03-16T19:32:06.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7f5eb4a81640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb01594f0 con 0x7f5eb0159110 2024-03-16T19:32:06.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7f5eb4a81640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb00c2c10 con 0x7f5eb0073180 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7f5eadd9b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f5eb015ed80 0x7f5eb006ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:38090/0 (socket says 172.21.15.172:38090) 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7f5eadd9b640 1 -- 172.21.15.172:0/72823686 learned_addr learned my addr 172.21.15.172:0/72823686 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcbaa28b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40b1ef0 0x7fcba4152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcbaa28b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba40c7400 0x7fcba4070a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcbaa28b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcba41518f0 0x7fcba41f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcbaa28b640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba41491a0 con 0x7fcba41518f0 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcbaa28b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba4152050 con 0x7fcba40b1ef0 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcbaa28b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba4152330 con 0x7fcba40c7400 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcba9289640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40b1ef0 0x7fcba4152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcba9289640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40b1ef0 0x7fcba4152d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40762/0 (socket says 172.21.15.172:40762) 2024-03-16T19:32:06.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcba9289640 1 -- 172.21.15.172:0/1037635362 learned_addr learned my addr 172.21.15.172:0/1037635362 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcba8a88640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba40c7400 0x7fcba4070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7fcba9289640 1 -- 172.21.15.172:0/1037635362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba40c7400 msgr2=0x7fcba4070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcba9289640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba40c7400 0x7fcba4070a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7f5eaed9d640 1 --2- 172.21.15.172:0/72823686 >> v2:172.21.15.172:3300/0 conn(0x7f5eb0073180 0x7f5eb00c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcba9289640 1 -- 172.21.15.172:0/1037635362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcba41518f0 msgr2=0x7fcba41f7e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcba9289640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcba41518f0 0x7fcba41f7e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcba9289640 1 -- 172.21.15.172:0/1037635362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c00bc80 con 0x7fcba40b1ef0 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcba8a88640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba40c7400 0x7fcba4070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcba9289640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40b1ef0 0x7fcba4152d50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcba000d800 tx=0x7fcba000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcb99ffb640 1 -- 172.21.15.172:0/1037635362 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba000b6c0 con 0x7fcba40b1ef0 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcb99ffb640 1 -- 172.21.15.172:0/1037635362 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba000b820 con 0x7fcba40b1ef0 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcb99ffb640 1 -- 172.21.15.172:0/1037635362 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba000bb90 con 0x7fcba40b1ef0 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba4148870 con 0x7fcba40b1ef0 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7faa44a22640 1 Processor -- start 2024-03-16T19:32:06.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7faa44a22640 1 -- start start 2024-03-16T19:32:06.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7faa44a22640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa38045aa0 0x7faa380d94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7faa44a22640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa3804c3e0 0x7faa3817c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7faa44a22640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa380d3e90 0x7faa380cf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7faa44a22640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa380d8290 con 0x7faa380d3e90 2024-03-16T19:32:06.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7faa3e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa3804c3e0 0x7faa3817c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba4148e30 con 0x7fcba40b1ef0 2024-03-16T19:32:06.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fa25ffff640 1 -- 172.21.15.172:0/3723432955 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2680052d0 con 0x7fa2780c6a30 2024-03-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fa25ffff640 1 -- 172.21.15.172:0/3723432955 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa268005430 con 0x7fa2780c6a30 2024-03-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fa25ffff640 1 -- 172.21.15.172:0/3723432955 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268012870 con 0x7fa2780c6a30 2024-03-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fa27e207640 1 -- 172.21.15.172:0/3723432955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2781f48d0 con 0x7fa2780c6a30 2024-03-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fa27e207640 1 -- 172.21.15.172:0/3723432955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2781e7ae0 con 0x7fa2780c6a30 2024-03-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7fc16bfff640 1 -- 172.21.15.172:0/3457261894 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc17400e960 con 0x7fc1780dba20 2024-03-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7fc16bfff640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc16406b3e0 0x7fc16406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7fc16bfff640 1 -- 172.21.15.172:0/3457261894 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fc17408a1f0 con 0x7fc1780dba20 2024-03-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7faa3e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa3804c3e0 0x7faa3817c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40788/0 (socket says 172.21.15.172:40788) 2024-03-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7faa3e59c640 1 -- 172.21.15.172:0/637891032 learned_addr learned my addr 172.21.15.172:0/637891032 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428016974 0 0) 0x7f5eb01594f0 con 0x7f5eb0159110 2024-03-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e98003520 con 0x7f5eb0159110 2024-03-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554262228 0 0) 0x7f5e98003520 con 0x7f5eb0159110 2024-03-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 >> v2:172.21.15.172:3300/0 conn(0x7f5eb0073180 msgr2=0x7f5eb00c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 --2- 172.21.15.172:0/72823686 >> v2:172.21.15.172:3300/0 conn(0x7f5eb0073180 0x7f5eb00c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 >> v1:172.21.15.172:6789/0 conn(0x7f5eb015ed80 legacy=0x7f5eb006ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb006af70 con 0x7f5eb0159110 2024-03-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f5ea0001670 con 0x7f5eb0159110 2024-03-16T19:32:06.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 >> v1:172.21.15.174:6789/0 conn(0x7f5eb0159110 legacy=0x7f5eb0162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5e980039c0 0x7f5e98003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e980047c0 0x7f5e98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7fcf84ff9640 1 -- 172.21.15.172:0/948528456 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fcf7000f850 con 0x7fcf880c3ea0 2024-03-16T19:32:06.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc178047470 con 0x7fc1780dba20 2024-03-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980051d0 0x7f5e9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9800c0f0 con 0x7f5e980051d0 2024-03-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9801d270 con 0x7f5e980047c0 2024-03-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9801d430 con 0x7f5e980039c0 2024-03-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f5eae59c640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5e980039c0 0x7f5e98003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7f5eadd9b640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e980047c0 0x7f5e98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7f5eaed9d640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980051d0 0x7f5e9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f1c4a719640 1 -- 172.21.15.172:0/547133525 wait complete. 2024-03-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7f1c4a719640 1 Processor -- start 2024-03-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7f1c4a719640 1 -- start start 2024-03-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7f1c4a719640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c44073180 0x7f1c44157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7f1c4a719640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c44159110 0x7f1c441e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.185+0000 7f1c4a719640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 0x7f1c44155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f1c4a719640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c441ea120 con 0x7f1c4415ed80 2024-03-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f1c4a719640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c44155660 con 0x7f1c44159110 2024-03-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f1c4a719640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c44155940 con 0x7f1c44073180 2024-03-16T19:32:06.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f1c437fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c44159110 0x7f1c441e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f1c437fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c44159110 0x7f1c441e8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40770/0 (socket says 172.21.15.172:40770) 2024-03-16T19:32:06.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f1c437fe640 1 -- 172.21.15.172:0/1376282271 learned_addr learned my addr 172.21.15.172:0/1376282271 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7f1c48cb6640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 0x7f1c44155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7fc17effd640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc16406b3e0 0x7fc16406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.218 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7faa3ed9d640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa38045aa0 0x7faa380d94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f1da5a71640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da014efe0 0x7f1da01f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f1da5a71640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da014efe0 0x7f1da01f0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56590/0 (socket says 172.21.15.172:56590) 2024-03-16T19:32:06.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.186+0000 7f1da5a71640 1 -- 172.21.15.172:0/1718870420 learned_addr learned my addr 172.21.15.172:0/1718870420 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7faa3f59e640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa380d3e90 0x7faa380cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa380dc320 con 0x7faa3804c3e0 2024-03-16T19:32:06.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3817cfe0 con 0x7faa38045aa0 2024-03-16T19:32:06.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.182+0000 7f1da5270640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da00b0de0 0x7f1da00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.184+0000 7f1da4a6f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1da00ce4e0 0x7f1da01565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7fcf84ff9640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcf6806b3b0 0x7fcf6806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7fcf8da84640 1 -- 172.21.15.172:0/948528456 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8814f130 con 0x7fcf880c3ea0 2024-03-16T19:32:06.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7f1da5a71640 1 -- 172.21.15.172:0/1718870420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1da00ce4e0 msgr2=0x7f1da01565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7f1da5a71640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1da00ce4e0 0x7f1da01565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7f1da5a71640 1 -- 172.21.15.172:0/1718870420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da00b0de0 msgr2=0x7f1da00b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7f1da5a71640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da00b0de0 0x7f1da00b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7f1da5a71640 1 -- 172.21.15.172:0/1718870420 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d98009b30 con 0x7f1da014efe0 2024-03-16T19:32:06.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7fcf84ff9640 1 -- 172.21.15.172:0/948528456 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fcf70091fa0 con 0x7fcf880c3ea0 2024-03-16T19:32:06.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.187+0000 7fcf877fe640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcf6806b3b0 0x7fcf6806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f1c437fe640 1 -- 172.21.15.172:0/1376282271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c44073180 msgr2=0x7f1c44157dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T19:32:06.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f1c437fe640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c44073180 0x7f1c44157dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f1c437fe640 1 -- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 msgr2=0x7f1c44155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7fc17effd640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc16406b3e0 0x7fc16406d8a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc170004740 tx=0x7fc1800b7040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7faa3e59c640 1 -- 172.21.15.172:0/637891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa38045aa0 msgr2=0x7faa380d94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7faa3e59c640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa38045aa0 0x7faa380d94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7faa3e59c640 1 -- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa380d3e90 msgr2=0x7faa380cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7faa3e59c640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa380d3e90 0x7faa380cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7faa3e59c640 1 -- 172.21.15.172:0/637891032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40073d10 con 0x7faa3804c3e0 2024-03-16T19:32:06.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f5eae59c640 1 -- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e980047c0 msgr2=0x7f5e98004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f5eae59c640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e980047c0 0x7f5e98004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f5eae59c640 1 -- 172.21.15.172:0/72823686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980051d0 msgr2=0x7f5e9800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:06.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f5eae59c640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980051d0 0x7f5e9800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f5eae59c640 1 -- 172.21.15.172:0/72823686 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e98003520 con 0x7f5e980039c0 2024-03-16T19:32:06.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f5eaed9d640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980051d0 0x7f5e9800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:06.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f5eaed9d640 1 -- 172.21.15.172:0/72823686 reap_dead start 2024-03-16T19:32:06.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7faa3e59c640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa3804c3e0 0x7faa3817c720 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faa40075730 tx=0x7faa40068590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7fa25ffff640 1 -- 172.21.15.172:0/3723432955 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa2680129d0 con 0x7fa2780c6a30 2024-03-16T19:32:06.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7fcb99ffb640 1 -- 172.21.15.172:0/1037635362 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fcba000bcf0 con 0x7fcba40b1ef0 2024-03-16T19:32:06.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb84005290 con 0x7fcba40b1ef0 2024-03-16T19:32:06.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f1c437fe640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 0x7f1c44155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7fcf877fe640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcf6806b3b0 0x7fcf6806d870 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcf78035000 tx=0x7fcf78030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7fa25dffb640 1 -- 172.21.15.172:0/3723432955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa278075a70 con 0x7fa2780c6a30 2024-03-16T19:32:06.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7fcb99ffb640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcb8806b400 0x7fcb8806d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f1da5a71640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da014efe0 0x7f1da01f0330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800efc0 tx=0x7f1d9800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f1c437fe640 1 -- 172.21.15.172:0/1376282271 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c009400 con 0x7f1c44159110 2024-03-16T19:32:06.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7fcb99ffb640 1 -- 172.21.15.172:0/1037635362 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fcba008a170 con 0x7fcba40b1ef0 2024-03-16T19:32:06.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f1c48cb6640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 0x7f1c44155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:06.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.188+0000 7f1c437fe640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c44159110 0x7f1c441e8360 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c0098c0 tx=0x7f1c2c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.189+0000 7fa25ffff640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa26406b400 0x7fa26406d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.189+0000 7fcba8a88640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcb8806b400 0x7fcb8806d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.189+0000 7fa25ffff640 1 -- 172.21.15.172:0/3723432955 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7fa268089620 con 0x7fa2780c6a30 2024-03-16T19:32:06.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.189+0000 7fa276ffd640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa26406b400 0x7fa26406d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.189+0000 7f5eae59c640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5e980039c0 0x7f5e98003de0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0007ee0 tx=0x7f5e980057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=169068c7eb5cfa9c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.189+0000 7fcba8a88640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcb8806b400 0x7fcb8806d8c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcba4147b40 tx=0x7fcb94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.189+0000 7fa276ffd640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa26406b400 0x7fa26406d8c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa2780cc000 tx=0x7fa27000b550 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.192+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea002c850 con 0x7f5e980039c0 2024-03-16T19:32:06.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.192+0000 7f1d95ffb640 1 -- 172.21.15.172:0/1718870420 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d98018070 con 0x7f1da014efe0 2024-03-16T19:32:06.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.192+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d90009400 con 0x7f1da014efe0 2024-03-16T19:32:06.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.192+0000 7fc16bfff640 1 -- 172.21.15.172:0/3457261894 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc174054e00 con 0x7fc1780dba20 2024-03-16T19:32:06.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.192+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da00b3af0 con 0x7f1da014efe0 2024-03-16T19:32:06.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.193+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da0075a70 con 0x7f1da014efe0 2024-03-16T19:32:06.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.196+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea002c9d0 con 0x7f5e980039c0 2024-03-16T19:32:06.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.196+0000 7f1d95ffb640 1 -- 172.21.15.172:0/1718870420 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9800f040 con 0x7f1da014efe0 2024-03-16T19:32:06.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.196+0000 7f1d95ffb640 1 -- 172.21.15.172:0/1718870420 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9800cde0 con 0x7f1da014efe0 2024-03-16T19:32:06.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.196+0000 7fa25ffff640 1 -- 172.21.15.172:0/3723432955 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa268054290 con 0x7fa2780c6a30 2024-03-16T19:32:06.232 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7f5ead59a640 1 -- 172.21.15.172:0/72823686 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea002cd80 con 0x7f5e980039c0 2024-03-16T19:32:06.233 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7fcf84ff9640 1 -- 172.21.15.172:0/948528456 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf7005cc40 con 0x7fcf880c3ea0 2024-03-16T19:32:06.233 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7f5eb4a81640 1 -- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5e980039c0 msgr2=0x7f5e98003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.233 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7f5eb4a81640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5e980039c0 0x7f5e98003de0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0007ee0 tx=0x7f5e980057f0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.233 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7f5eb4a81640 1 -- 172.21.15.172:0/72823686 shutdown_connections 2024-03-16T19:32:06.233 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7f5eb4a81640 1 --2- 172.21.15.172:0/72823686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5e980039c0 0x7f5e98003de0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7f5eb4a81640 1 -- 172.21.15.172:0/72823686 >> 172.21.15.172:0/72823686 conn(0x7f5eb00cc030 msgr2=0x7f5eb00cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7f5eb4a81640 1 -- 172.21.15.172:0/72823686 shutdown_connections 2024-03-16T19:32:06.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7f5eb4a81640 1 -- 172.21.15.172:0/72823686 wait complete. 2024-03-16T19:32:06.234 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.197+0000 7f1d95ffb640 1 -- 172.21.15.172:0/1718870420 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1d98003990 con 0x7f1da014efe0 2024-03-16T19:32:06.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.198+0000 7f1d95ffb640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1d8406b3e0 0x7f1d8406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.198+0000 7f1d95ffb640 1 -- 172.21.15.172:0/1718870420 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f1d9808aa60 con 0x7f1da014efe0 2024-03-16T19:32:06.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.198+0000 7f1da5270640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1d8406b3e0 0x7f1d8406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.198+0000 7f1da5270640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1d8406b3e0 0x7f1d8406d8a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1d90036000 tx=0x7f1d90008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.199+0000 7f5eb4a81640 1 Processor -- start 2024-03-16T19:32:06.235 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.199+0000 7f5eb4a81640 1 -- start start 2024-03-16T19:32:06.236 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.200+0000 7f1d95ffb640 1 -- 172.21.15.172:0/1718870420 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d98055710 con 0x7f1da014efe0 2024-03-16T19:32:06.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eb4a81640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980039c0 0x7f5eb00c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eb4a81640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb00cb7f0 0x7f5eb0160e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eae59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980039c0 0x7f5eb00c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eae59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980039c0 0x7f5eb00c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39250/0 (socket says 172.21.15.172:39250) 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eae59c640 1 -- 172.21.15.172:0/359503170 learned_addr learned my addr 172.21.15.172:0/359503170 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eb4a81640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5eb0160670 0x7f5eb014f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb014c6f0 con 0x7f5e980039c0 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb0151ad0 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb014c920 con 0x7f5eb0160670 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eadd9b640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb00cb7f0 0x7f5eb0160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.201+0000 7f5eaed9d640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5eb0160670 0x7f5eb014f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.202+0000 7f5eadd9b640 1 -- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5eb0160670 msgr2=0x7f5eb014f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.202+0000 7f5eadd9b640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5eb0160670 0x7f5eb014f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.202+0000 7f5eadd9b640 1 -- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980039c0 msgr2=0x7f5eb00c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.202+0000 7f5eadd9b640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980039c0 0x7f5eb00c8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.202+0000 7f5eadd9b640 1 -- 172.21.15.172:0/359503170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb0150f00 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.202+0000 7f5eae59c640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980039c0 0x7f5eb00c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:06.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.202+0000 7f5eadd9b640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb00cb7f0 0x7f5eb0160e10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00649e0 tx=0x7f5ea40019e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.205+0000 7f5e9effd640 1 -- 172.21.15.172:0/359503170 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4011580 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.205+0000 7fcb99ffb640 1 -- 172.21.15.172:0/1037635362 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcba0054dc0 con 0x7fcba40b1ef0 2024-03-16T19:32:06.239 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.205+0000 7f5e9effd640 1 -- 172.21.15.172:0/359503170 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea40116e0 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.240 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.205+0000 7f5e9effd640 1 -- 172.21.15.172:0/359503170 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4011a50 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.240 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.205+0000 7f1c40ff9640 1 -- 172.21.15.172:0/1376282271 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c03bc70 con 0x7f1c44159110 2024-03-16T19:32:06.240 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.205+0000 7f1c40ff9640 1 -- 172.21.15.172:0/1376282271 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2c03bdf0 con 0x7f1c44159110 2024-03-16T19:32:06.240 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.205+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c441e93a0 con 0x7f1c44159110 2024-03-16T19:32:06.240 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.205+0000 7f1c40ff9640 1 -- 172.21.15.172:0/1376282271 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c045630 con 0x7f1c44159110 2024-03-16T19:32:06.240 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.205+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c441ea650 con 0x7f1c44159110 2024-03-16T19:32:06.240 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.206+0000 7faa277fe640 1 -- 172.21.15.172:0/637891032 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40070070 con 0x7faa3804c3e0 2024-03-16T19:32:06.241 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.206+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb0151f60 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.241 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.206+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb0151480 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.241 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.206+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa34009400 con 0x7faa3804c3e0 2024-03-16T19:32:06.241 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.206+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa380d7c40 con 0x7faa3804c3e0 2024-03-16T19:32:06.241 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.206+0000 7faa277fe640 1 -- 172.21.15.172:0/637891032 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa400767b0 con 0x7faa3804c3e0 2024-03-16T19:32:06.241 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.206+0000 7faa277fe640 1 -- 172.21.15.172:0/637891032 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40091070 con 0x7faa3804c3e0 2024-03-16T19:32:06.242 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.207+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c44075a70 con 0x7f1c44159110 2024-03-16T19:32:06.242 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.207+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa380e30e0 con 0x7faa3804c3e0 2024-03-16T19:32:06.242 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.207+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb0075a70 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.243 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.208+0000 7f1c40ff9640 1 -- 172.21.15.172:0/1376282271 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1c2c02f050 con 0x7f1c44159110 2024-03-16T19:32:06.243 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.209+0000 7f1c40ff9640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2406b480 0x7f1c2406d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.243 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.208+0000 7faa277fe640 1 -- 172.21.15.172:0/637891032 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7faa40096420 con 0x7faa3804c3e0 2024-03-16T19:32:06.243 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.209+0000 7faa277fe640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2c06b480 0x7faa2c06d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.209+0000 7f1c40ff9640 1 -- 172.21.15.172:0/1376282271 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f1c2c0abb50 con 0x7f1c44159110 2024-03-16T19:32:06.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.209+0000 7f5e9effd640 1 -- 172.21.15.172:0/359503170 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5ea4011bb0 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.209+0000 7faa3ed9d640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2c06b480 0x7faa2c06d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.209+0000 7f5e9effd640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5e9006b3e0 0x7f5e9006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:06.245 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.209+0000 7f1c43fff640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2406b480 0x7f1c2406d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.245 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.210+0000 7f5e9effd640 1 -- 172.21.15.172:0/359503170 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7f5ea4089e50 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.245 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.210+0000 7faa277fe640 1 -- 172.21.15.172:0/637891032 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5799+0+0 (secure 0 0 0) 0x7faa401055d0 con 0x7faa3804c3e0 2024-03-16T19:32:06.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.210+0000 7faa3ed9d640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2c06b480 0x7faa2c06d940 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faa34036000 tx=0x7faa3403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.210+0000 7f5eae59c640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5e9006b3e0 0x7f5e9006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:06.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.211+0000 7f5eae59c640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5e9006b3e0 0x7f5e9006d8a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5ea002cfd0 tx=0x7f5ea0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.246 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.211+0000 7f1c43fff640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2406b480 0x7f1c2406d940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c001590 tx=0x7f1c3c0078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:06.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.217+0000 7f5e9effd640 1 -- 172.21.15.172:0/359503170 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ea4054af0 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.225+0000 7f1c40ff9640 1 -- 172.21.15.172:0/1376282271 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c2c0767c0 con 0x7f1c44159110 2024-03-16T19:32:06.247 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.234+0000 7faa277fe640 1 -- 172.21.15.172:0/637891032 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa400d01b0 con 0x7faa3804c3e0 2024-03-16T19:32:06.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.583+0000 7fa25dffb640 1 -- 172.21.15.172:0/3723432955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa2780c1280 con 0x7fa2780c6a30 2024-03-16T19:32:06.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.584+0000 7fa25ffff640 1 -- 172.21.15.172:0/3723432955 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa268057950 con 0x7fa2780c6a30 2024-03-16T19:32:06.565 INFO:teuthology.orchestra.run.smithi172.stdout:55834574887 2024-03-16T19:32:06.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.585+0000 7fcf8da84640 1 -- 172.21.15.172:0/948528456 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fcf88075a70 con 0x7fcf880c3ea0 2024-03-16T19:32:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.585+0000 7fcf84ff9640 1 -- 172.21.15.172:0/948528456 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcf70060300 con 0x7fcf880c3ea0 2024-03-16T19:32:06.566 INFO:teuthology.orchestra.run.smithi172.stdout:55834574886 2024-03-16T19:32:06.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.587+0000 7fa25dffb640 1 -- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa26406b400 msgr2=0x7fa26406d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.587+0000 7fa25dffb640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa26406b400 0x7fa26406d8c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa2780cc000 tx=0x7fa27000b550 comp rx=0 tx=0).stop 2024-03-16T19:32:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.587+0000 7fa25dffb640 1 -- 172.21.15.172:0/3723432955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa2780c6a30 msgr2=0x7fa2781e5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.588+0000 7fa25dffb640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa2780c6a30 0x7fa2781e5d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa26800c460 tx=0x7fa26800c920 comp rx=0 tx=0).stop 2024-03-16T19:32:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.588+0000 7fa25dffb640 1 -- 172.21.15.172:0/3723432955 shutdown_connections 2024-03-16T19:32:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.588+0000 7fa25dffb640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa278150da0 0x7fa2781e86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.588+0000 7fa25dffb640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2780c85b0 0x7fa2781f5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.588+0000 7fa25dffb640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa26406b400 0x7fa26406d8c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.588+0000 7fa25dffb640 1 --2- 172.21.15.172:0/3723432955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa2780c6a30 0x7fa2781e5d40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.588+0000 7fa25dffb640 1 -- 172.21.15.172:0/3723432955 >> 172.21.15.172:0/3723432955 conn(0x7fa2780cc030 msgr2=0x7fa27814ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.588+0000 7fa25dffb640 1 -- 172.21.15.172:0/3723432955 shutdown_connections 2024-03-16T19:32:06.570 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.588+0000 7fa25dffb640 1 -- 172.21.15.172:0/3723432955 wait complete. 2024-03-16T19:32:06.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 -- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcf6806b3b0 msgr2=0x7fcf6806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcf6806b3b0 0x7fcf6806d870 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcf78035000 tx=0x7fcf78030000 comp rx=0 tx=0).stop 2024-03-16T19:32:06.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 -- 172.21.15.172:0/948528456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf880c3ea0 msgr2=0x7fcf88155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf880c3ea0 0x7fcf88155cb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcf70011f20 tx=0x7fcf70014550 comp rx=0 tx=0).stop 2024-03-16T19:32:06.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 -- 172.21.15.172:0/948528456 shutdown_connections 2024-03-16T19:32:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf88151910 0x7fcf881e2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf880c3ea0 0x7fcf88155cb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcf6806b3b0 0x7fcf6806d870 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 --2- 172.21.15.172:0/948528456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf880c3930 0x7fcf881de240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 -- 172.21.15.172:0/948528456 >> 172.21.15.172:0/948528456 conn(0x7fcf8806db70 msgr2=0x7fcf8806bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.590+0000 7fcf7dffb640 1 -- 172.21.15.172:0/948528456 shutdown_connections 2024-03-16T19:32:06.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.592+0000 7fcf7dffb640 1 -- 172.21.15.172:0/948528456 wait complete. 2024-03-16T19:32:06.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.599+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f1da01effe0 con 0x7f1da014efe0 2024-03-16T19:32:06.581 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.600+0000 7f1d95ffb640 1 -- 172.21.15.172:0/1718870420 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1d98025390 con 0x7f1da014efe0 2024-03-16T19:32:06.581 INFO:teuthology.orchestra.run.smithi172.stdout:55834574886 2024-03-16T19:32:06.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.602+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc178005380 con 0x7fc1780dba20 2024-03-16T19:32:06.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.603+0000 7fc16bfff640 1 -- 172.21.15.172:0/3457261894 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc1740584c0 con 0x7fc1780dba20 2024-03-16T19:32:06.584 INFO:teuthology.orchestra.run.smithi172.stdout:55834574885 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.603+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1d8406b3e0 msgr2=0x7f1d8406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.603+0000 7f1da74d4640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1d8406b3e0 0x7f1d8406d8a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1d90036000 tx=0x7f1d90008030 comp rx=0 tx=0).stop 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.603+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da014efe0 msgr2=0x7f1da01f0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.603+0000 7f1da74d4640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da014efe0 0x7f1da01f0330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800efc0 tx=0x7f1d9800c490 comp rx=0 tx=0).stop 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.604+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 shutdown_connections 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.604+0000 7f1da74d4640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1d8406b3e0 0x7f1d8406d8a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.604+0000 7f1da74d4640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1da014efe0 0x7f1da01f0330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.604+0000 7f1da74d4640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1da00ce4e0 0x7f1da01565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.604+0000 7f1da74d4640 1 --2- 172.21.15.172:0/1718870420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1da00b0de0 0x7f1da00b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.604+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 >> 172.21.15.172:0/1718870420 conn(0x7f1da0066e00 msgr2=0x7f1da00c8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.604+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 shutdown_connections 2024-03-16T19:32:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.604+0000 7f1da74d4640 1 -- 172.21.15.172:0/1718870420 wait complete. 2024-03-16T19:32:06.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.606+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc16406b3e0 msgr2=0x7fc16406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.606+0000 7fc1849c1640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc16406b3e0 0x7fc16406d8a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc170004740 tx=0x7fc1800b7040 comp rx=0 tx=0).stop 2024-03-16T19:32:06.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1780dba20 msgr2=0x7fc1780dbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1780dba20 0x7fc1780dbe00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc174010890 tx=0x7fc174010d50 comp rx=0 tx=0).stop 2024-03-16T19:32:06.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 shutdown_connections 2024-03-16T19:32:06.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc178048c00 0x7fc1780d5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc178047e40 0x7fc1780486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc16406b3e0 0x7fc16406d8a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 --2- 172.21.15.172:0/3457261894 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1780dba20 0x7fc1780dbe00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 >> 172.21.15.172:0/3457261894 conn(0x7fc178050170 msgr2=0x7fc1780e2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.589 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 shutdown_connections 2024-03-16T19:32:06.590 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.607+0000 7fc1849c1640 1 -- 172.21.15.172:0/3457261894 wait complete. 2024-03-16T19:32:06.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574888 got 55834574887 for osd.1 2024-03-16T19:32:06.591 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574886 for osd.3 2024-03-16T19:32:06.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.616+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fcb84005810 con 0x7fcba40b1ef0 2024-03-16T19:32:06.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.616+0000 7fcb99ffb640 1 -- 172.21.15.172:0/1037635362 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcba0058480 con 0x7fcba40b1ef0 2024-03-16T19:32:06.597 INFO:teuthology.orchestra.run.smithi172.stdout:55834574885 2024-03-16T19:32:06.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.619+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcb8806b400 msgr2=0x7fcb8806d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.619+0000 7fcbaa28b640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcb8806b400 0x7fcb8806d8c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcba4147b40 tx=0x7fcb94009040 comp rx=0 tx=0).stop 2024-03-16T19:32:06.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.619+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40b1ef0 msgr2=0x7fcba4152d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.619+0000 7fcbaa28b640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40b1ef0 0x7fcba4152d50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcba000d800 tx=0x7fcba000dcc0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.619+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5eb0072d50 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.620+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 shutdown_connections 2024-03-16T19:32:06.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.620+0000 7fcbaa28b640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcba41518f0 0x7fcba41f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.620+0000 7fcbaa28b640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fcb8806b400 0x7fcb8806d8c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.620+0000 7fcbaa28b640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcba40c7400 0x7fcba4070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.620+0000 7fcbaa28b640 1 --2- 172.21.15.172:0/1037635362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcba40b1ef0 0x7fcba4152d50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.620+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 >> 172.21.15.172:0/1037635362 conn(0x7fcba40cbfc0 msgr2=0x7fcba40729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.620+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 shutdown_connections 2024-03-16T19:32:06.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.620+0000 7fcbaa28b640 1 -- 172.21.15.172:0/1037635362 wait complete. 2024-03-16T19:32:06.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.620+0000 7f5e9effd640 1 -- 172.21.15.172:0/359503170 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5ea403d070 con 0x7f5eb00cb7f0 2024-03-16T19:32:06.602 INFO:teuthology.orchestra.run.smithi172.stdout:55834574885 2024-03-16T19:32:06.603 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574886 for osd.6 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5e9006b3e0 msgr2=0x7f5e9006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5e9006b3e0 0x7f5e9006d8a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5ea002cfd0 tx=0x7f5ea0009030 comp rx=0 tx=0).stop 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb00cb7f0 msgr2=0x7f5eb0160e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb00cb7f0 0x7f5eb0160e10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00649e0 tx=0x7f5ea40019e0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 shutdown_connections 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5e9006b3e0 0x7f5e9006d8a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5eb0160670 0x7f5eb014f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb00cb7f0 0x7f5eb0160e10 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 --2- 172.21.15.172:0/359503170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5e980039c0 0x7f5eb00c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.622+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 >> 172.21.15.172:0/359503170 conn(0x7f5eb00cc030 msgr2=0x7f5eb00c6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.623+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 shutdown_connections 2024-03-16T19:32:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.623+0000 7f5eb4a81640 1 -- 172.21.15.172:0/359503170 wait complete. 2024-03-16T19:32:06.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.624+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1c44072d50 con 0x7f1c44159110 2024-03-16T19:32:06.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574886 got 55834574885 for osd.4 2024-03-16T19:32:06.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.624+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7faa380057d0 con 0x7faa3804c3e0 2024-03-16T19:32:06.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.624+0000 7f1c40ff9640 1 -- 172.21.15.172:0/1376282271 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1c2c035090 con 0x7f1c44159110 2024-03-16T19:32:06.606 INFO:teuthology.orchestra.run.smithi172.stdout:377957122052 2024-03-16T19:32:06.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.624+0000 7faa277fe640 1 -- 172.21.15.172:0/637891032 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faa400d3870 con 0x7faa3804c3e0 2024-03-16T19:32:06.607 INFO:teuthology.orchestra.run.smithi172.stdout:55834574884 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2c06b480 msgr2=0x7faa2c06d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2c06b480 0x7faa2c06d940 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faa34036000 tx=0x7faa3403b040 comp rx=0 tx=0).stop 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa3804c3e0 msgr2=0x7faa3817c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa3804c3e0 0x7faa3817c720 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faa40075730 tx=0x7faa40068590 comp rx=0 tx=0).stop 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 shutdown_connections 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa380d3e90 0x7faa380cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa3804c3e0 0x7faa3817c720 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7faa2c06b480 0x7faa2c06d940 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 --2- 172.21.15.172:0/637891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa38045aa0 0x7faa380d94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 >> 172.21.15.172:0/637891032 conn(0x7faa380505c0 msgr2=0x7faa380059e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 shutdown_connections 2024-03-16T19:32:06.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7faa44a22640 1 -- 172.21.15.172:0/637891032 wait complete. 2024-03-16T19:32:06.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2406b480 msgr2=0x7f1c2406d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7f1c4a719640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2406b480 0x7f1c2406d940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c001590 tx=0x7f1c3c0078f0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c44159110 msgr2=0x7f1c441e8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7f1c4a719640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c44159110 0x7f1c441e8360 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c0098c0 tx=0x7f1c2c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 shutdown_connections 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.627+0000 7f1c4a719640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c4415ed80 0x7f1c44155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.628+0000 7f1c4a719640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c44159110 0x7f1c441e8360 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.628+0000 7f1c4a719640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1c2406b480 0x7f1c2406d940 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.628+0000 7f1c4a719640 1 --2- 172.21.15.172:0/1376282271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c44073180 0x7f1c44157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.628+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 >> 172.21.15.172:0/1376282271 conn(0x7f1c440cc030 msgr2=0x7f1c440c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.628+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 shutdown_connections 2024-03-16T19:32:06.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:06.628+0000 7f1c4a719640 1 -- 172.21.15.172:0/1376282271 wait complete. 2024-03-16T19:32:06.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574886 got 55834574885 for osd.0 2024-03-16T19:32:06.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574886 got 55834574885 for osd.7 2024-03-16T19:32:06.621 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574885 got 55834574884 for osd.5 2024-03-16T19:32:06.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 377957122053 got 377957122052 for osd.2 2024-03-16T19:32:07.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-16T19:32:07.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-16T19:32:07.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-16T19:32:07.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-16T19:32:07.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-16T19:32:07.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-16T19:32:07.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-16T19:32:07.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-16T19:32:07.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.773+0000 7f5b38863640 1 Processor -- start 2024-03-16T19:32:07.754 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.773+0000 7f5b38863640 1 -- start start 2024-03-16T19:32:07.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.774+0000 7f5b38863640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5b340681c0 0x7f5b340c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.774+0000 7f5b38863640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5b34150cc0 0x7f5b3406e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.755 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.774+0000 7f5b38863640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b340c86a0 con 0x7f5b341508a0 2024-03-16T19:32:07.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.774+0000 7f5b38863640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b3415ac50 con 0x7f5b34150cc0 2024-03-16T19:32:07.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.774+0000 7f5b38863640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b3415a970 con 0x7f5b340681c0 2024-03-16T19:32:07.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.775+0000 7f5b3359e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5b34150cc0 0x7f5b3406e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.775+0000 7f5b3359e640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5b34150cc0 0x7f5b3406e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39264/0 (socket says 172.21.15.172:39264) 2024-03-16T19:32:07.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.775+0000 7f5b3359e640 1 -- 172.21.15.172:0/1764556364 learned_addr learned my addr 172.21.15.172:0/1764556364 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.757 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.774+0000 7f83d4450640 1 Processor -- start 2024-03-16T19:32:07.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.777+0000 7f5b3359e640 1 -- 172.21.15.172:0/1764556364 >> v2:172.21.15.174:3300/0 conn(0x7f5b340681c0 msgr2=0x7f5b340c7fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:32:07.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.777+0000 7f5b3359e640 1 --2- 172.21.15.172:0/1764556364 >> v2:172.21.15.174:3300/0 conn(0x7f5b340681c0 0x7f5b340c7fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.777+0000 7f5b3359e640 1 -- 172.21.15.172:0/1764556364 >> v1:172.21.15.174:6789/0 conn(0x7f5b341508a0 legacy=0x7f5b340c8e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T19:32:07.758 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.777+0000 7f5b3359e640 1 -- 172.21.15.172:0/1764556364 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b340c71b0 con 0x7f5b34150cc0 2024-03-16T19:32:07.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.774+0000 7f83d4450640 1 -- start start 2024-03-16T19:32:07.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.775+0000 7f83d4450640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83c40462f0 0x7f83c4007d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.775+0000 7f83d4450640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c404bc20 0x7f83c40470a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.775+0000 7f83d4450640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c40078d0 con 0x7f83c40dd6b0 2024-03-16T19:32:07.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.775+0000 7f83d4450640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c40dda90 con 0x7f83c40462f0 2024-03-16T19:32:07.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.775+0000 7f83d4450640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c404c090 con 0x7f83c404bc20 2024-03-16T19:32:07.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.776+0000 7f83d19eb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83c40462f0 0x7f83c4007d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.760 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.776+0000 7f83d19eb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f83c40462f0 0x7f83c4007d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39272/0 (socket says 172.21.15.172:39272) 2024-03-16T19:32:07.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.776+0000 7f83d19eb640 1 -- 172.21.15.172:0/1075515304 learned_addr learned my addr 172.21.15.172:0/1075515304 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.776+0000 7f83d29ed640 1 --2- 172.21.15.172:0/1075515304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c404bc20 0x7f83c40470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.776+0000 7f83d29ed640 1 -- 172.21.15.172:0/1075515304 >> v2:172.21.15.172:3300/0 conn(0x7f83c40462f0 msgr2=0x7f83c4007d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:07.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.776+0000 7f83d29ed640 1 --2- 172.21.15.172:0/1075515304 >> v2:172.21.15.172:3300/0 conn(0x7f83c40462f0 0x7f83c4007d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.776+0000 7f83d29ed640 1 -- 172.21.15.172:0/1075515304 >> v1:172.21.15.172:6789/0 conn(0x7f83c40dd6b0 legacy=0x7f83c4047be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:07.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.776+0000 7f83d29ed640 1 -- 172.21.15.172:0/1075515304 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c4006fe0 con 0x7f83c404bc20 2024-03-16T19:32:07.761 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.778+0000 7f83d29ed640 1 --2- 172.21.15.172:0/1075515304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c404bc20 0x7f83c40470a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f83c80097a0 tx=0x7f83c802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88cbb33c479bdb34 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.778+0000 7f5b3359e640 1 --2- 172.21.15.172:0/1764556364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b34150cc0 0x7f5b3406e150 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5b340734d0 tx=0x7f5b2c02c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8657686d8fb60a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.779+0000 7f5b3259c640 1 -- 172.21.15.172:0/1764556364 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c02cb90 con 0x7f5b34150cc0 2024-03-16T19:32:07.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.779+0000 7f5b3259c640 1 -- 172.21.15.172:0/1764556364 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2c02ccf0 con 0x7f5b34150cc0 2024-03-16T19:32:07.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.780+0000 7f5b38863640 1 -- 172.21.15.172:0/1764556364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b34150cc0 msgr2=0x7f5b3406e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.780+0000 7f5b38863640 1 --2- 172.21.15.172:0/1764556364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b34150cc0 0x7f5b3406e150 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5b340734d0 tx=0x7f5b2c02c1f0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.779+0000 7f83d11ea640 1 -- 172.21.15.172:0/1075515304 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c802e070 con 0x7f83c404bc20 2024-03-16T19:32:07.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.779+0000 7f83d11ea640 1 -- 172.21.15.172:0/1075515304 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c80315a0 con 0x7f83c404bc20 2024-03-16T19:32:07.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.780+0000 7f83d4450640 1 -- 172.21.15.172:0/1075515304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c404bc20 msgr2=0x7f83c40470a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.780+0000 7f83d4450640 1 --2- 172.21.15.172:0/1075515304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c404bc20 0x7f83c40470a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f83c80097a0 tx=0x7f83c802b840 comp rx=0 tx=0).stop 2024-03-16T19:32:07.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.780+0000 7f5b38863640 1 -- 172.21.15.172:0/1764556364 shutdown_connections 2024-03-16T19:32:07.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.780+0000 7f83d4450640 1 -- 172.21.15.172:0/1075515304 shutdown_connections 2024-03-16T19:32:07.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.780+0000 7f83d4450640 1 --2- 172.21.15.172:0/1075515304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c404bc20 0x7f83c40470a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.781+0000 7f83d4450640 1 --2- 172.21.15.172:0/1075515304 >> v2:172.21.15.172:3300/0 conn(0x7f83c40462f0 0x7f83c4007d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.781+0000 7f83d4450640 1 -- 172.21.15.172:0/1075515304 >> 172.21.15.172:0/1075515304 conn(0x7f83c404dd20 msgr2=0x7f83c404e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:07.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.780+0000 7f5b38863640 1 --2- 172.21.15.172:0/1764556364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b34150cc0 0x7f5b3406e150 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.781+0000 7f5b38863640 1 --2- 172.21.15.172:0/1764556364 >> v2:172.21.15.174:3300/0 conn(0x7f5b340681c0 0x7f5b340c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.781+0000 7f83d4450640 1 -- 172.21.15.172:0/1075515304 shutdown_connections 2024-03-16T19:32:07.764 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.781+0000 7f5b38863640 1 -- 172.21.15.172:0/1764556364 >> 172.21.15.172:0/1764556364 conn(0x7f5b3406aed0 msgr2=0x7f5b3406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:07.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.781+0000 7f83d4450640 1 -- 172.21.15.172:0/1075515304 wait complete. 2024-03-16T19:32:07.765 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.782+0000 7f5b38863640 1 -- 172.21.15.172:0/1764556364 shutdown_connections 2024-03-16T19:32:07.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.783+0000 7f83d4450640 1 Processor -- start 2024-03-16T19:32:07.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.783+0000 7f83d4450640 1 -- start start 2024-03-16T19:32:07.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.782+0000 7f5b38863640 1 -- 172.21.15.172:0/1764556364 wait complete. 2024-03-16T19:32:07.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.784+0000 7f5b38863640 1 Processor -- start 2024-03-16T19:32:07.766 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.784+0000 7f83d4450640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c40462f0 0x7f83c40488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.767 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.784+0000 7f5b38863640 1 -- start start 2024-03-16T19:32:07.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.784+0000 7f83d4450640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c404bc20 0x7f83c4049190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f83d19eb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c404bc20 0x7f83c4049190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f83d19eb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c404bc20 0x7f83c4049190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39284/0 (socket says 172.21.15.172:39284) 2024-03-16T19:32:07.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f83d19eb640 1 -- 172.21.15.172:0/2777552066 learned_addr learned my addr 172.21.15.172:0/2777552066 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f83d4450640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83c40dd6b0 0x7f83c40496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f83d4450640 1 -- 172.21.15.172:0/2777552066 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c4048e10 con 0x7f83c404bc20 2024-03-16T19:32:07.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f5b38863640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5b340681c0 0x7f5b3414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f5b38863640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b341508a0 0x7f5b34153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.768 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f5b38863640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b34150cc0 0x7f5b34145bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f5b38863640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b340c3590 con 0x7f5b341508a0 2024-03-16T19:32:07.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f5b38863640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b34159910 con 0x7f5b34150cc0 2024-03-16T19:32:07.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f5b38863640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b34159e40 con 0x7f5b340681c0 2024-03-16T19:32:07.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f83d4450640 1 -- 172.21.15.172:0/2777552066 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40341a0 con 0x7f83c40dd6b0 2024-03-16T19:32:07.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b32d9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5b340681c0 0x7f5b3414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b32d9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5b340681c0 0x7f5b3414ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56650/0 (socket says 172.21.15.172:56650) 2024-03-16T19:32:07.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b2bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b341508a0 0x7f5b34153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b32d9d640 1 -- 172.21.15.172:0/801101217 learned_addr learned my addr 172.21.15.172:0/801101217 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b3359e640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b34150cc0 0x7f5b34145bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b32d9d640 1 -- 172.21.15.172:0/801101217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b34150cc0 msgr2=0x7f5b34145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b32d9d640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b34150cc0 0x7f5b34145bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b32d9d640 1 -- 172.21.15.172:0/801101217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b341508a0 msgr2=0x7f5b34153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b32d9d640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b341508a0 0x7f5b34153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.785+0000 7f83d4450640 1 -- 172.21.15.172:0/2777552066 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c4034690 con 0x7f83c40462f0 2024-03-16T19:32:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b32d9d640 1 -- 172.21.15.172:0/801101217 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c009b30 con 0x7f5b340681c0 2024-03-16T19:32:07.771 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.786+0000 7f5b32d9d640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5b340681c0 0x7f5b3414ab80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00b410 tx=0x7f5b1c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f83d29ed640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83c40dd6b0 0x7f83c40496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f83d21ec640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c40462f0 0x7f83c40488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f83d19eb640 1 -- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c40462f0 msgr2=0x7f83c40488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f83d19eb640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c40462f0 0x7f83c40488d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f83d19eb640 1 -- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83c40dd6b0 msgr2=0x7f83c40496d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f83d19eb640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83c40dd6b0 0x7f83c40496d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f83d19eb640 1 -- 172.21.15.172:0/2777552066 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c8009400 con 0x7f83c404bc20 2024-03-16T19:32:07.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f83d21ec640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c40462f0 0x7f83c40488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:32:07.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.788+0000 7f83d19eb640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c404bc20 0x7f83c4049190 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f83cc02d4f0 tx=0x7f83cc068590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f5b2b7fe640 1 -- 172.21.15.172:0/801101217 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c011830 con 0x7f5b340681c0 2024-03-16T19:32:07.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b34159b60 con 0x7f5b340681c0 2024-03-16T19:32:07.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.787+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b34152060 con 0x7f5b340681c0 2024-03-16T19:32:07.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.788+0000 7f5b2b7fe640 1 -- 172.21.15.172:0/801101217 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c011990 con 0x7f5b340681c0 2024-03-16T19:32:07.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.789+0000 7f5b2b7fe640 1 -- 172.21.15.172:0/801101217 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c011d40 con 0x7f5b340681c0 2024-03-16T19:32:07.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.789+0000 7f83c2ffd640 1 -- 172.21.15.172:0/2777552066 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc070070 con 0x7f83c404bc20 2024-03-16T19:32:07.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.789+0000 7f83d4450640 1 -- 172.21.15.172:0/2777552066 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c404a850 con 0x7f83c404bc20 2024-03-16T19:32:07.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.789+0000 7f83d4450640 1 -- 172.21.15.172:0/2777552066 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c40e6ef0 con 0x7f83c404bc20 2024-03-16T19:32:07.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.789+0000 7f83c2ffd640 1 -- 172.21.15.172:0/2777552066 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc0765a0 con 0x7f83c404bc20 2024-03-16T19:32:07.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.791+0000 7f5b2b7fe640 1 -- 172.21.15.172:0/801101217 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5b1c01a400 con 0x7f5b340681c0 2024-03-16T19:32:07.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.790+0000 7f83c2ffd640 1 -- 172.21.15.172:0/2777552066 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc091070 con 0x7f83c404bc20 2024-03-16T19:32:07.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.791+0000 7f5b2b7fe640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5b1806b480 0x7f5b1806d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.792+0000 7f5b2bfff640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5b1806b480 0x7f5b1806d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.774 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.792+0000 7f5b2b7fe640 1 -- 172.21.15.172:0/801101217 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f5b1c089dc0 con 0x7f5b340681c0 2024-03-16T19:32:07.775 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.792+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0c005290 con 0x7f5b340681c0 2024-03-16T19:32:07.775 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.792+0000 7f5b2bfff640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5b1806b480 0x7f5b1806d940 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5b341570f0 tx=0x7f5b240090e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.775 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.793+0000 7f83c2ffd640 1 -- 172.21.15.172:0/2777552066 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f83cc076950 con 0x7f83c404bc20 2024-03-16T19:32:07.775 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.793+0000 7f83c2ffd640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f83b006b4b0 0x7f83b006d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.793+0000 7f83d21ec640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f83b006b4b0 0x7f83b006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.794+0000 7f83d4450640 1 -- 172.21.15.172:0/2777552066 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b4005290 con 0x7f83c404bc20 2024-03-16T19:32:07.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.794+0000 7f83d21ec640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f83b006b4b0 0x7f83b006d970 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f83bc003eb0 tx=0x7f83bc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.794+0000 7f83c2ffd640 1 -- 172.21.15.172:0/2777552066 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f83cc105030 con 0x7f83c404bc20 2024-03-16T19:32:07.778 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.797+0000 7f5b2b7fe640 1 -- 172.21.15.172:0/801101217 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b1c015020 con 0x7f5b340681c0 2024-03-16T19:32:07.784 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.803+0000 7f83c2ffd640 1 -- 172.21.15.172:0/2777552066 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83cc0cfc60 con 0x7f83c404bc20 2024-03-16T19:32:07.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.830+0000 7f91b3fff640 1 Processor -- start 2024-03-16T19:32:07.811 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.830+0000 7f91b3fff640 1 -- start start 2024-03-16T19:32:07.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.831+0000 7f91b3fff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f91b406c610 0x7f91b40b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.831+0000 7f91b3fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91b4073000 0x7f91b406c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.831+0000 7f91b3fff640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b4072460 con 0x7f91b406d1c0 2024-03-16T19:32:07.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.831+0000 7f91b3fff640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b4168430 con 0x7f91b406c610 2024-03-16T19:32:07.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.831+0000 7f91b3fff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b406cf30 con 0x7f91b4073000 2024-03-16T19:32:07.814 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.831+0000 7f91b27fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f91b406d1c0 0x7f91b40728f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:38102/0 (socket says 172.21.15.172:38102) 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.831+0000 7f91b27fc640 1 -- 172.21.15.172:0/1410950231 learned_addr learned my addr 172.21.15.172:0/1410950231 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.831+0000 7f91b37fe640 1 --2- 172.21.15.172:0/1410950231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91b4073000 0x7f91b406c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.832+0000 7f91b2ffd640 1 --2- 172.21.15.172:0/1410950231 >> v2:172.21.15.172:3300/0 conn(0x7f91b406c610 0x7f91b40b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.832+0000 7f91b1ffb640 1 -- 172.21.15.172:0/1410950231 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315313903 0 0) 0x7f91b4072460 con 0x7f91b406d1c0 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.832+0000 7f91b1ffb640 1 -- 172.21.15.172:0/1410950231 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f919c003520 con 0x7f91b406d1c0 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.832+0000 7f91b2ffd640 1 -- 172.21.15.172:0/1410950231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91b4073000 msgr2=0x7f91b406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.832+0000 7f91b2ffd640 1 --2- 172.21.15.172:0/1410950231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91b4073000 0x7f91b406c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.832+0000 7f91b2ffd640 1 -- 172.21.15.172:0/1410950231 >> v1:172.21.15.172:6789/0 conn(0x7f91b406d1c0 legacy=0x7f91b40728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.832+0000 7f91b2ffd640 1 -- 172.21.15.172:0/1410950231 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b4167e90 con 0x7f91b406c610 2024-03-16T19:32:07.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.833+0000 7fa8d79ac640 1 Processor -- start 2024-03-16T19:32:07.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.833+0000 7f9dbf0f4640 1 Processor -- start 2024-03-16T19:32:07.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.833+0000 7fa8d79ac640 1 -- start start 2024-03-16T19:32:07.816 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.834+0000 7f91b2ffd640 1 --2- 172.21.15.172:0/1410950231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b406c610 0x7f91b40b1b60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f91a800fc10 tx=0x7f91a8034290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cb2f5031f441ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.833+0000 7f9dbf0f4640 1 -- start start 2024-03-16T19:32:07.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.834+0000 7f91b1ffb640 1 -- 172.21.15.172:0/1410950231 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8034e30 con 0x7f91b406c610 2024-03-16T19:32:07.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.834+0000 7f91b1ffb640 1 -- 172.21.15.172:0/1410950231 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a803c440 con 0x7f91b406c610 2024-03-16T19:32:07.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fa8d5748640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa8d0150cc0 0x7fa8d006aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:38116/0 (socket says 172.21.15.172:38116) 2024-03-16T19:32:07.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fa8d5748640 1 -- 172.21.15.172:0/3526515156 learned_addr learned my addr 172.21.15.172:0/3526515156 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:07.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f91b3fff640 1 -- 172.21.15.172:0/1410950231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b406c610 msgr2=0x7f91b40b1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f91b3fff640 1 --2- 172.21.15.172:0/1410950231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b406c610 0x7f91b40b1b60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f91a800fc10 tx=0x7f91a8034290 comp rx=0 tx=0).stop 2024-03-16T19:32:07.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f9dbf0f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db00dac60 0x7f9db004b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f9dbf0f4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db0007710 con 0x7f9db004b500 2024-03-16T19:32:07.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f9dbf0f4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db00068b0 con 0x7f9db0034a40 2024-03-16T19:32:07.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f9dbf0f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db004be20 con 0x7f9db00dac60 2024-03-16T19:32:07.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f9dbd8f1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db00dac60 0x7f9db004b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.833+0000 7fe419855640 1 Processor -- start 2024-03-16T19:32:07.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f9dbd8f1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db00dac60 0x7f9db004b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56680/0 (socket says 172.21.15.172:56680) 2024-03-16T19:32:07.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7f9dbd8f1640 1 -- 172.21.15.172:0/1516741584 learned_addr learned my addr 172.21.15.172:0/1516741584 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.833+0000 7fe419855640 1 -- start start 2024-03-16T19:32:07.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe419855640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c0f6d40 0x7fe40c0fc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/3526515156 >> v2:172.21.15.174:3300/0 conn(0x7fa8d00c6a30 0x7fa8d00c8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fa8d79ac640 1 -- 172.21.15.172:0/3526515156 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d0156bc0 con 0x7fa8d0150cc0 2024-03-16T19:32:07.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fa8d79ac640 1 -- 172.21.15.172:0/3526515156 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d006c070 con 0x7fa8d0070ad0 2024-03-16T19:32:07.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fa8d79ac640 1 -- 172.21.15.172:0/3526515156 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d01510a0 con 0x7fa8d00c6a30 2024-03-16T19:32:07.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe419855640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe40c045210 0x7fe40c045660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe419855640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c007590 con 0x7fe40c005e80 2024-03-16T19:32:07.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fa8d4f47640 1 --2- 172.21.15.172:0/3526515156 >> v2:172.21.15.174:3300/0 conn(0x7fa8d00c6a30 0x7fa8d00c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe419855640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40c047150 con 0x7fe40c045210 2024-03-16T19:32:07.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe419855640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0f7120 con 0x7fe40c0f6d40 2024-03-16T19:32:07.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe412ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c0f6d40 0x7fe40c0fc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe412ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c0f6d40 0x7fe40c0fc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56672/0 (socket says 172.21.15.172:56672) 2024-03-16T19:32:07.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe412ffd640 1 -- 172.21.15.172:0/1980935582 learned_addr learned my addr 172.21.15.172:0/1980935582 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe4137fe640 1 --2- 172.21.15.172:0/1980935582 >> v2:172.21.15.174:3300/0 conn(0x7fe40c045210 0x7fe40c045660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe412ffd640 1 -- 172.21.15.172:0/1980935582 >> v2:172.21.15.174:3300/0 conn(0x7fe40c045210 msgr2=0x7fe40c045660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:07.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe412ffd640 1 --2- 172.21.15.172:0/1980935582 >> v2:172.21.15.174:3300/0 conn(0x7fe40c045210 0x7fe40c045660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe412ffd640 1 -- 172.21.15.172:0/1980935582 >> v1:172.21.15.174:6789/0 conn(0x7fe40c005e80 legacy=0x7fe40c04b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fe412ffd640 1 -- 172.21.15.172:0/1980935582 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c045ba0 con 0x7fe40c0f6d40 2024-03-16T19:32:07.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7f9dbd8f1640 1 -- 172.21.15.172:0/1516741584 >> v1:172.21.15.174:6789/0 conn(0x7f9db0034a40 legacy=0x7f9db0034ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7f9dbd8f1640 1 -- 172.21.15.172:0/1516741584 >> v1:172.21.15.172:6789/0 conn(0x7f9db004b500 legacy=0x7f9db0005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:07.823 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7f9dbd8f1640 1 -- 172.21.15.172:0/1516741584 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db0006b90 con 0x7f9db00dac60 2024-03-16T19:32:07.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fa8d4f47640 1 -- 172.21.15.172:0/3526515156 >> v1:172.21.15.174:6789/0 conn(0x7fa8d0070ad0 legacy=0x7fa8d0157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fa8d4f47640 1 -- 172.21.15.172:0/3526515156 >> v1:172.21.15.172:6789/0 conn(0x7fa8d0150cc0 legacy=0x7fa8d006aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fa8d4f47640 1 -- 172.21.15.172:0/3526515156 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d015ab30 con 0x7fa8d00c6a30 2024-03-16T19:32:07.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f6307519640 1 Processor -- start 2024-03-16T19:32:07.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7f6307519640 1 -- start start 2024-03-16T19:32:07.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7f6307519640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f62f80fc970 0x7f62f804b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7f6307519640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f8007590 con 0x7f62f80f6d40 2024-03-16T19:32:07.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7f6307519640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f80066a0 con 0x7f62f804b5c0 2024-03-16T19:32:07.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7f6307519640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f8047150 con 0x7f62f80fc970 2024-03-16T19:32:07.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7f6305ab6640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f62f804b5c0 0x7f62f8045210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:56718/0 (socket says 172.21.15.172:56718) 2024-03-16T19:32:07.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f6305ab6640 1 -- 172.21.15.172:0/4183889994 learned_addr learned my addr 172.21.15.172:0/4183889994 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:07.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fe412ffd640 1 --2- 172.21.15.172:0/1980935582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c0f6d40 0x7fe40c0fc970 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe4040088f0 tx=0x7fe404034430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2ff7e311b71ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fb40e026640 1 Processor -- start 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.835+0000 7fb40e026640 1 -- start start 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fb40e026640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb408073040 0x7fb40806bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fb40e026640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4080c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fb40e026640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb408167f40 0x7fb4080c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fb40e026640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4080c1670 con 0x7fb408167f40 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fb40e026640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4080c1910 con 0x7fb408073040 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.836+0000 7fb40e026640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4080c1bf0 con 0x7fb4080c7c00 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40c823640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4080c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40c823640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4080c29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56682/0 (socket says 172.21.15.172:56682) 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40d024640 1 --2- 172.21.15.172:0/383394169 >> v2:172.21.15.174:3300/0 conn(0x7fb408073040 0x7fb40806bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40c823640 1 -- 172.21.15.172:0/383394169 learned_addr learned my addr 172.21.15.172:0/383394169 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f62f7fff640 1 -- 172.21.15.172:0/4183889994 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326994169 0 0) 0x7f62f80066a0 con 0x7f62f804b5c0 2024-03-16T19:32:07.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f62f7fff640 1 -- 172.21.15.172:0/4183889994 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ec003520 con 0x7f62f804b5c0 2024-03-16T19:32:07.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fe411ffb640 1 -- 172.21.15.172:0/1980935582 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400a4e0 con 0x7fe40c0f6d40 2024-03-16T19:32:07.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fe411ffb640 1 -- 172.21.15.172:0/1980935582 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe404037050 con 0x7fe40c0f6d40 2024-03-16T19:32:07.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f6304ab4640 1 --2- 172.21.15.172:0/4183889994 >> v2:172.21.15.172:3300/0 conn(0x7f62f80fc970 0x7f62f804b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40d825640 1 --2- 172.21.15.172:0/383394169 >> v2:172.21.15.172:3300/0 conn(0x7fb408167f40 0x7fb4080c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40c823640 1 -- 172.21.15.172:0/383394169 >> v2:172.21.15.174:3300/0 conn(0x7fb408073040 msgr2=0x7fb40806bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:07.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40c823640 1 --2- 172.21.15.172:0/383394169 >> v2:172.21.15.174:3300/0 conn(0x7fb408073040 0x7fb40806bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40c823640 1 -- 172.21.15.172:0/383394169 >> v2:172.21.15.172:3300/0 conn(0x7fb408167f40 msgr2=0x7fb4080c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:07.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40c823640 1 --2- 172.21.15.172:0/383394169 >> v2:172.21.15.172:3300/0 conn(0x7fb408167f40 0x7fb4080c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7fb40c823640 1 -- 172.21.15.172:0/383394169 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080c1ed0 con 0x7fb4080c7c00 2024-03-16T19:32:07.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f91b3fff640 1 -- 172.21.15.172:0/1410950231 shutdown_connections 2024-03-16T19:32:07.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f91b3fff640 1 --2- 172.21.15.172:0/1410950231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91b4073000 0x7f91b406c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f91b3fff640 1 --2- 172.21.15.172:0/1410950231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b406c610 0x7f91b40b1b60 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f91b3fff640 1 -- 172.21.15.172:0/1410950231 >> 172.21.15.172:0/1410950231 conn(0x7f91b406e0d0 msgr2=0x7f91b406e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:07.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f62f7fff640 1 -- 172.21.15.172:0/4183889994 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838504296 0 0) 0x7f62f8007590 con 0x7f62f80f6d40 2024-03-16T19:32:07.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.837+0000 7f62f7fff640 1 -- 172.21.15.172:0/4183889994 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f80066a0 con 0x7f62f80f6d40 2024-03-16T19:32:07.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fe411ffb640 1 -- 172.21.15.172:0/1980935582 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400a860 con 0x7fe40c0f6d40 2024-03-16T19:32:07.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fa8d4f47640 1 --2- 172.21.15.172:0/3526515156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8d00c6a30 0x7fa8d00c8260 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa8c00097a0 tx=0x7fa8c002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94bcb8bb9117bccf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7f91b3fff640 1 -- 172.21.15.172:0/1410950231 shutdown_connections 2024-03-16T19:32:07.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7f91b3fff640 1 -- 172.21.15.172:0/1410950231 wait complete. 2024-03-16T19:32:07.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7f6304ab4640 1 -- 172.21.15.172:0/4183889994 >> v1:172.21.15.174:6789/0 conn(0x7f62f804b5c0 legacy=0x7f62f8045210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7f6304ab4640 1 -- 172.21.15.172:0/4183889994 >> v1:172.21.15.172:6789/0 conn(0x7f62f80f6d40 legacy=0x7f62f8005e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7f6304ab4640 1 -- 172.21.15.172:0/4183889994 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f8006980 con 0x7f62f80fc970 2024-03-16T19:32:07.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fa8c7fff640 1 -- 172.21.15.172:0/3526515156 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c002f810 con 0x7fa8d00c6a30 2024-03-16T19:32:07.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fa8c7fff640 1 -- 172.21.15.172:0/3526515156 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c002f970 con 0x7fa8d00c6a30 2024-03-16T19:32:07.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7f9dbd8f1640 1 --2- 172.21.15.172:0/1516741584 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db00dac60 0x7f9db004b8e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9db40098a0 tx=0x7f9db402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4ad59acef03401f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fe419855640 1 -- 172.21.15.172:0/1980935582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c0f6d40 msgr2=0x7fe40c0fc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fe419855640 1 --2- 172.21.15.172:0/1980935582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c0f6d40 0x7fe40c0fc970 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe4040088f0 tx=0x7fe404034430 comp rx=0 tx=0).stop 2024-03-16T19:32:07.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fe419855640 1 -- 172.21.15.172:0/1980935582 shutdown_connections 2024-03-16T19:32:07.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fe419855640 1 --2- 172.21.15.172:0/1980935582 >> v2:172.21.15.174:3300/0 conn(0x7fe40c045210 0x7fe40c045660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fe419855640 1 --2- 172.21.15.172:0/1980935582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c0f6d40 0x7fe40c0fc970 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fe419855640 1 -- 172.21.15.172:0/1980935582 >> 172.21.15.172:0/1980935582 conn(0x7fe40c050170 msgr2=0x7fe40c050580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:07.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7fa8c7fff640 1 -- 172.21.15.172:0/3526515156 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c002f810 con 0x7fa8d00c6a30 2024-03-16T19:32:07.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fb40c823640 1 --2- 172.21.15.172:0/383394169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4080c29b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb4080ccd60 tx=0x7fb40000ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d31c23bf6c70bce4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fe419855640 1 -- 172.21.15.172:0/1980935582 shutdown_connections 2024-03-16T19:32:07.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.838+0000 7fe419855640 1 -- 172.21.15.172:0/1980935582 wait complete. 2024-03-16T19:32:07.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7fa8d79ac640 1 -- 172.21.15.172:0/3526515156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8d00c6a30 msgr2=0x7fa8d00c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/3526515156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8d00c6a30 0x7fa8d00c8260 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa8c00097a0 tx=0x7fa8c002b9f0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7fb3fffff640 1 -- 172.21.15.172:0/383394169 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40002ce80 con 0x7fb4080c7c00 2024-03-16T19:32:07.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7fb3fffff640 1 -- 172.21.15.172:0/383394169 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb400030070 con 0x7fb4080c7c00 2024-03-16T19:32:07.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7f6304ab4640 1 --2- 172.21.15.172:0/4183889994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62f80fc970 0x7f62f804b080 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0087a0 tx=0x7f62fc030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f71b285853f9c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7f62f7fff640 1 -- 172.21.15.172:0/4183889994 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300073d10 con 0x7f62f80fc970 2024-03-16T19:32:07.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7fb3fffff640 1 -- 172.21.15.172:0/383394169 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400036600 con 0x7fb4080c7c00 2024-03-16T19:32:07.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7fa8d79ac640 1 -- 172.21.15.172:0/3526515156 shutdown_connections 2024-03-16T19:32:07.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7f62f7fff640 1 -- 172.21.15.172:0/4183889994 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc034070 con 0x7f62f80fc970 2024-03-16T19:32:07.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.839+0000 7f62f7fff640 1 -- 172.21.15.172:0/4183889994 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc030d50 con 0x7f62f80fc970 2024-03-16T19:32:07.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/3526515156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8d00c6a30 0x7fa8d00c8260 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fa8d79ac640 1 -- 172.21.15.172:0/3526515156 >> 172.21.15.172:0/3526515156 conn(0x7fa8d00cc030 msgr2=0x7fa8d00cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:07.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fb40e026640 1 -- 172.21.15.172:0/383394169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 msgr2=0x7fb4080c29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fb40e026640 1 --2- 172.21.15.172:0/383394169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4080c29b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb4080ccd60 tx=0x7fb40000ad10 comp rx=0 tx=0).stop 2024-03-16T19:32:07.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fa8d79ac640 1 -- 172.21.15.172:0/3526515156 shutdown_connections 2024-03-16T19:32:07.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fb40e026640 1 -- 172.21.15.172:0/383394169 shutdown_connections 2024-03-16T19:32:07.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fb40e026640 1 --2- 172.21.15.172:0/383394169 >> v2:172.21.15.172:3300/0 conn(0x7fb408167f40 0x7fb4080c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fb40e026640 1 --2- 172.21.15.172:0/383394169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4080c29b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fb40e026640 1 --2- 172.21.15.172:0/383394169 >> v2:172.21.15.174:3300/0 conn(0x7fb408073040 0x7fb40806bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fb40e026640 1 -- 172.21.15.172:0/383394169 >> 172.21.15.172:0/383394169 conn(0x7fb4080cbfc0 msgr2=0x7fb4080cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:07.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fa8d79ac640 1 -- 172.21.15.172:0/3526515156 wait complete. 2024-03-16T19:32:07.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7f91b3fff640 1 Processor -- start 2024-03-16T19:32:07.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fb40e026640 1 -- 172.21.15.172:0/383394169 shutdown_connections 2024-03-16T19:32:07.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fb40e026640 1 -- 172.21.15.172:0/383394169 wait complete. 2024-03-16T19:32:07.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7fe419855640 1 Processor -- start 2024-03-16T19:32:07.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7f9dbd0f0640 1 -- 172.21.15.172:0/1516741584 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db402f830 con 0x7f9db00dac60 2024-03-16T19:32:07.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7f9dbd0f0640 1 -- 172.21.15.172:0/1516741584 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db402f990 con 0x7f9db00dac60 2024-03-16T19:32:07.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7f9dbd0f0640 1 -- 172.21.15.172:0/1516741584 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db402fd20 con 0x7f9db00dac60 2024-03-16T19:32:07.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7f6307519640 1 -- 172.21.15.172:0/4183889994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62f80fc970 msgr2=0x7f62f804b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7f6307519640 1 --2- 172.21.15.172:0/4183889994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62f80fc970 0x7f62f804b080 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0087a0 tx=0x7f62fc030000 comp rx=0 tx=0).stop 2024-03-16T19:32:07.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7fe419855640 1 -- start start 2024-03-16T19:32:07.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f6307519640 1 -- 172.21.15.172:0/4183889994 shutdown_connections 2024-03-16T19:32:07.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f6307519640 1 --2- 172.21.15.172:0/4183889994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62f80fc970 0x7f62f804b080 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f6307519640 1 -- 172.21.15.172:0/4183889994 >> 172.21.15.172:0/4183889994 conn(0x7f62f8050170 msgr2=0x7f62f8050580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:07.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1516741584 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db00dac60 msgr2=0x7f9db004b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f9dbf0f4640 1 --2- 172.21.15.172:0/1516741584 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db00dac60 0x7f9db004b8e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9db40098a0 tx=0x7f9db402ba10 comp rx=0 tx=0).stop 2024-03-16T19:32:07.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.840+0000 7f91b3fff640 1 -- start start 2024-03-16T19:32:07.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f91b3fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91b406c610 0x7f91b416b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f91b3fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91b406d1c0 0x7f91b41fdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f91b3fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b4073000 0x7f91b40c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f91b27fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91b406d1c0 0x7f91b41fdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f91b27fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91b406d1c0 0x7f91b41fdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40848/0 (socket says 172.21.15.172:40848) 2024-03-16T19:32:07.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fb40e026640 1 Processor -- start 2024-03-16T19:32:07.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f6307519640 1 -- 172.21.15.172:0/4183889994 shutdown_connections 2024-03-16T19:32:07.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f6307519640 1 -- 172.21.15.172:0/4183889994 wait complete. 2024-03-16T19:32:07.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fb40e026640 1 -- start start 2024-03-16T19:32:07.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fe419855640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c005e80 0x7fe40c0fbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f91b37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b4073000 0x7f91b40c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f91b37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b4073000 0x7f91b40c3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39316/0 (socket says 172.21.15.172:39316) 2024-03-16T19:32:07.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fe419855640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe40c045210 0x7fe40c0fc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f91b27fc640 1 -- 172.21.15.172:0/826138930 learned_addr learned my addr 172.21.15.172:0/826138930 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b416b6e0 con 0x7f91b4073000 2024-03-16T19:32:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b41fe0f0 con 0x7f91b406d1c0 2024-03-16T19:32:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b416a700 con 0x7f91b406c610 2024-03-16T19:32:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fe412ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c005e80 0x7fe40c0fbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.844 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fe412ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c005e80 0x7fe40c0fbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56698/0 (socket says 172.21.15.172:56698) 2024-03-16T19:32:07.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fe412ffd640 1 -- 172.21.15.172:0/2471933502 learned_addr learned my addr 172.21.15.172:0/2471933502 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fe419855640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40c0f6d40 0x7fe40c0f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fa8d79ac640 1 Processor -- start 2024-03-16T19:32:07.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1516741584 shutdown_connections 2024-03-16T19:32:07.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f9dbf0f4640 1 --2- 172.21.15.172:0/1516741584 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db00dac60 0x7f9db004b8e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.841+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1516741584 >> 172.21.15.172:0/1516741584 conn(0x7f9db0050180 msgr2=0x7f9db0050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:07.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1516741584 shutdown_connections 2024-03-16T19:32:07.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1516741584 wait complete. 2024-03-16T19:32:07.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.842+0000 7fe4127fc640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe40c045210 0x7fe40c0fc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fe419855640 1 -- 172.21.15.172:0/2471933502 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0fcad0 con 0x7fe40c0f6d40 2024-03-16T19:32:07.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fe419855640 1 -- 172.21.15.172:0/2471933502 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0f0480 con 0x7fe40c045210 2024-03-16T19:32:07.846 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fe419855640 1 -- 172.21.15.172:0/2471933502 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0f0720 con 0x7fe40c005e80 2024-03-16T19:32:07.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fa8d79ac640 1 -- start start 2024-03-16T19:32:07.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7f6307519640 1 Processor -- start 2024-03-16T19:32:07.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7f6307519640 1 -- start start 2024-03-16T19:32:07.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fb40e026640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408073040 0x7fb4082061e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fb40e026640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4081f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fb40e026640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb408167f40 0x7fb4080c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fe412ffd640 1 -- 172.21.15.172:0/2471933502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe40c045210 msgr2=0x7fe40c0fc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fe412ffd640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe40c045210 0x7fe40c0fc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fb40e026640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb408164260 con 0x7fb408167f40 2024-03-16T19:32:07.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fb40e026640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4081f4af0 con 0x7fb408073040 2024-03-16T19:32:07.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fb40e026640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4082057d0 con 0x7fb4080c7c00 2024-03-16T19:32:07.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fe412ffd640 1 -- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40c0f6d40 msgr2=0x7fe40c0f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fe412ffd640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40c0f6d40 0x7fe40c0f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7fe412ffd640 1 -- 172.21.15.172:0/2471933502 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe404008550 con 0x7fe40c005e80 2024-03-16T19:32:07.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40d024640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408073040 0x7fb4082061e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40c823640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4081f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40d825640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb408167f40 0x7fb4080c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fe412ffd640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c005e80 0x7fe40c0fbf70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe404008a10 tx=0x7fe4040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7f91b37fe640 1 -- 172.21.15.172:0/826138930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91b406c610 msgr2=0x7f91b416b1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T19:32:07.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7f91b37fe640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91b406c610 0x7f91b416b1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7f91b37fe640 1 -- 172.21.15.172:0/826138930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91b406d1c0 msgr2=0x7f91b41fdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7f91b37fe640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91b406d1c0 0x7f91b41fdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.850 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.843+0000 7f91b37fe640 1 -- 172.21.15.172:0/826138930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a800f610 con 0x7f91b4073000 2024-03-16T19:32:07.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40d825640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb408167f40 0x7fb4080c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:39328/0 (socket says 172.21.15.172:39328) 2024-03-16T19:32:07.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40d825640 1 -- 172.21.15.172:0/131662410 learned_addr learned my addr 172.21.15.172:0/131662410 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fe3fbfff640 1 -- 172.21.15.172:0/2471933502 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe404046630 con 0x7fe40c005e80 2024-03-16T19:32:07.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fe3fbfff640 1 -- 172.21.15.172:0/2471933502 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe404046790 con 0x7fe40c005e80 2024-03-16T19:32:07.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7f9dbf0f4640 1 Processor -- start 2024-03-16T19:32:07.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40d825640 1 -- 172.21.15.172:0/131662410 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 msgr2=0x7fb4081f5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fa8d79ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8d0070ad0 0x7fa8d013f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40d825640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4081f5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40d825640 1 -- 172.21.15.172:0/131662410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408073040 msgr2=0x7fb4082061e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fe3fbfff640 1 -- 172.21.15.172:0/2471933502 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe404046b40 con 0x7fe40c005e80 2024-03-16T19:32:07.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fa8d79ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8d00c6a30 0x7fa8d01ef790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7f91b37fe640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b4073000 0x7f91b40c3b50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009f00 tx=0x7f91ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7f6307519640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62f804b5c0 0x7f62f804f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40d825640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408073040 0x7fb4082061e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fa8d5748640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8d0070ad0 0x7fa8d013f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d5748640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8d0070ad0 0x7fa8d013f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56732/0 (socket says 172.21.15.172:56732) 2024-03-16T19:32:07.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d5748640 1 -- 172.21.15.172:0/2403022709 learned_addr learned my addr 172.21.15.172:0/2403022709 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7fb40d825640 1 -- 172.21.15.172:0/131662410 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4000077f0 con 0x7fb408167f40 2024-03-16T19:32:07.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d0150cc0 0x7fa8d01efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d4f47640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8d00c6a30 0x7fa8d01ef790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f919b7fe640 1 -- 172.21.15.172:0/826138930 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac018070 con 0x7f91b4073000 2024-03-16T19:32:07.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fb40d825640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb408167f40 0x7fb4080c6920 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d5d0 tx=0x7fb3f800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.855 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f919b7fe640 1 -- 172.21.15.172:0/826138930 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ac00ece0 con 0x7f91b4073000 2024-03-16T19:32:07.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7f6307519640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62f80f6d40 0x7f62f818bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d0072730 con 0x7fa8d0150cc0 2024-03-16T19:32:07.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d0072030 con 0x7fa8d00c6a30 2024-03-16T19:32:07.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00cba40 con 0x7fa8d0070ad0 2024-03-16T19:32:07.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7f6307519640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f80fc970 0x7f62f818cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.856 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7f6307519640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f804b9d0 con 0x7f62f804b5c0 2024-03-16T19:32:07.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d5f49640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d0150cc0 0x7fa8d01efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7f6307519640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f80ece90 con 0x7f62f80fc970 2024-03-16T19:32:07.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fb3fdffb640 1 -- 172.21.15.172:0/131662410 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f80125f0 con 0x7fb408167f40 2024-03-16T19:32:07.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.844+0000 7f6307519640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f80f82b0 con 0x7f62f80f6d40 2024-03-16T19:32:07.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f6304ab4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62f80f6d40 0x7f62f818bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f6305ab6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f80fc970 0x7f62f818cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb408163ca0 con 0x7fb408167f40 2024-03-16T19:32:07.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fb3fdffb640 1 -- 172.21.15.172:0/131662410 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f8012770 con 0x7fb408167f40 2024-03-16T19:32:07.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fb3fdffb640 1 -- 172.21.15.172:0/131662410 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8012b20 con 0x7fb408167f40 2024-03-16T19:32:07.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f6304ab4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62f80f6d40 0x7f62f818bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56744/0 (socket says 172.21.15.172:56744) 2024-03-16T19:32:07.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d5748640 1 -- 172.21.15.172:0/2403022709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8d00c6a30 msgr2=0x7fa8d01ef790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d5748640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8d00c6a30 0x7fa8d01ef790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d5748640 1 -- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d0150cc0 msgr2=0x7fa8d01efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f6304ab4640 1 -- 172.21.15.172:0/1884807249 learned_addr learned my addr 172.21.15.172:0/1884807249 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f6304ab4640 1 -- 172.21.15.172:0/1884807249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f80fc970 msgr2=0x7f62f818cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d5748640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d0150cc0 0x7fa8d01efcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7fa8d5748640 1 -- 172.21.15.172:0/2403022709 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c0009400 con 0x7fa8d0070ad0 2024-03-16T19:32:07.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f6304ab4640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f80fc970 0x7f62f818cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f6304ab4640 1 -- 172.21.15.172:0/1884807249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62f804b5c0 msgr2=0x7f62f804f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f6304ab4640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62f804b5c0 0x7f62f804f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f6304ab4640 1 -- 172.21.15.172:0/1884807249 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc008480 con 0x7f62f80f6d40 2024-03-16T19:32:07.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7fa8d5f49640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d0150cc0 0x7fa8d01efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:32:07.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b4168fc0 con 0x7f91b4073000 2024-03-16T19:32:07.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.845+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b4168ce0 con 0x7f91b4073000 2024-03-16T19:32:07.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7f919b7fe640 1 -- 172.21.15.172:0/826138930 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac003bb0 con 0x7f91b4073000 2024-03-16T19:32:07.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7fa8d5748640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8d0070ad0 0x7fa8d013f150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00c480 tx=0x7fa8cc00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4082059e0 con 0x7fb408167f40 2024-03-16T19:32:07.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7fa8c5ffb640 1 -- 172.21.15.172:0/2403022709 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc005ce0 con 0x7fa8d0070ad0 2024-03-16T19:32:07.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7fa8c5ffb640 1 -- 172.21.15.172:0/2403022709 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc005e40 con 0x7fa8d0070ad0 2024-03-16T19:32:07.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7f6304ab4640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62f80f6d40 0x7f62f818bff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f62fc03c040 tx=0x7f62fc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7fa8c5ffb640 1 -- 172.21.15.172:0/2403022709 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0072f0 con 0x7fa8d0070ad0 2024-03-16T19:32:07.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d014d340 con 0x7fa8d0070ad0 2024-03-16T19:32:07.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.846+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d00729a0 con 0x7fa8d0070ad0 2024-03-16T19:32:07.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b416a540 con 0x7f91b4073000 2024-03-16T19:32:07.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.847+0000 7f62f5ffb640 1 -- 172.21.15.172:0/1884807249 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc03ab30 con 0x7f62f80f6d40 2024-03-16T19:32:07.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.847+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f8034600 con 0x7f62f80f6d40 2024-03-16T19:32:07.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.847+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f818de90 con 0x7f62f80f6d40 2024-03-16T19:32:07.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.847+0000 7f62f5ffb640 1 -- 172.21.15.172:0/1884807249 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc03ac90 con 0x7f62f80f6d40 2024-03-16T19:32:07.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.847+0000 7f62f5ffb640 1 -- 172.21.15.172:0/1884807249 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc044410 con 0x7f62f80f6d40 2024-03-16T19:32:07.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.847+0000 7f9dbf0f4640 1 -- start start 2024-03-16T19:32:07.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7f9dbf0f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db0034a40 0x7f9db00d5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7f9dbf0f4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9db004b500 0x7f9db00d7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7f9dbf0f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db00dac60 0x7f9db00d7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7f9dbf0f4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00d6c50 con 0x7f9db004b500 2024-03-16T19:32:07.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7f9dbf0f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00c6660 con 0x7f9db00dac60 2024-03-16T19:32:07.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7f9dbf0f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db0182b80 con 0x7f9db0034a40 2024-03-16T19:32:07.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7f919b7fe640 1 -- 172.21.15.172:0/826138930 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f91ac014420 con 0x7f91b4073000 2024-03-16T19:32:07.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7f919b7fe640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f91a006b400 0x7f91a006d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7f919b7fe640 1 -- 172.21.15.172:0/826138930 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f91ac089d70 con 0x7f91b4073000 2024-03-16T19:32:07.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7fb3fdffb640 1 -- 172.21.15.172:0/131662410 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb3f8012c80 con 0x7fb408167f40 2024-03-16T19:32:07.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7f91b2ffd640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f91a006b400 0x7f91a006d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7fb3fdffb640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb3ec06b480 0x7fb3ec06d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7fb3fdffb640 1 -- 172.21.15.172:0/131662410 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fb3f808a040 con 0x7fb408167f40 2024-03-16T19:32:07.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.848+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4080707d0 con 0x7fb408167f40 2024-03-16T19:32:07.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7fb40d024640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb3ec06b480 0x7fb3ec06d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f91b2ffd640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f91a006b400 0x7f91a006d8c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f91a803e000 tx=0x7f91a8038000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e0005290 con 0x7f62f80f6d40 2024-03-16T19:32:07.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f62f5ffb640 1 -- 172.21.15.172:0/1884807249 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f62fc04c3f0 con 0x7f62f80f6d40 2024-03-16T19:32:07.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7fe419855640 1 -- 172.21.15.172:0/2471933502 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c0f0a00 con 0x7fe40c005e80 2024-03-16T19:32:07.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7fe419855640 1 -- 172.21.15.172:0/2471933502 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c0fd430 con 0x7fe40c005e80 2024-03-16T19:32:07.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f62f5ffb640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f62e406b4b0 0x7f62e406d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7f9dbe8f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db00dac60 0x7f9db00d7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7f9dbe8f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db00dac60 0x7f9db00d7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40884/0 (socket says 172.21.15.172:40884) 2024-03-16T19:32:07.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.849+0000 7f9dbe8f3640 1 -- 172.21.15.172:0/1522290670 learned_addr learned my addr 172.21.15.172:0/1522290670 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:07.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbd8f1640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9db004b500 0x7f9db00d7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbe8f3640 1 -- 172.21.15.172:0/1522290670 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db0034a40 msgr2=0x7f9db00d5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbe0f2640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db0034a40 0x7f9db00d5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbe8f3640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db0034a40 0x7f9db00d5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbe8f3640 1 -- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9db004b500 msgr2=0x7f9db00d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:07.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbe8f3640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9db004b500 0x7f9db00d7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:07.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.851+0000 7fa8c5ffb640 1 -- 172.21.15.172:0/2403022709 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa8cc007450 con 0x7fa8d0070ad0 2024-03-16T19:32:07.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbe8f3640 1 -- 172.21.15.172:0/1522290670 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db40094d0 con 0x7f9db00dac60 2024-03-16T19:32:07.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbd8f1640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9db004b500 0x7f9db00d7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:07.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbe0f2640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db0034a40 0x7f9db00d5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:07.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9dbe8f3640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db00dac60 0x7f9db00d7fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9db806a510 tx=0x7f9db8068590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9daeffd640 1 -- 172.21.15.172:0/1522290670 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8070070 con 0x7f9db00dac60 2024-03-16T19:32:07.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7f9daeffd640 1 -- 172.21.15.172:0/1522290670 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db80766f0 con 0x7f9db00dac60 2024-03-16T19:32:07.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.851+0000 7f62f5ffb640 1 -- 172.21.15.172:0/1884807249 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f62fc0ab0d0 con 0x7f62f80f6d40 2024-03-16T19:32:07.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.850+0000 7fb40d024640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb3ec06b480 0x7fb3ec06d940 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb40400d430 tx=0x7fb40400f040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.851+0000 7f63052b5640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f62e406b4b0 0x7f62e406d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.851+0000 7f9daeffd640 1 -- 172.21.15.172:0/1522290670 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8095420 con 0x7f9db00dac60 2024-03-16T19:32:07.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.851+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db0181d20 con 0x7f9db00dac60 2024-03-16T19:32:07.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.851+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db01822e0 con 0x7f9db00dac60 2024-03-16T19:32:07.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.851+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d0075a70 con 0x7fa8d0070ad0 2024-03-16T19:32:07.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.851+0000 7fa8c5ffb640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa8b406b480 0x7fa8b406d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.852+0000 7fe419855640 1 -- 172.21.15.172:0/2471933502 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c005380 con 0x7fe40c005e80 2024-03-16T19:32:07.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.852+0000 7fa8c5ffb640 1 -- 172.21.15.172:0/2403022709 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fa8cc089170 con 0x7fa8d0070ad0 2024-03-16T19:32:07.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.852+0000 7fa8d4f47640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa8b406b480 0x7fa8b406d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.872 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.852+0000 7f63052b5640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f62e406b4b0 0x7f62e406d970 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f62f00010e0 tx=0x7f62f0007950 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.852+0000 7fa8d4f47640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa8b406b480 0x7fa8b406d940 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0036000 tx=0x7fa8c0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.853+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db004a7f0 con 0x7f9db00dac60 2024-03-16T19:32:07.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.853+0000 7f9daeffd640 1 -- 172.21.15.172:0/1522290670 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9db8092020 con 0x7f9db00dac60 2024-03-16T19:32:07.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.853+0000 7f9daeffd640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9c06b400 0x7f9d9c06d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.854+0000 7f9daeffd640 1 -- 172.21.15.172:0/1522290670 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f9db8104e80 con 0x7f9db00dac60 2024-03-16T19:32:07.873 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.854+0000 7f9dbe0f2640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9c06b400 0x7f9d9c06d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.854+0000 7f62f5ffb640 1 -- 172.21.15.172:0/1884807249 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62fc075d00 con 0x7f62f80f6d40 2024-03-16T19:32:07.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.854+0000 7f9dbe0f2640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9c06b400 0x7f9d9c06d8c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9da8007fd0 tx=0x7f9da8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.855+0000 7f919b7fe640 1 -- 172.21.15.172:0/826138930 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91ac08e070 con 0x7f91b4073000 2024-03-16T19:32:07.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.856+0000 7fe3fbfff640 1 -- 172.21.15.172:0/2471933502 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fe404046ca0 con 0x7fe40c005e80 2024-03-16T19:32:07.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.856+0000 7fe3fbfff640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe40006b430 0x7fe40006d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:07.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.857+0000 7fe4127fc640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe40006b430 0x7fe40006d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:07.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.857+0000 7fe4127fc640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe40006b430 0x7fe40006d8f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0fbcd0 tx=0x7fe408009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:07.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.858+0000 7fe3fbfff640 1 -- 172.21.15.172:0/2471933502 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fe4040839d0 con 0x7fe40c005e80 2024-03-16T19:32:07.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.858+0000 7f9daeffd640 1 -- 172.21.15.172:0/1522290670 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9db80cfa20 con 0x7f9db00dac60 2024-03-16T19:32:07.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.859+0000 7fa8c5ffb640 1 -- 172.21.15.172:0/2403022709 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8cc053e50 con 0x7fa8d0070ad0 2024-03-16T19:32:07.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.861+0000 7fb3fdffb640 1 -- 172.21.15.172:0/131662410 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3f8054ca0 con 0x7fb408167f40 2024-03-16T19:32:07.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:07.863+0000 7fe3fbfff640 1 -- 172.21.15.172:0/2471933502 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe40407fc70 con 0x7fe40c005e80 2024-03-16T19:32:08.187 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.201+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5b0c005540 con 0x7f5b340681c0 2024-03-16T19:32:08.193 INFO:teuthology.orchestra.run.smithi172.stdout:55834574888 2024-03-16T19:32:08.195 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.209+0000 7f5b2b7fe640 1 -- 172.21.15.172:0/801101217 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5b1c054960 con 0x7f5b340681c0 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5b1806b480 msgr2=0x7f5b1806d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5b1806b480 0x7f5b1806d940 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5b341570f0 tx=0x7f5b240090e0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5b340681c0 msgr2=0x7f5b3414ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5b340681c0 0x7f5b3414ab80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00b410 tx=0x7f5b1c00b8d0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 shutdown_connections 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b34150cc0 0x7f5b34145bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b341508a0 0x7f5b34153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5b1806b480 0x7f5b1806d940 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 --2- 172.21.15.172:0/801101217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5b340681c0 0x7f5b3414ab80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.219+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 >> 172.21.15.172:0/801101217 conn(0x7f5b3406aed0 msgr2=0x7f5b3414ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.220+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 shutdown_connections 2024-03-16T19:32:08.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.224+0000 7f5b38863640 1 -- 172.21.15.172:0/801101217 wait complete. 2024-03-16T19:32:08.226 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574888 got 55834574888 for osd.1 2024-03-16T19:32:08.227 DEBUG:teuthology.parallel:result is None 2024-03-16T19:32:08.237 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.257+0000 7f83d4450640 1 -- 172.21.15.172:0/2777552066 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f83b4005540 con 0x7f83c404bc20 2024-03-16T19:32:08.238 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.257+0000 7f83c2ffd640 1 -- 172.21.15.172:0/2777552066 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f83cc0cde60 con 0x7f83c404bc20 2024-03-16T19:32:08.238 INFO:teuthology.orchestra.run.smithi172.stdout:55834574887 2024-03-16T19:32:08.243 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.262+0000 7f83c0ff9640 1 -- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f83b006b4b0 msgr2=0x7f83b006d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.243 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.262+0000 7f83c0ff9640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f83b006b4b0 0x7f83b006d970 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f83bc003eb0 tx=0x7f83bc009040 comp rx=0 tx=0).stop 2024-03-16T19:32:08.243 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.262+0000 7f83c0ff9640 1 -- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c404bc20 msgr2=0x7f83c4049190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.243 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.262+0000 7f83c0ff9640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c404bc20 0x7f83c4049190 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f83cc02d4f0 tx=0x7f83cc068590 comp rx=0 tx=0).stop 2024-03-16T19:32:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.262+0000 7f83c0ff9640 1 -- 172.21.15.172:0/2777552066 shutdown_connections 2024-03-16T19:32:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.262+0000 7f83c0ff9640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83c40dd6b0 0x7f83c40496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.262+0000 7f83c0ff9640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83c404bc20 0x7f83c4049190 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.262+0000 7f83c0ff9640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f83b006b4b0 0x7f83b006d970 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.263+0000 7f83c0ff9640 1 --2- 172.21.15.172:0/2777552066 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83c40462f0 0x7f83c40488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.263+0000 7f83c0ff9640 1 -- 172.21.15.172:0/2777552066 >> 172.21.15.172:0/2777552066 conn(0x7f83c404dd20 msgr2=0x7f83c40470a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.263+0000 7f83c0ff9640 1 -- 172.21.15.172:0/2777552066 shutdown_connections 2024-03-16T19:32:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.263+0000 7f83c0ff9640 1 -- 172.21.15.172:0/2777552066 wait complete. 2024-03-16T19:32:08.257 INFO:teuthology.orchestra.run.smithi172.stdout:55834574886 2024-03-16T19:32:08.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.276+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f62e0005540 con 0x7f62f80f6d40 2024-03-16T19:32:08.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.276+0000 7f62f5ffb640 1 -- 172.21.15.172:0/1884807249 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f62fc0793c0 con 0x7f62f80f6d40 2024-03-16T19:32:08.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f91b4070780 con 0x7f91b4073000 2024-03-16T19:32:08.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb4080758f0 con 0x7fb408167f40 2024-03-16T19:32:08.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f62e406b4b0 msgr2=0x7f62e406d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f62e406b4b0 0x7f62e406d970 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f62f00010e0 tx=0x7f62f0007950 comp rx=0 tx=0).stop 2024-03-16T19:32:08.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62f80f6d40 msgr2=0x7f62f818bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62f80f6d40 0x7f62f818bff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f62fc03c040 tx=0x7f62fc004730 comp rx=0 tx=0).stop 2024-03-16T19:32:08.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f919b7fe640 1 -- 172.21.15.172:0/826138930 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f91ac054920 con 0x7f91b4073000 2024-03-16T19:32:08.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 shutdown_connections 2024-03-16T19:32:08.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f80fc970 0x7f62f818cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f62e406b4b0 0x7f62e406d970 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62f80f6d40 0x7f62f818bff0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 --2- 172.21.15.172:0/1884807249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f62f804b5c0 0x7f62f804f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.263 INFO:teuthology.orchestra.run.smithi172.stdout:55834574886 2024-03-16T19:32:08.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.280+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 >> 172.21.15.172:0/1884807249 conn(0x7f62f8050170 msgr2=0x7f62f80059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.281+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 shutdown_connections 2024-03-16T19:32:08.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.281+0000 7f6307519640 1 -- 172.21.15.172:0/1884807249 wait complete. 2024-03-16T19:32:08.264 INFO:teuthology.orchestra.run.smithi172.stdout:55834574886 2024-03-16T19:32:08.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.281+0000 7fb3fdffb640 1 -- 172.21.15.172:0/131662410 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb3f8058360 con 0x7fb408167f40 2024-03-16T19:32:08.265 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.3 2024-03-16T19:32:08.266 DEBUG:teuthology.parallel:result is None 2024-03-16T19:32:08.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f91a006b400 msgr2=0x7f91a006d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f91a006b400 0x7f91a006d8c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f91a803e000 tx=0x7f91a8038000 comp rx=0 tx=0).stop 2024-03-16T19:32:08.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b4073000 msgr2=0x7f91b40c3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b4073000 0x7f91b40c3b50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009f00 tx=0x7f91ac00e4b0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 shutdown_connections 2024-03-16T19:32:08.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f91b4073000 0x7f91b40c3b50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91b406d1c0 0x7f91b41fdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f91a006b400 0x7f91a006d8c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 --2- 172.21.15.172:0/826138930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91b406c610 0x7f91b416b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 >> 172.21.15.172:0/826138930 conn(0x7f91b406e0d0 msgr2=0x7f91b40c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 shutdown_connections 2024-03-16T19:32:08.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7f91b3fff640 1 -- 172.21.15.172:0/826138930 wait complete. 2024-03-16T19:32:08.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb3ec06b480 msgr2=0x7fb3ec06d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7fb40e026640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb3ec06b480 0x7fb3ec06d940 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb40400d430 tx=0x7fb40400f040 comp rx=0 tx=0).stop 2024-03-16T19:32:08.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb408167f40 msgr2=0x7fb4080c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.284+0000 7fb40e026640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb408167f40 0x7fb4080c6920 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d5d0 tx=0x7fb3f800da90 comp rx=0 tx=0).stop 2024-03-16T19:32:08.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.285+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 shutdown_connections 2024-03-16T19:32:08.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.285+0000 7fb40e026640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb408167f40 0x7fb4080c6920 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.285+0000 7fb40e026640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb3ec06b480 0x7fb3ec06d940 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.285+0000 7fb40e026640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4080c7c00 0x7fb4081f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.285+0000 7fb40e026640 1 --2- 172.21.15.172:0/131662410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408073040 0x7fb4082061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.285+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 >> 172.21.15.172:0/131662410 conn(0x7fb4080cbfc0 msgr2=0x7fb40808bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.285+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 shutdown_connections 2024-03-16T19:32:08.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.286+0000 7fb40e026640 1 -- 172.21.15.172:0/131662410 wait complete. 2024-03-16T19:32:08.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.289+0000 7fe419855640 1 -- 172.21.15.172:0/2471933502 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe40c04c250 con 0x7fe40c005e80 2024-03-16T19:32:08.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.290+0000 7fe3fbfff640 1 -- 172.21.15.172:0/2471933502 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe404040020 con 0x7fe40c005e80 2024-03-16T19:32:08.271 INFO:teuthology.orchestra.run.smithi172.stdout:55834574885 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.294+0000 7fe3f9ffb640 1 -- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe40006b430 msgr2=0x7fe40006d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.294+0000 7fe3f9ffb640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe40006b430 0x7fe40006d8f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0fbcd0 tx=0x7fe408009040 comp rx=0 tx=0).stop 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.294+0000 7fe3f9ffb640 1 -- 172.21.15.172:0/2471933502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c005e80 msgr2=0x7fe40c0fbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.294+0000 7fe3f9ffb640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c005e80 0x7fe40c0fbf70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe404008a10 tx=0x7fe4040046c0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.295+0000 7fe3f9ffb640 1 -- 172.21.15.172:0/2471933502 shutdown_connections 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.295+0000 7fe3f9ffb640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe40c0f6d40 0x7fe40c0f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.295+0000 7fe3f9ffb640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe40c045210 0x7fe40c0fc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.295+0000 7fe3f9ffb640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe40006b430 0x7fe40006d8f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.295+0000 7fe3f9ffb640 1 --2- 172.21.15.172:0/2471933502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe40c005e80 0x7fe40c0fbf70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.295+0000 7fe3f9ffb640 1 -- 172.21.15.172:0/2471933502 >> 172.21.15.172:0/2471933502 conn(0x7fe40c050170 msgr2=0x7fe40c0f0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.295+0000 7fe3f9ffb640 1 -- 172.21.15.172:0/2471933502 shutdown_connections 2024-03-16T19:32:08.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.295+0000 7fe3f9ffb640 1 -- 172.21.15.172:0/2471933502 wait complete. 2024-03-16T19:32:08.277 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.296+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9db0005390 con 0x7f9db00dac60 2024-03-16T19:32:08.278 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.297+0000 7f9daeffd640 1 -- 172.21.15.172:0/1522290670 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9db80cfa20 con 0x7f9db00dac60 2024-03-16T19:32:08.279 INFO:teuthology.orchestra.run.smithi172.stdout:377957122053 2024-03-16T19:32:08.282 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574886 got 55834574886 for osd.7 2024-03-16T19:32:08.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.300+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa8d01543d0 con 0x7fa8d0070ad0 2024-03-16T19:32:08.282 DEBUG:teuthology.parallel:result is None 2024-03-16T19:32:08.282 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.302+0000 7fa8c5ffb640 1 -- 172.21.15.172:0/2403022709 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa8cc057510 con 0x7fa8d0070ad0 2024-03-16T19:32:08.283 INFO:teuthology.orchestra.run.smithi172.stdout:55834574887 2024-03-16T19:32:08.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.302+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9c06b400 msgr2=0x7f9d9c06d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.302+0000 7f9dbf0f4640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9c06b400 0x7f9d9c06d8c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9da8007fd0 tx=0x7f9da8009040 comp rx=0 tx=0).stop 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.302+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db00dac60 msgr2=0x7f9db00d7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.303+0000 7f9dbf0f4640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db00dac60 0x7f9db00d7fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9db806a510 tx=0x7f9db8068590 comp rx=0 tx=0).stop 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.303+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 shutdown_connections 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.303+0000 7f9dbf0f4640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db00dac60 0x7f9db00d7fc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.303+0000 7f9dbf0f4640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9db004b500 0x7f9db00d7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.303+0000 7f9dbf0f4640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d9c06b400 0x7f9d9c06d8c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.303+0000 7f9dbf0f4640 1 --2- 172.21.15.172:0/1522290670 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9db0034a40 0x7f9db00d5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.303+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 >> 172.21.15.172:0/1522290670 conn(0x7f9db0050180 msgr2=0x7f9db00da090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.303+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 shutdown_connections 2024-03-16T19:32:08.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.303+0000 7f9dbf0f4640 1 -- 172.21.15.172:0/1522290670 wait complete. 2024-03-16T19:32:08.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa8b406b480 msgr2=0x7fa8b406d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa8b406b480 0x7fa8b406d940 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0036000 tx=0x7fa8c0008030 comp rx=0 tx=0).stop 2024-03-16T19:32:08.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8d0070ad0 msgr2=0x7fa8d013f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8d0070ad0 0x7fa8d013f150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00c480 tx=0x7fa8cc00c940 comp rx=0 tx=0).stop 2024-03-16T19:32:08.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 shutdown_connections 2024-03-16T19:32:08.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8d0150cc0 0x7fa8d01efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8d00c6a30 0x7fa8d01ef790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa8b406b480 0x7fa8b406d940 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 --2- 172.21.15.172:0/2403022709 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8d0070ad0 0x7fa8d013f150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 >> 172.21.15.172:0/2403022709 conn(0x7fa8d00cc030 msgr2=0x7fa8d00c8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.306+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 shutdown_connections 2024-03-16T19:32:08.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.307+0000 7fa8d79ac640 1 -- 172.21.15.172:0/2403022709 wait complete. 2024-03-16T19:32:08.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574886 got 55834574886 for osd.4 2024-03-16T19:32:08.289 DEBUG:teuthology.parallel:result is None 2024-03-16T19:32:08.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574886 got 55834574886 for osd.0 2024-03-16T19:32:08.290 DEBUG:teuthology.parallel:result is None 2024-03-16T19:32:08.296 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574885 got 55834574885 for osd.5 2024-03-16T19:32:08.297 DEBUG:teuthology.parallel:result is None 2024-03-16T19:32:08.302 INFO:tasks.ceph.ceph_manager.ceph:need seq 377957122053 got 377957122053 for osd.2 2024-03-16T19:32:08.302 DEBUG:teuthology.parallel:result is None 2024-03-16T19:32:08.303 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.6 2024-03-16T19:32:08.303 DEBUG:teuthology.parallel:result is None 2024-03-16T19:32:08.303 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-16T19:32:08.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:08.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.512+0000 7fb0738ff640 1 Processor -- start 2024-03-16T19:32:08.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.512+0000 7fb0738ff640 1 -- start start 2024-03-16T19:32:08.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.513+0000 7fb0738ff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb06c071be0 0x7fb06c071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:08.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.513+0000 7fb0738ff640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c06ce70 con 0x7fb06c15ff20 2024-03-16T19:32:08.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.513+0000 7fb0738ff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c0c63b0 con 0x7fb06c06bfe0 2024-03-16T19:32:08.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.513+0000 7fb0738ff640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c0c55f0 con 0x7fb06c071be0 2024-03-16T19:32:08.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.513+0000 7fb07169b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb06c071be0 0x7fb06c071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:08.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.513+0000 7fb07169b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb06c071be0 0x7fb06c071fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57600/0 (socket says 172.21.15.172:57600) 2024-03-16T19:32:08.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.513+0000 7fb07169b640 1 -- 172.21.15.172:0/3610329180 learned_addr learned my addr 172.21.15.172:0/3610329180 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:08.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.514+0000 7fb07169b640 1 -- 172.21.15.172:0/3610329180 >> v1:172.21.15.174:6789/0 conn(0x7fb06c06bfe0 legacy=0x7fb06c0c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.514+0000 7fb07169b640 1 -- 172.21.15.172:0/3610329180 >> v1:172.21.15.172:6789/0 conn(0x7fb06c15ff20 legacy=0x7fb06c162310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.514+0000 7fb07169b640 1 -- 172.21.15.172:0/3610329180 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c06cb90 con 0x7fb06c071be0 2024-03-16T19:32:08.496 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.515+0000 7fb07169b640 1 --2- 172.21.15.172:0/3610329180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c071be0 0x7fb06c071fc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb064009e50 tx=0x7fb06402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64d90b2637af9e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:08.496 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.515+0000 7fb063fff640 1 -- 172.21.15.172:0/3610329180 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06402cbb0 con 0x7fb06c071be0 2024-03-16T19:32:08.496 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.515+0000 7fb063fff640 1 -- 172.21.15.172:0/3610329180 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06402cd10 con 0x7fb06c071be0 2024-03-16T19:32:08.496 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.515+0000 7fb063fff640 1 -- 172.21.15.172:0/3610329180 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064034610 con 0x7fb06c071be0 2024-03-16T19:32:08.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.516+0000 7fb0738ff640 1 -- 172.21.15.172:0/3610329180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c071be0 msgr2=0x7fb06c071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.516+0000 7fb0738ff640 1 --2- 172.21.15.172:0/3610329180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c071be0 0x7fb06c071fc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb064009e50 tx=0x7fb06402c010 comp rx=0 tx=0).stop 2024-03-16T19:32:08.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.516+0000 7fb0738ff640 1 -- 172.21.15.172:0/3610329180 shutdown_connections 2024-03-16T19:32:08.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.516+0000 7fb0738ff640 1 --2- 172.21.15.172:0/3610329180 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c071be0 0x7fb06c071fc0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.516+0000 7fb0738ff640 1 -- 172.21.15.172:0/3610329180 >> 172.21.15.172:0/3610329180 conn(0x7fb06c072b40 msgr2=0x7fb06c072f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.516+0000 7fb0738ff640 1 -- 172.21.15.172:0/3610329180 shutdown_connections 2024-03-16T19:32:08.497 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.516+0000 7fb0738ff640 1 -- 172.21.15.172:0/3610329180 wait complete. 2024-03-16T19:32:08.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.518+0000 7fb0738ff640 1 Processor -- start 2024-03-16T19:32:08.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.518+0000 7fb0738ff640 1 -- start start 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb0738ff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c06bfe0 0x7fb06c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb0738ff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb06c071be0 0x7fb06c15b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb0738ff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb06c15ff20 0x7fb06c152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb07169b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c06bfe0 0x7fb06c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb0738ff640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c1f4f40 con 0x7fb06c06bfe0 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb0738ff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c159f80 con 0x7fb06c15ff20 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb0738ff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c152b70 con 0x7fb06c071be0 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb07169b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c06bfe0 0x7fb06c158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57614/0 (socket says 172.21.15.172:57614) 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb07169b640 1 -- 172.21.15.172:0/4242120211 learned_addr learned my addr 172.21.15.172:0/4242120211 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb071e9c640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb06c15ff20 0x7fb06c152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb070e9a640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb06c071be0 0x7fb06c15b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:08.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb07169b640 1 -- 172.21.15.172:0/4242120211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb06c071be0 msgr2=0x7fb06c15b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb07169b640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb06c071be0 0x7fb06c15b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb07169b640 1 -- 172.21.15.172:0/4242120211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb06c15ff20 msgr2=0x7fb06c152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb07169b640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb06c15ff20 0x7fb06c152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb07169b640 1 -- 172.21.15.172:0/4242120211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb064009b30 con 0x7fb06c06bfe0 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.519+0000 7fb070e9a640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb06c071be0 0x7fb06c15b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.520+0000 7fb07169b640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c06bfe0 0x7fb06c158770 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb064000f80 tx=0x7fb064004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.520+0000 7fb061ffb640 1 -- 172.21.15.172:0/4242120211 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06403e450 con 0x7fb06c06bfe0 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.520+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06c159020 con 0x7fb06c06bfe0 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.520+0000 7fb061ffb640 1 -- 172.21.15.172:0/4242120211 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06403e5b0 con 0x7fb06c06bfe0 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.520+0000 7fb061ffb640 1 -- 172.21.15.172:0/4242120211 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06403e450 con 0x7fb06c06bfe0 2024-03-16T19:32:08.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.520+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c1595e0 con 0x7fb06c06bfe0 2024-03-16T19:32:08.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.522+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c158540 con 0x7fb06c06bfe0 2024-03-16T19:32:08.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.522+0000 7fb061ffb640 1 -- 172.21.15.172:0/4242120211 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb06403e720 con 0x7fb06c06bfe0 2024-03-16T19:32:08.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.522+0000 7fb061ffb640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb05006b470 0x7fb05006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:08.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.522+0000 7fb061ffb640 1 -- 172.21.15.172:0/4242120211 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fb06407b0b0 con 0x7fb06c06bfe0 2024-03-16T19:32:08.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.523+0000 7fb070e9a640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb05006b470 0x7fb05006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:08.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.524+0000 7fb070e9a640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb05006b470 0x7fb05006d930 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb06c167bb0 tx=0x7fb05c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:08.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.529+0000 7fb061ffb640 1 -- 172.21.15.172:0/4242120211 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb064077ae0 con 0x7fb06c06bfe0 2024-03-16T19:32:08.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.837+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb06c0c8560 con 0x7fb05006b470 2024-03-16T19:32:08.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.847+0000 7fb061ffb640 1 -- 172.21.15.172:0/4242120211 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252318 (secure 0 0 0) 0x7fb05c00fa30 con 0x7fb05006b470 2024-03-16T19:32:08.831 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:08.832 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb05006b470 msgr2=0x7fb05006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb05006b470 0x7fb05006d930 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb06c167bb0 tx=0x7fb05c009040 comp rx=0 tx=0).stop 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c06bfe0 msgr2=0x7fb06c158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c06bfe0 0x7fb06c158770 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb064000f80 tx=0x7fb064004a40 comp rx=0 tx=0).stop 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 shutdown_connections 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb06c15ff20 0x7fb06c152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb05006b470 0x7fb05006d930 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb06c071be0 0x7fb06c15b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 --2- 172.21.15.172:0/4242120211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb06c06bfe0 0x7fb06c158770 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 >> 172.21.15.172:0/4242120211 conn(0x7fb06c072b40 msgr2=0x7fb06c1508b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 shutdown_connections 2024-03-16T19:32:08.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:08.852+0000 7fb0738ff640 1 -- 172.21.15.172:0/4242120211 wait complete. 2024-03-16T19:32:08.848 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-03-16T19:32:08.230941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630862,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":245,"num_write_kb":2489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228360,"kb_used_data":11040,"kb_used_omap":16,"kb_used_meta":217199,"kb_avail":754746360,"statfs":{"total":773094113280,"available":772860272640,"internally_reserved":0,"allocated":11304960,"data_stored":5962108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":222412203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.179506"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081164+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:07.081164+0000","last_peered":"2024-03-16T19:32:07.081164+0000","last_clean":"2024-03-16T19:32:07.081164+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:07.081164+0000","last_undegraded":"2024-03-16T19:32:07.081164+0000","last_fullsized":"2024-03-16T19:32:07.081164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":185,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294885+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:05.294885+0000","last_peered":"2024-03-16T19:32:05.294885+0000","last_clean":"2024-03-16T19:32:05.294885+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:05.294885+0000","last_undegraded":"2024-03-16T19:32:05.294885+0000","last_fullsized":"2024-03-16T19:32:05.294885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077201+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:07.077201+0000","last_peered":"2024-03-16T19:32:07.077201+0000","last_clean":"2024-03-16T19:32:07.077201+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:07.077201+0000","last_undegraded":"2024-03-16T19:32:07.077201+0000","last_fullsized":"2024-03-16T19:32:07.077201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:47.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083818+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:08.083818+0000","last_peered":"2024-03-16T19:32:08.083818+0000","last_clean":"2024-03-16T19:32:08.083818+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:08.083818+0000","last_undegraded":"2024-03-16T19:32:08.083818+0000","last_fullsized":"2024-03-16T19:32:08.083818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080899+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:07.080899+0000","last_peered":"2024-03-16T19:32:07.080899+0000","last_clean":"2024-03-16T19:32:07.080899+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:07.080899+0000","last_undegraded":"2024-03-16T19:32:07.080899+0000","last_fullsized":"2024-03-16T19:32:07.080899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083640+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:08.083640+0000","last_peered":"2024-03-16T19:32:08.083640+0000","last_clean":"2024-03-16T19:32:08.083640+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:08.083640+0000","last_undegraded":"2024-03-16T19:32:08.083640+0000","last_fullsized":"2024-03-16T19:32:08.083640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":187,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080786+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:07.080786+0000","last_peered":"2024-03-16T19:32:07.080786+0000","last_clean":"2024-03-16T19:32:07.080786+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:07.080786+0000","last_undegraded":"2024-03-16T19:32:07.080786+0000","last_fullsized":"2024-03-16T19:32:07.080786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086880+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:08.086880+0000","last_peered":"2024-03-16T19:32:08.086880+0000","last_clean":"2024-03-16T19:32:08.086880+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:08.086880+0000","last_undegraded":"2024-03-16T19:32:08.086880+0000","last_fullsized":"2024-03-16T19:32:08.086880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:58:31.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081177+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:07.081177+0000","last_peered":"2024-03-16T19:32:07.081177+0000","last_clean":"2024-03-16T19:32:07.081177+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:07.081177+0000","last_undegraded":"2024-03-16T19:32:07.081177+0000","last_fullsized":"2024-03-16T19:32:07.081177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620209+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:07.620209+0000","last_peered":"2024-03-16T19:32:07.620209+0000","last_clean":"2024-03-16T19:32:07.620209+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:07.620209+0000","last_undegraded":"2024-03-16T19:32:07.620209+0000","last_fullsized":"2024-03-16T19:32:07.620209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620126+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:07.620126+0000","last_peered":"2024-03-16T19:32:07.620126+0000","last_clean":"2024-03-16T19:32:07.620126+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:07.620126+0000","last_undegraded":"2024-03-16T19:32:07.620126+0000","last_fullsized":"2024-03-16T19:32:07.620126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084143+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:08.084143+0000","last_peered":"2024-03-16T19:32:08.084143+0000","last_clean":"2024-03-16T19:32:08.084143+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:08.084143+0000","last_undegraded":"2024-03-16T19:32:08.084143+0000","last_fullsized":"2024-03-16T19:32:08.084143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088886+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:08.088886+0000","last_peered":"2024-03-16T19:32:08.088886+0000","last_clean":"2024-03-16T19:32:08.088886+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:08.088886+0000","last_undegraded":"2024-03-16T19:32:08.088886+0000","last_fullsized":"2024-03-16T19:32:08.088886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:52.102388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"84'3","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077055+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:07.077055+0000","last_peered":"2024-03-16T19:32:07.077055+0000","last_clean":"2024-03-16T19:32:07.077055+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:07.077055+0000","last_undegraded":"2024-03-16T19:32:07.077055+0000","last_fullsized":"2024-03-16T19:32:07.077055+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:39.289238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":175,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294251+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:05.294251+0000","last_peered":"2024-03-16T19:32:05.294251+0000","last_clean":"2024-03-16T19:32:05.294251+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:05.294251+0000","last_undegraded":"2024-03-16T19:32:05.294251+0000","last_fullsized":"2024-03-16T19:32:05.294251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083403+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:08.083403+0000","last_peered":"2024-03-16T19:32:08.083403+0000","last_clean":"2024-03-16T19:32:08.083403+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:08.083403+0000","last_undegraded":"2024-03-16T19:32:08.083403+0000","last_fullsized":"2024-03-16T19:32:08.083403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084254+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:08.084254+0000","last_peered":"2024-03-16T19:32:08.084254+0000","last_clean":"2024-03-16T19:32:08.084254+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:08.084254+0000","last_undegraded":"2024-03-16T19:32:08.084254+0000","last_fullsized":"2024-03-16T19:32:08.084254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080010+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:07.080010+0000","last_peered":"2024-03-16T19:32:07.080010+0000","last_clean":"2024-03-16T19:32:07.080010+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:07.080010+0000","last_undegraded":"2024-03-16T19:32:07.080010+0000","last_fullsized":"2024-03-16T19:32:07.080010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:36:06.175327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081155+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:07.081155+0000","last_peered":"2024-03-16T19:32:07.081155+0000","last_clean":"2024-03-16T19:32:07.081155+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:07.081155+0000","last_undegraded":"2024-03-16T19:32:07.081155+0000","last_fullsized":"2024-03-16T19:32:07.081155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620154+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:07.620154+0000","last_peered":"2024-03-16T19:32:07.620154+0000","last_clean":"2024-03-16T19:32:07.620154+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:07.620154+0000","last_undegraded":"2024-03-16T19:32:07.620154+0000","last_fullsized":"2024-03-16T19:32:07.620154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077197+0000","last_change":"2024-03-16T19:31:51.807295+0000","last_active":"2024-03-16T19:32:07.077197+0000","last_peered":"2024-03-16T19:32:07.077197+0000","last_clean":"2024-03-16T19:32:07.077197+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:07.077197+0000","last_undegraded":"2024-03-16T19:32:07.077197+0000","last_fullsized":"2024-03-16T19:32:07.077197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:04.578640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":50,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079813+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:07.079813+0000","last_peered":"2024-03-16T19:32:07.079813+0000","last_clean":"2024-03-16T19:32:07.079813+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:07.079813+0000","last_undegraded":"2024-03-16T19:32:07.079813+0000","last_fullsized":"2024-03-16T19:32:07.079813+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:41:06.377004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088245+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:08.088245+0000","last_peered":"2024-03-16T19:32:08.088245+0000","last_clean":"2024-03-16T19:32:08.088245+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:08.088245+0000","last_undegraded":"2024-03-16T19:32:08.088245+0000","last_fullsized":"2024-03-16T19:32:08.088245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:47:08.251341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083454+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:08.083454+0000","last_peered":"2024-03-16T19:32:08.083454+0000","last_clean":"2024-03-16T19:32:08.083454+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:08.083454+0000","last_undegraded":"2024-03-16T19:32:08.083454+0000","last_fullsized":"2024-03-16T19:32:08.083454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086134+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:08.086134+0000","last_peered":"2024-03-16T19:32:08.086134+0000","last_clean":"2024-03-16T19:32:08.086134+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:08.086134+0000","last_undegraded":"2024-03-16T19:32:08.086134+0000","last_fullsized":"2024-03-16T19:32:08.086134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:10.818175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620238+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:07.620238+0000","last_peered":"2024-03-16T19:32:07.620238+0000","last_clean":"2024-03-16T19:32:07.620238+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:07.620238+0000","last_undegraded":"2024-03-16T19:32:07.620238+0000","last_fullsized":"2024-03-16T19:32:07.620238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":184,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294420+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:05.294420+0000","last_peered":"2024-03-16T19:32:05.294420+0000","last_clean":"2024-03-16T19:32:05.294420+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:05.294420+0000","last_undegraded":"2024-03-16T19:32:05.294420+0000","last_fullsized":"2024-03-16T19:32:05.294420+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087892+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:08.087892+0000","last_peered":"2024-03-16T19:32:08.087892+0000","last_clean":"2024-03-16T19:32:08.087892+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:08.087892+0000","last_undegraded":"2024-03-16T19:32:08.087892+0000","last_fullsized":"2024-03-16T19:32:08.087892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:12:08.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077141+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:07.077141+0000","last_peered":"2024-03-16T19:32:07.077141+0000","last_clean":"2024-03-16T19:32:07.077141+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:07.077141+0000","last_undegraded":"2024-03-16T19:32:07.077141+0000","last_fullsized":"2024-03-16T19:32:07.077141+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:12.646493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083749+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:08.083749+0000","last_peered":"2024-03-16T19:32:08.083749+0000","last_clean":"2024-03-16T19:32:08.083749+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:08.083749+0000","last_undegraded":"2024-03-16T19:32:08.083749+0000","last_fullsized":"2024-03-16T19:32:08.083749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079774+0000","last_change":"2024-03-16T19:31:49.974297+0000","last_active":"2024-03-16T19:32:07.079774+0000","last_peered":"2024-03-16T19:32:07.079774+0000","last_clean":"2024-03-16T19:32:07.079774+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:07.079774+0000","last_undegraded":"2024-03-16T19:32:07.079774+0000","last_fullsized":"2024-03-16T19:32:07.079774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:51:57.535372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":186,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.619981+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:07.619981+0000","last_peered":"2024-03-16T19:32:07.619981+0000","last_clean":"2024-03-16T19:32:07.619981+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:07.619981+0000","last_undegraded":"2024-03-16T19:32:07.619981+0000","last_fullsized":"2024-03-16T19:32:07.619981+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084174+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:32:08.084174+0000","last_peered":"2024-03-16T19:32:08.084174+0000","last_clean":"2024-03-16T19:32:08.084174+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:08.084174+0000","last_undegraded":"2024-03-16T19:32:08.084174+0000","last_fullsized":"2024-03-16T19:32:08.084174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089011+0000","last_change":"2024-03-16T19:31:26.808032+0000","last_active":"2024-03-16T19:32:08.089011+0000","last_peered":"2024-03-16T19:32:08.089011+0000","last_clean":"2024-03-16T19:32:08.089011+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:08.089011+0000","last_undegraded":"2024-03-16T19:32:08.089011+0000","last_fullsized":"2024-03-16T19:32:08.089011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:15.218778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077339+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:07.077339+0000","last_peered":"2024-03-16T19:32:07.077339+0000","last_clean":"2024-03-16T19:32:07.077339+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:07.077339+0000","last_undegraded":"2024-03-16T19:32:07.077339+0000","last_fullsized":"2024-03-16T19:32:07.077339+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:16:31.606469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620053+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:07.620053+0000","last_peered":"2024-03-16T19:32:07.620053+0000","last_clean":"2024-03-16T19:32:07.620053+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:07.620053+0000","last_undegraded":"2024-03-16T19:32:07.620053+0000","last_fullsized":"2024-03-16T19:32:07.620053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086078+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086078+0000","last_peered":"2024-03-16T19:32:08.086078+0000","last_clean":"2024-03-16T19:32:08.086078+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086078+0000","last_undegraded":"2024-03-16T19:32:08.086078+0000","last_fullsized":"2024-03-16T19:32:08.086078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:23:19.824904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086998+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:08.086998+0000","last_peered":"2024-03-16T19:32:08.086998+0000","last_clean":"2024-03-16T19:32:08.086998+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:08.086998+0000","last_undegraded":"2024-03-16T19:32:08.086998+0000","last_fullsized":"2024-03-16T19:32:08.086998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:45.782085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":185,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294229+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:05.294229+0000","last_peered":"2024-03-16T19:32:05.294229+0000","last_clean":"2024-03-16T19:32:05.294229+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:05.294229+0000","last_undegraded":"2024-03-16T19:32:05.294229+0000","last_fullsized":"2024-03-16T19:32:05.294229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620263+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:07.620263+0000","last_peered":"2024-03-16T19:32:07.620263+0000","last_clean":"2024-03-16T19:32:07.620263+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:07.620263+0000","last_undegraded":"2024-03-16T19:32:07.620263+0000","last_fullsized":"2024-03-16T19:32:07.620263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083754+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:08.083754+0000","last_peered":"2024-03-16T19:32:08.083754+0000","last_clean":"2024-03-16T19:32:08.083754+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:08.083754+0000","last_undegraded":"2024-03-16T19:32:08.083754+0000","last_fullsized":"2024-03-16T19:32:08.083754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077496+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:07.077496+0000","last_peered":"2024-03-16T19:32:07.077496+0000","last_clean":"2024-03-16T19:32:07.077496+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:07.077496+0000","last_undegraded":"2024-03-16T19:32:07.077496+0000","last_fullsized":"2024-03-16T19:32:07.077496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:16.418962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":182,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294504+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:05.294504+0000","last_peered":"2024-03-16T19:32:05.294504+0000","last_clean":"2024-03-16T19:32:05.294504+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:05.294504+0000","last_undegraded":"2024-03-16T19:32:05.294504+0000","last_fullsized":"2024-03-16T19:32:05.294504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"peering","last_fresh":"2024-03-16T19:32:05.059246+0000","last_change":"2024-03-16T19:32:07.978687+0000","last_active":"2024-03-16T19:32:07.978687+0000","last_peered":"2024-03-16T19:32:05.059246+0000","last_clean":"2024-03-16T19:32:05.059246+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:32:05.059246+0000","last_undegraded":"2024-03-16T19:32:05.059246+0000","last_fullsized":"2024-03-16T19:32:05.059246+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"86'23","reported_seq":231,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620164+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:07.620164+0000","last_peered":"2024-03-16T19:32:07.620164+0000","last_clean":"2024-03-16T19:32:07.620164+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:07.620164+0000","last_undegraded":"2024-03-16T19:32:07.620164+0000","last_fullsized":"2024-03-16T19:32:07.620164+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39292,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":187,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080997+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:07.080997+0000","last_peered":"2024-03-16T19:32:07.080997+0000","last_clean":"2024-03-16T19:32:07.080997+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:07.080997+0000","last_undegraded":"2024-03-16T19:32:07.080997+0000","last_fullsized":"2024-03-16T19:32:07.080997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077290+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:07.077290+0000","last_peered":"2024-03-16T19:32:07.077290+0000","last_clean":"2024-03-16T19:32:07.077290+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:07.077290+0000","last_undegraded":"2024-03-16T19:32:07.077290+0000","last_fullsized":"2024-03-16T19:32:07.077290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:59.818393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.076992+0000","last_change":"2024-03-16T19:31:51.807895+0000","last_active":"2024-03-16T19:32:07.076992+0000","last_peered":"2024-03-16T19:32:07.076992+0000","last_clean":"2024-03-16T19:32:07.076992+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:07.076992+0000","last_undegraded":"2024-03-16T19:32:07.076992+0000","last_fullsized":"2024-03-16T19:32:07.076992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:25.988237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080958+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:07.080958+0000","last_peered":"2024-03-16T19:32:07.080958+0000","last_clean":"2024-03-16T19:32:07.080958+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:07.080958+0000","last_undegraded":"2024-03-16T19:32:07.080958+0000","last_fullsized":"2024-03-16T19:32:07.080958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":178,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620021+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:07.620021+0000","last_peered":"2024-03-16T19:32:07.620021+0000","last_clean":"2024-03-16T19:32:07.620021+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:07.620021+0000","last_undegraded":"2024-03-16T19:32:07.620021+0000","last_fullsized":"2024-03-16T19:32:07.620021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":186,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.619987+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:07.619987+0000","last_peered":"2024-03-16T19:32:07.619987+0000","last_clean":"2024-03-16T19:32:07.619987+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:07.619987+0000","last_undegraded":"2024-03-16T19:32:07.619987+0000","last_fullsized":"2024-03-16T19:32:07.619987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":56,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079696+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:07.079696+0000","last_peered":"2024-03-16T19:32:07.079696+0000","last_clean":"2024-03-16T19:32:07.079696+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:07.079696+0000","last_undegraded":"2024-03-16T19:32:07.079696+0000","last_fullsized":"2024-03-16T19:32:07.079696+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:33.634146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080858+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:07.080858+0000","last_peered":"2024-03-16T19:32:07.080858+0000","last_clean":"2024-03-16T19:32:07.080858+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:07.080858+0000","last_undegraded":"2024-03-16T19:32:07.080858+0000","last_fullsized":"2024-03-16T19:32:07.080858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084319+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084319+0000","last_peered":"2024-03-16T19:32:08.084319+0000","last_clean":"2024-03-16T19:32:08.084319+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084319+0000","last_undegraded":"2024-03-16T19:32:08.084319+0000","last_fullsized":"2024-03-16T19:32:08.084319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:53.021797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081088+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:07.081088+0000","last_peered":"2024-03-16T19:32:07.081088+0000","last_clean":"2024-03-16T19:32:07.081088+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:07.081088+0000","last_undegraded":"2024-03-16T19:32:07.081088+0000","last_fullsized":"2024-03-16T19:32:07.081088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":53,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079883+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:07.079883+0000","last_peered":"2024-03-16T19:32:07.079883+0000","last_clean":"2024-03-16T19:32:07.079883+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:07.079883+0000","last_undegraded":"2024-03-16T19:32:07.079883+0000","last_fullsized":"2024-03-16T19:32:07.079883+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:07:39.853290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081238+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:07.081238+0000","last_peered":"2024-03-16T19:32:07.081238+0000","last_clean":"2024-03-16T19:32:07.081238+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:07.081238+0000","last_undegraded":"2024-03-16T19:32:07.081238+0000","last_fullsized":"2024-03-16T19:32:07.081238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":178,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294600+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:05.294600+0000","last_peered":"2024-03-16T19:32:05.294600+0000","last_clean":"2024-03-16T19:32:05.294600+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:05.294600+0000","last_undegraded":"2024-03-16T19:32:05.294600+0000","last_fullsized":"2024-03-16T19:32:05.294600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084002+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:08.084002+0000","last_peered":"2024-03-16T19:32:08.084002+0000","last_clean":"2024-03-16T19:32:08.084002+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:08.084002+0000","last_undegraded":"2024-03-16T19:32:08.084002+0000","last_fullsized":"2024-03-16T19:32:08.084002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079907+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:07.079907+0000","last_peered":"2024-03-16T19:32:07.079907+0000","last_clean":"2024-03-16T19:32:07.079907+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:07.079907+0000","last_undegraded":"2024-03-16T19:32:07.079907+0000","last_fullsized":"2024-03-16T19:32:07.079907+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:59:55.019036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":146,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080905+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:07.080905+0000","last_peered":"2024-03-16T19:32:07.080905+0000","last_clean":"2024-03-16T19:32:07.080905+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:07.080905+0000","last_undegraded":"2024-03-16T19:32:07.080905+0000","last_fullsized":"2024-03-16T19:32:07.080905+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":361,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080930+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:07.080930+0000","last_peered":"2024-03-16T19:32:07.080930+0000","last_clean":"2024-03-16T19:32:07.080930+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:07.080930+0000","last_undegraded":"2024-03-16T19:32:07.080930+0000","last_fullsized":"2024-03-16T19:32:07.080930+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087541+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:08.087541+0000","last_peered":"2024-03-16T19:32:08.087541+0000","last_clean":"2024-03-16T19:32:08.087541+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:08.087541+0000","last_undegraded":"2024-03-16T19:32:08.087541+0000","last_fullsized":"2024-03-16T19:32:08.087541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:16:54.768710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080738+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:07.080738+0000","last_peered":"2024-03-16T19:32:07.080738+0000","last_clean":"2024-03-16T19:32:07.080738+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:07.080738+0000","last_undegraded":"2024-03-16T19:32:07.080738+0000","last_fullsized":"2024-03-16T19:32:07.080738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077009+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:07.077009+0000","last_peered":"2024-03-16T19:32:07.077009+0000","last_clean":"2024-03-16T19:32:07.077009+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:07.077009+0000","last_undegraded":"2024-03-16T19:32:07.077009+0000","last_fullsized":"2024-03-16T19:32:07.077009+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:44.308565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":50,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079702+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:07.079702+0000","last_peered":"2024-03-16T19:32:07.079702+0000","last_clean":"2024-03-16T19:32:07.079702+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:07.079702+0000","last_undegraded":"2024-03-16T19:32:07.079702+0000","last_fullsized":"2024-03-16T19:32:07.079702+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:20:21.014268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":182,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294258+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:05.294258+0000","last_peered":"2024-03-16T19:32:05.294258+0000","last_clean":"2024-03-16T19:32:05.294258+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:05.294258+0000","last_undegraded":"2024-03-16T19:32:05.294258+0000","last_fullsized":"2024-03-16T19:32:05.294258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084314+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:08.084314+0000","last_peered":"2024-03-16T19:32:08.084314+0000","last_clean":"2024-03-16T19:32:08.084314+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:08.084314+0000","last_undegraded":"2024-03-16T19:32:08.084314+0000","last_fullsized":"2024-03-16T19:32:08.084314+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:01.856961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":187,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294385+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:05.294385+0000","last_peered":"2024-03-16T19:32:05.294385+0000","last_clean":"2024-03-16T19:32:05.294385+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:05.294385+0000","last_undegraded":"2024-03-16T19:32:05.294385+0000","last_fullsized":"2024-03-16T19:32:05.294385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084193+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:08.084193+0000","last_peered":"2024-03-16T19:32:08.084193+0000","last_clean":"2024-03-16T19:32:08.084193+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:08.084193+0000","last_undegraded":"2024-03-16T19:32:08.084193+0000","last_fullsized":"2024-03-16T19:32:08.084193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081176+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:07.081176+0000","last_peered":"2024-03-16T19:32:07.081176+0000","last_clean":"2024-03-16T19:32:07.081176+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:07.081176+0000","last_undegraded":"2024-03-16T19:32:07.081176+0000","last_fullsized":"2024-03-16T19:32:07.081176+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":182,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294339+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:05.294339+0000","last_peered":"2024-03-16T19:32:05.294339+0000","last_clean":"2024-03-16T19:32:05.294339+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:05.294339+0000","last_undegraded":"2024-03-16T19:32:05.294339+0000","last_fullsized":"2024-03-16T19:32:05.294339+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'11","reported_seq":188,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294977+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:05.294977+0000","last_peered":"2024-03-16T19:32:05.294977+0000","last_clean":"2024-03-16T19:32:05.294977+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:05.294977+0000","last_undegraded":"2024-03-16T19:32:05.294977+0000","last_fullsized":"2024-03-16T19:32:05.294977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088741+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:08.088741+0000","last_peered":"2024-03-16T19:32:08.088741+0000","last_clean":"2024-03-16T19:32:08.088741+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:08.088741+0000","last_undegraded":"2024-03-16T19:32:08.088741+0000","last_fullsized":"2024-03-16T19:32:08.088741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:18.996236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620312+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:07.620312+0000","last_peered":"2024-03-16T19:32:07.620312+0000","last_clean":"2024-03-16T19:32:07.620312+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:07.620312+0000","last_undegraded":"2024-03-16T19:32:07.620312+0000","last_fullsized":"2024-03-16T19:32:07.620312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077386+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:07.077386+0000","last_peered":"2024-03-16T19:32:07.077386+0000","last_clean":"2024-03-16T19:32:07.077386+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:07.077386+0000","last_undegraded":"2024-03-16T19:32:07.077386+0000","last_fullsized":"2024-03-16T19:32:07.077386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:02.024100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080734+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:07.080734+0000","last_peered":"2024-03-16T19:32:07.080734+0000","last_clean":"2024-03-16T19:32:07.080734+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:07.080734+0000","last_undegraded":"2024-03-16T19:32:07.080734+0000","last_fullsized":"2024-03-16T19:32:07.080734+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083795+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:08.083795+0000","last_peered":"2024-03-16T19:32:08.083795+0000","last_clean":"2024-03-16T19:32:08.083795+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:08.083795+0000","last_undegraded":"2024-03-16T19:32:08.083795+0000","last_fullsized":"2024-03-16T19:32:08.083795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:46:49.628764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":177,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294344+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:05.294344+0000","last_peered":"2024-03-16T19:32:05.294344+0000","last_clean":"2024-03-16T19:32:05.294344+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:05.294344+0000","last_undegraded":"2024-03-16T19:32:05.294344+0000","last_fullsized":"2024-03-16T19:32:05.294344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":64,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.181279+0000","last_change":"2024-03-16T19:31:26.809671+0000","last_active":"2024-03-16T19:32:05.181279+0000","last_peered":"2024-03-16T19:32:05.181279+0000","last_clean":"2024-03-16T19:32:05.181279+0000","last_became_active":"2024-03-16T19:31:26.809344+0000","last_became_peered":"2024-03-16T19:31:26.809344+0000","last_unstale":"2024-03-16T19:32:05.181279+0000","last_undegraded":"2024-03-16T19:32:05.181279+0000","last_fullsized":"2024-03-16T19:32:05.181279+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:49.907553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":187,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294590+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:05.294590+0000","last_peered":"2024-03-16T19:32:05.294590+0000","last_clean":"2024-03-16T19:32:05.294590+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:05.294590+0000","last_undegraded":"2024-03-16T19:32:05.294590+0000","last_fullsized":"2024-03-16T19:32:05.294590+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089208+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:08.089208+0000","last_peered":"2024-03-16T19:32:08.089208+0000","last_clean":"2024-03-16T19:32:08.089208+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:08.089208+0000","last_undegraded":"2024-03-16T19:32:08.089208+0000","last_fullsized":"2024-03-16T19:32:08.089208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:03.618618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080995+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:07.080995+0000","last_peered":"2024-03-16T19:32:07.080995+0000","last_clean":"2024-03-16T19:32:07.080995+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:07.080995+0000","last_undegraded":"2024-03-16T19:32:07.080995+0000","last_fullsized":"2024-03-16T19:32:07.080995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620022+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:07.620022+0000","last_peered":"2024-03-16T19:32:07.620022+0000","last_clean":"2024-03-16T19:32:07.620022+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:07.620022+0000","last_undegraded":"2024-03-16T19:32:07.620022+0000","last_fullsized":"2024-03-16T19:32:07.620022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084106+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:08.084106+0000","last_peered":"2024-03-16T19:32:08.084106+0000","last_clean":"2024-03-16T19:32:08.084106+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:08.084106+0000","last_undegraded":"2024-03-16T19:32:08.084106+0000","last_fullsized":"2024-03-16T19:32:08.084106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081285+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:07.081285+0000","last_peered":"2024-03-16T19:32:07.081285+0000","last_clean":"2024-03-16T19:32:07.081285+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:07.081285+0000","last_undegraded":"2024-03-16T19:32:07.081285+0000","last_fullsized":"2024-03-16T19:32:07.081285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083913+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:08.083913+0000","last_peered":"2024-03-16T19:32:08.083913+0000","last_clean":"2024-03-16T19:32:08.083913+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:08.083913+0000","last_undegraded":"2024-03-16T19:32:08.083913+0000","last_fullsized":"2024-03-16T19:32:08.083913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081056+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:07.081056+0000","last_peered":"2024-03-16T19:32:07.081056+0000","last_clean":"2024-03-16T19:32:07.081056+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:07.081056+0000","last_undegraded":"2024-03-16T19:32:07.081056+0000","last_fullsized":"2024-03-16T19:32:07.081056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":162,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077568+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:07.077568+0000","last_peered":"2024-03-16T19:32:07.077568+0000","last_clean":"2024-03-16T19:32:07.077568+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:07.077568+0000","last_undegraded":"2024-03-16T19:32:07.077568+0000","last_fullsized":"2024-03-16T19:32:07.077568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:19.317523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088482+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088482+0000","last_peered":"2024-03-16T19:32:08.088482+0000","last_clean":"2024-03-16T19:32:08.088482+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088482+0000","last_undegraded":"2024-03-16T19:32:08.088482+0000","last_fullsized":"2024-03-16T19:32:08.088482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:58:01.695714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086003+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:08.086003+0000","last_peered":"2024-03-16T19:32:08.086003+0000","last_clean":"2024-03-16T19:32:08.086003+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:08.086003+0000","last_undegraded":"2024-03-16T19:32:08.086003+0000","last_fullsized":"2024-03-16T19:32:08.086003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:17.040490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077402+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:07.077402+0000","last_peered":"2024-03-16T19:32:07.077402+0000","last_clean":"2024-03-16T19:32:07.077402+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:07.077402+0000","last_undegraded":"2024-03-16T19:32:07.077402+0000","last_fullsized":"2024-03-16T19:32:07.077402+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:28:10.116570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40494,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":80988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28824,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343016,"statfs":{"total":96636764160,"available":96607248384,"internally_reserved":0,"allocated":1847296,"data_stored":1180033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343552,"statfs":{"total":96636764160,"available":96607797248,"internally_reserved":0,"allocated":1298432,"data_stored":628375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1253376,"data_stored":589173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1290240,"data_stored":622664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122054,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29384,"kb_used_data":1212,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342456,"statfs":{"total":96636764160,"available":96606674944,"internally_reserved":0,"allocated":1241088,"data_stored":583372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28240,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343600,"statfs":{"total":96636764160,"available":96607846400,"internally_reserved":0,"allocated":1245184,"data_stored":583128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28832,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343008,"statfs":{"total":96636764160,"available":96607240192,"internally_reserved":0,"allocated":1855488,"data_stored":1180089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:08.852 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:09.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.053+0000 7fbd3791e640 1 Processor -- start 2024-03-16T19:32:09.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.053+0000 7fbd3791e640 1 -- start start 2024-03-16T19:32:09.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd3791e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbd3006f490 0x7fbd3015aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd3791e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd300ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd3791e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd3015b940 con 0x7fbd30154d20 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd3791e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd3015b380 con 0x7fbd3006f490 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd3791e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3015b660 con 0x7fbd300ce2a0 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd35ebb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd300ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd34eb9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbd3006f490 0x7fbd3015aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd35ebb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd300ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:47774/0 (socket says 172.21.15.172:47774) 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd35ebb640 1 -- 172.21.15.172:0/3965952335 learned_addr learned my addr 172.21.15.172:0/3965952335 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd356ba640 1 --1- 172.21.15.172:0/3965952335 >> v1:172.21.15.174:6789/0 conn(0x7fbd30154d20 0x7fbd3006cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:49312/0 (socket says 172.21.15.172:49312) 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd35ebb640 1 -- 172.21.15.172:0/3965952335 >> v2:172.21.15.174:3300/0 conn(0x7fbd3006f490 msgr2=0x7fbd3015aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd35ebb640 1 --2- 172.21.15.172:0/3965952335 >> v2:172.21.15.174:3300/0 conn(0x7fbd3006f490 0x7fbd3015aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd35ebb640 1 -- 172.21.15.172:0/3965952335 >> v1:172.21.15.174:6789/0 conn(0x7fbd30154d20 legacy=0x7fbd3006cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd35ebb640 1 -- 172.21.15.172:0/3965952335 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3006c080 con 0x7fbd300ce2a0 2024-03-16T19:32:09.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.055+0000 7fbd34eb9640 1 --2- 172.21.15.172:0/3965952335 >> v2:172.21.15.174:3300/0 conn(0x7fbd3006f490 0x7fbd3015aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.056+0000 7fbd35ebb640 1 --2- 172.21.15.172:0/3965952335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd300ce730 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd28009870 tx=0x7fbd2802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e27cf16d67ac4af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.056+0000 7fbd2ffff640 1 -- 172.21.15.172:0/3965952335 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2802f830 con 0x7fbd300ce2a0 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.056+0000 7fbd2ffff640 1 -- 172.21.15.172:0/3965952335 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2802f990 con 0x7fbd300ce2a0 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.056+0000 7fbd2ffff640 1 -- 172.21.15.172:0/3965952335 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2802fd20 con 0x7fbd300ce2a0 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.057+0000 7fbd3791e640 1 -- 172.21.15.172:0/3965952335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 msgr2=0x7fbd300ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.057+0000 7fbd3791e640 1 --2- 172.21.15.172:0/3965952335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd300ce730 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd28009870 tx=0x7fbd2802ba10 comp rx=0 tx=0).stop 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.057+0000 7fbd3791e640 1 -- 172.21.15.172:0/3965952335 shutdown_connections 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.057+0000 7fbd3791e640 1 --2- 172.21.15.172:0/3965952335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd300ce730 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.057+0000 7fbd3791e640 1 --2- 172.21.15.172:0/3965952335 >> v2:172.21.15.174:3300/0 conn(0x7fbd3006f490 0x7fbd3015aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.057+0000 7fbd3791e640 1 -- 172.21.15.172:0/3965952335 >> 172.21.15.172:0/3965952335 conn(0x7fbd30066c20 msgr2=0x7fbd30067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:09.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.057+0000 7fbd3791e640 1 -- 172.21.15.172:0/3965952335 shutdown_connections 2024-03-16T19:32:09.040 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.057+0000 7fbd3791e640 1 -- 172.21.15.172:0/3965952335 wait complete. 2024-03-16T19:32:09.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.059+0000 7fbd3791e640 1 Processor -- start 2024-03-16T19:32:09.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.059+0000 7fbd3791e640 1 -- start start 2024-03-16T19:32:09.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd3791e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd3006f490 0x7fbd30159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd3791e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd3015a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd3791e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd30154d20 0x7fbd3014db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd3791e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd301f8ac0 con 0x7fbd30154d20 2024-03-16T19:32:09.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd3791e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd30150d30 con 0x7fbd3006f490 2024-03-16T19:32:09.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd3791e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd30153ba0 con 0x7fbd300ce2a0 2024-03-16T19:32:09.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd34eb9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd3015a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd34eb9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd3015a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:47788/0 (socket says 172.21.15.172:47788) 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd34eb9640 1 -- 172.21.15.172:0/4027386685 learned_addr learned my addr 172.21.15.172:0/4027386685 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd356ba640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd3006f490 0x7fbd30159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd35ebb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd30154d20 0x7fbd3014db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd34eb9640 1 -- 172.21.15.172:0/4027386685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd3006f490 msgr2=0x7fbd30159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd34eb9640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd3006f490 0x7fbd30159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd34eb9640 1 -- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd30154d20 msgr2=0x7fbd3014db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd34eb9640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd30154d20 0x7fbd3014db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd34eb9640 1 -- 172.21.15.172:0/4027386685 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd280094d0 con 0x7fbd300ce2a0 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.060+0000 7fbd35ebb640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd30154d20 0x7fbd3014db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:09.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.061+0000 7fbd34eb9640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd3015a9d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400c7f0 tx=0x7fbd2400cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:09.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.061+0000 7fbd2dffb640 1 -- 172.21.15.172:0/4027386685 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd240079d0 con 0x7fbd300ce2a0 2024-03-16T19:32:09.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.061+0000 7fbd2dffb640 1 -- 172.21.15.172:0/4027386685 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd24007b30 con 0x7fbd300ce2a0 2024-03-16T19:32:09.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.061+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd301f7f40 con 0x7fbd300ce2a0 2024-03-16T19:32:09.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.061+0000 7fbd2dffb640 1 -- 172.21.15.172:0/4027386685 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2400ce40 con 0x7fbd300ce2a0 2024-03-16T19:32:09.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.061+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd301f8500 con 0x7fbd300ce2a0 2024-03-16T19:32:09.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.062+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd30150b00 con 0x7fbd300ce2a0 2024-03-16T19:32:09.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.063+0000 7fbd2dffb640 1 -- 172.21.15.172:0/4027386685 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbd240113f0 con 0x7fbd300ce2a0 2024-03-16T19:32:09.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.063+0000 7fbd2dffb640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbd1c06b3b0 0x7fbd1c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.063+0000 7fbd2dffb640 1 -- 172.21.15.172:0/4027386685 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fbd24089d80 con 0x7fbd300ce2a0 2024-03-16T19:32:09.044 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.063+0000 7fbd356ba640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbd1c06b3b0 0x7fbd1c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.064+0000 7fbd356ba640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbd1c06b3b0 0x7fbd1c06d870 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbd20007fd0 tx=0x7fbd20001630 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:09.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.067+0000 7fbd2dffb640 1 -- 172.21.15.172:0/4027386685 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd30150b00 con 0x7fbd300ce2a0 2024-03-16T19:32:09.356 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.375+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd300c8df0 con 0x7fbd1c06b3b0 2024-03-16T19:32:09.365 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.385+0000 7fbd2dffb640 1 -- 172.21.15.172:0/4027386685 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252318 (secure 0 0 0) 0x7fbd20001fb0 con 0x7fbd1c06b3b0 2024-03-16T19:32:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbd1c06b3b0 msgr2=0x7fbd1c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbd1c06b3b0 0x7fbd1c06d870 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbd20007fd0 tx=0x7fbd20001630 comp rx=0 tx=0).stop 2024-03-16T19:32:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 msgr2=0x7fbd3015a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd3015a9d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400c7f0 tx=0x7fbd2400cbf0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 shutdown_connections 2024-03-16T19:32:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd30154d20 0x7fbd3014db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbd1c06b3b0 0x7fbd1c06d870 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd300ce2a0 0x7fbd3015a9d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 --2- 172.21.15.172:0/4027386685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd3006f490 0x7fbd30159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 >> 172.21.15.172:0/4027386685 conn(0x7fbd30066c20 msgr2=0x7fbd300706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 shutdown_connections 2024-03-16T19:32:09.372 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.389+0000 7fbd3791e640 1 -- 172.21.15.172:0/4027386685 wait complete. 2024-03-16T19:32:09.385 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-03-16T19:32:08.230941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630862,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":245,"num_write_kb":2489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228360,"kb_used_data":11040,"kb_used_omap":16,"kb_used_meta":217199,"kb_avail":754746360,"statfs":{"total":773094113280,"available":772860272640,"internally_reserved":0,"allocated":11304960,"data_stored":5962108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":222412203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.179506"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081164+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:07.081164+0000","last_peered":"2024-03-16T19:32:07.081164+0000","last_clean":"2024-03-16T19:32:07.081164+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:07.081164+0000","last_undegraded":"2024-03-16T19:32:07.081164+0000","last_fullsized":"2024-03-16T19:32:07.081164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":185,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294885+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:05.294885+0000","last_peered":"2024-03-16T19:32:05.294885+0000","last_clean":"2024-03-16T19:32:05.294885+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:05.294885+0000","last_undegraded":"2024-03-16T19:32:05.294885+0000","last_fullsized":"2024-03-16T19:32:05.294885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077201+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:07.077201+0000","last_peered":"2024-03-16T19:32:07.077201+0000","last_clean":"2024-03-16T19:32:07.077201+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:07.077201+0000","last_undegraded":"2024-03-16T19:32:07.077201+0000","last_fullsized":"2024-03-16T19:32:07.077201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:47.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083818+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:08.083818+0000","last_peered":"2024-03-16T19:32:08.083818+0000","last_clean":"2024-03-16T19:32:08.083818+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:08.083818+0000","last_undegraded":"2024-03-16T19:32:08.083818+0000","last_fullsized":"2024-03-16T19:32:08.083818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080899+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:07.080899+0000","last_peered":"2024-03-16T19:32:07.080899+0000","last_clean":"2024-03-16T19:32:07.080899+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:07.080899+0000","last_undegraded":"2024-03-16T19:32:07.080899+0000","last_fullsized":"2024-03-16T19:32:07.080899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083640+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:08.083640+0000","last_peered":"2024-03-16T19:32:08.083640+0000","last_clean":"2024-03-16T19:32:08.083640+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:08.083640+0000","last_undegraded":"2024-03-16T19:32:08.083640+0000","last_fullsized":"2024-03-16T19:32:08.083640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":187,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080786+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:07.080786+0000","last_peered":"2024-03-16T19:32:07.080786+0000","last_clean":"2024-03-16T19:32:07.080786+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:07.080786+0000","last_undegraded":"2024-03-16T19:32:07.080786+0000","last_fullsized":"2024-03-16T19:32:07.080786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086880+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:08.086880+0000","last_peered":"2024-03-16T19:32:08.086880+0000","last_clean":"2024-03-16T19:32:08.086880+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:08.086880+0000","last_undegraded":"2024-03-16T19:32:08.086880+0000","last_fullsized":"2024-03-16T19:32:08.086880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:58:31.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081177+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:07.081177+0000","last_peered":"2024-03-16T19:32:07.081177+0000","last_clean":"2024-03-16T19:32:07.081177+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:07.081177+0000","last_undegraded":"2024-03-16T19:32:07.081177+0000","last_fullsized":"2024-03-16T19:32:07.081177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620209+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:07.620209+0000","last_peered":"2024-03-16T19:32:07.620209+0000","last_clean":"2024-03-16T19:32:07.620209+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:07.620209+0000","last_undegraded":"2024-03-16T19:32:07.620209+0000","last_fullsized":"2024-03-16T19:32:07.620209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620126+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:07.620126+0000","last_peered":"2024-03-16T19:32:07.620126+0000","last_clean":"2024-03-16T19:32:07.620126+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:07.620126+0000","last_undegraded":"2024-03-16T19:32:07.620126+0000","last_fullsized":"2024-03-16T19:32:07.620126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084143+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:08.084143+0000","last_peered":"2024-03-16T19:32:08.084143+0000","last_clean":"2024-03-16T19:32:08.084143+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:08.084143+0000","last_undegraded":"2024-03-16T19:32:08.084143+0000","last_fullsized":"2024-03-16T19:32:08.084143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088886+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:08.088886+0000","last_peered":"2024-03-16T19:32:08.088886+0000","last_clean":"2024-03-16T19:32:08.088886+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:08.088886+0000","last_undegraded":"2024-03-16T19:32:08.088886+0000","last_fullsized":"2024-03-16T19:32:08.088886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:52.102388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"84'3","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077055+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:07.077055+0000","last_peered":"2024-03-16T19:32:07.077055+0000","last_clean":"2024-03-16T19:32:07.077055+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:07.077055+0000","last_undegraded":"2024-03-16T19:32:07.077055+0000","last_fullsized":"2024-03-16T19:32:07.077055+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:39.289238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":175,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294251+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:05.294251+0000","last_peered":"2024-03-16T19:32:05.294251+0000","last_clean":"2024-03-16T19:32:05.294251+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:05.294251+0000","last_undegraded":"2024-03-16T19:32:05.294251+0000","last_fullsized":"2024-03-16T19:32:05.294251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083403+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:08.083403+0000","last_peered":"2024-03-16T19:32:08.083403+0000","last_clean":"2024-03-16T19:32:08.083403+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:08.083403+0000","last_undegraded":"2024-03-16T19:32:08.083403+0000","last_fullsized":"2024-03-16T19:32:08.083403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084254+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:08.084254+0000","last_peered":"2024-03-16T19:32:08.084254+0000","last_clean":"2024-03-16T19:32:08.084254+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:08.084254+0000","last_undegraded":"2024-03-16T19:32:08.084254+0000","last_fullsized":"2024-03-16T19:32:08.084254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080010+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:07.080010+0000","last_peered":"2024-03-16T19:32:07.080010+0000","last_clean":"2024-03-16T19:32:07.080010+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:07.080010+0000","last_undegraded":"2024-03-16T19:32:07.080010+0000","last_fullsized":"2024-03-16T19:32:07.080010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:36:06.175327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081155+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:07.081155+0000","last_peered":"2024-03-16T19:32:07.081155+0000","last_clean":"2024-03-16T19:32:07.081155+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:07.081155+0000","last_undegraded":"2024-03-16T19:32:07.081155+0000","last_fullsized":"2024-03-16T19:32:07.081155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620154+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:07.620154+0000","last_peered":"2024-03-16T19:32:07.620154+0000","last_clean":"2024-03-16T19:32:07.620154+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:07.620154+0000","last_undegraded":"2024-03-16T19:32:07.620154+0000","last_fullsized":"2024-03-16T19:32:07.620154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077197+0000","last_change":"2024-03-16T19:31:51.807295+0000","last_active":"2024-03-16T19:32:07.077197+0000","last_peered":"2024-03-16T19:32:07.077197+0000","last_clean":"2024-03-16T19:32:07.077197+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:07.077197+0000","last_undegraded":"2024-03-16T19:32:07.077197+0000","last_fullsized":"2024-03-16T19:32:07.077197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:04.578640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":50,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079813+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:07.079813+0000","last_peered":"2024-03-16T19:32:07.079813+0000","last_clean":"2024-03-16T19:32:07.079813+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:07.079813+0000","last_undegraded":"2024-03-16T19:32:07.079813+0000","last_fullsized":"2024-03-16T19:32:07.079813+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:41:06.377004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088245+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:08.088245+0000","last_peered":"2024-03-16T19:32:08.088245+0000","last_clean":"2024-03-16T19:32:08.088245+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:08.088245+0000","last_undegraded":"2024-03-16T19:32:08.088245+0000","last_fullsized":"2024-03-16T19:32:08.088245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:47:08.251341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083454+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:08.083454+0000","last_peered":"2024-03-16T19:32:08.083454+0000","last_clean":"2024-03-16T19:32:08.083454+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:08.083454+0000","last_undegraded":"2024-03-16T19:32:08.083454+0000","last_fullsized":"2024-03-16T19:32:08.083454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086134+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:08.086134+0000","last_peered":"2024-03-16T19:32:08.086134+0000","last_clean":"2024-03-16T19:32:08.086134+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:08.086134+0000","last_undegraded":"2024-03-16T19:32:08.086134+0000","last_fullsized":"2024-03-16T19:32:08.086134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:10.818175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620238+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:07.620238+0000","last_peered":"2024-03-16T19:32:07.620238+0000","last_clean":"2024-03-16T19:32:07.620238+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:07.620238+0000","last_undegraded":"2024-03-16T19:32:07.620238+0000","last_fullsized":"2024-03-16T19:32:07.620238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":184,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294420+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:05.294420+0000","last_peered":"2024-03-16T19:32:05.294420+0000","last_clean":"2024-03-16T19:32:05.294420+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:05.294420+0000","last_undegraded":"2024-03-16T19:32:05.294420+0000","last_fullsized":"2024-03-16T19:32:05.294420+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087892+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:08.087892+0000","last_peered":"2024-03-16T19:32:08.087892+0000","last_clean":"2024-03-16T19:32:08.087892+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:08.087892+0000","last_undegraded":"2024-03-16T19:32:08.087892+0000","last_fullsized":"2024-03-16T19:32:08.087892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:12:08.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077141+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:07.077141+0000","last_peered":"2024-03-16T19:32:07.077141+0000","last_clean":"2024-03-16T19:32:07.077141+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:07.077141+0000","last_undegraded":"2024-03-16T19:32:07.077141+0000","last_fullsized":"2024-03-16T19:32:07.077141+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:12.646493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083749+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:08.083749+0000","last_peered":"2024-03-16T19:32:08.083749+0000","last_clean":"2024-03-16T19:32:08.083749+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:08.083749+0000","last_undegraded":"2024-03-16T19:32:08.083749+0000","last_fullsized":"2024-03-16T19:32:08.083749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079774+0000","last_change":"2024-03-16T19:31:49.974297+0000","last_active":"2024-03-16T19:32:07.079774+0000","last_peered":"2024-03-16T19:32:07.079774+0000","last_clean":"2024-03-16T19:32:07.079774+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:07.079774+0000","last_undegraded":"2024-03-16T19:32:07.079774+0000","last_fullsized":"2024-03-16T19:32:07.079774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:51:57.535372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":186,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.619981+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:07.619981+0000","last_peered":"2024-03-16T19:32:07.619981+0000","last_clean":"2024-03-16T19:32:07.619981+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:07.619981+0000","last_undegraded":"2024-03-16T19:32:07.619981+0000","last_fullsized":"2024-03-16T19:32:07.619981+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084174+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:32:08.084174+0000","last_peered":"2024-03-16T19:32:08.084174+0000","last_clean":"2024-03-16T19:32:08.084174+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:08.084174+0000","last_undegraded":"2024-03-16T19:32:08.084174+0000","last_fullsized":"2024-03-16T19:32:08.084174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089011+0000","last_change":"2024-03-16T19:31:26.808032+0000","last_active":"2024-03-16T19:32:08.089011+0000","last_peered":"2024-03-16T19:32:08.089011+0000","last_clean":"2024-03-16T19:32:08.089011+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:08.089011+0000","last_undegraded":"2024-03-16T19:32:08.089011+0000","last_fullsized":"2024-03-16T19:32:08.089011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:15.218778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077339+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:07.077339+0000","last_peered":"2024-03-16T19:32:07.077339+0000","last_clean":"2024-03-16T19:32:07.077339+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:07.077339+0000","last_undegraded":"2024-03-16T19:32:07.077339+0000","last_fullsized":"2024-03-16T19:32:07.077339+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:16:31.606469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620053+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:07.620053+0000","last_peered":"2024-03-16T19:32:07.620053+0000","last_clean":"2024-03-16T19:32:07.620053+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:07.620053+0000","last_undegraded":"2024-03-16T19:32:07.620053+0000","last_fullsized":"2024-03-16T19:32:07.620053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086078+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086078+0000","last_peered":"2024-03-16T19:32:08.086078+0000","last_clean":"2024-03-16T19:32:08.086078+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086078+0000","last_undegraded":"2024-03-16T19:32:08.086078+0000","last_fullsized":"2024-03-16T19:32:08.086078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:23:19.824904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086998+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:08.086998+0000","last_peered":"2024-03-16T19:32:08.086998+0000","last_clean":"2024-03-16T19:32:08.086998+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:08.086998+0000","last_undegraded":"2024-03-16T19:32:08.086998+0000","last_fullsized":"2024-03-16T19:32:08.086998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:45.782085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":185,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294229+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:05.294229+0000","last_peered":"2024-03-16T19:32:05.294229+0000","last_clean":"2024-03-16T19:32:05.294229+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:05.294229+0000","last_undegraded":"2024-03-16T19:32:05.294229+0000","last_fullsized":"2024-03-16T19:32:05.294229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620263+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:07.620263+0000","last_peered":"2024-03-16T19:32:07.620263+0000","last_clean":"2024-03-16T19:32:07.620263+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:07.620263+0000","last_undegraded":"2024-03-16T19:32:07.620263+0000","last_fullsized":"2024-03-16T19:32:07.620263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083754+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:08.083754+0000","last_peered":"2024-03-16T19:32:08.083754+0000","last_clean":"2024-03-16T19:32:08.083754+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:08.083754+0000","last_undegraded":"2024-03-16T19:32:08.083754+0000","last_fullsized":"2024-03-16T19:32:08.083754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077496+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:07.077496+0000","last_peered":"2024-03-16T19:32:07.077496+0000","last_clean":"2024-03-16T19:32:07.077496+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:07.077496+0000","last_undegraded":"2024-03-16T19:32:07.077496+0000","last_fullsized":"2024-03-16T19:32:07.077496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:16.418962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":182,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294504+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:05.294504+0000","last_peered":"2024-03-16T19:32:05.294504+0000","last_clean":"2024-03-16T19:32:05.294504+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:05.294504+0000","last_undegraded":"2024-03-16T19:32:05.294504+0000","last_fullsized":"2024-03-16T19:32:05.294504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"peering","last_fresh":"2024-03-16T19:32:05.059246+0000","last_change":"2024-03-16T19:32:07.978687+0000","last_active":"2024-03-16T19:32:07.978687+0000","last_peered":"2024-03-16T19:32:05.059246+0000","last_clean":"2024-03-16T19:32:05.059246+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:32:05.059246+0000","last_undegraded":"2024-03-16T19:32:05.059246+0000","last_fullsized":"2024-03-16T19:32:05.059246+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"86'23","reported_seq":231,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620164+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:07.620164+0000","last_peered":"2024-03-16T19:32:07.620164+0000","last_clean":"2024-03-16T19:32:07.620164+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:07.620164+0000","last_undegraded":"2024-03-16T19:32:07.620164+0000","last_fullsized":"2024-03-16T19:32:07.620164+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39292,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":187,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080997+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:07.080997+0000","last_peered":"2024-03-16T19:32:07.080997+0000","last_clean":"2024-03-16T19:32:07.080997+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:07.080997+0000","last_undegraded":"2024-03-16T19:32:07.080997+0000","last_fullsized":"2024-03-16T19:32:07.080997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077290+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:07.077290+0000","last_peered":"2024-03-16T19:32:07.077290+0000","last_clean":"2024-03-16T19:32:07.077290+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:07.077290+0000","last_undegraded":"2024-03-16T19:32:07.077290+0000","last_fullsized":"2024-03-16T19:32:07.077290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:59.818393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.076992+0000","last_change":"2024-03-16T19:31:51.807895+0000","last_active":"2024-03-16T19:32:07.076992+0000","last_peered":"2024-03-16T19:32:07.076992+0000","last_clean":"2024-03-16T19:32:07.076992+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:07.076992+0000","last_undegraded":"2024-03-16T19:32:07.076992+0000","last_fullsized":"2024-03-16T19:32:07.076992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:25.988237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080958+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:07.080958+0000","last_peered":"2024-03-16T19:32:07.080958+0000","last_clean":"2024-03-16T19:32:07.080958+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:07.080958+0000","last_undegraded":"2024-03-16T19:32:07.080958+0000","last_fullsized":"2024-03-16T19:32:07.080958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":178,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620021+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:07.620021+0000","last_peered":"2024-03-16T19:32:07.620021+0000","last_clean":"2024-03-16T19:32:07.620021+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:07.620021+0000","last_undegraded":"2024-03-16T19:32:07.620021+0000","last_fullsized":"2024-03-16T19:32:07.620021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":186,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.619987+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:07.619987+0000","last_peered":"2024-03-16T19:32:07.619987+0000","last_clean":"2024-03-16T19:32:07.619987+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:07.619987+0000","last_undegraded":"2024-03-16T19:32:07.619987+0000","last_fullsized":"2024-03-16T19:32:07.619987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":56,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079696+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:07.079696+0000","last_peered":"2024-03-16T19:32:07.079696+0000","last_clean":"2024-03-16T19:32:07.079696+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:07.079696+0000","last_undegraded":"2024-03-16T19:32:07.079696+0000","last_fullsized":"2024-03-16T19:32:07.079696+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:33.634146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080858+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:07.080858+0000","last_peered":"2024-03-16T19:32:07.080858+0000","last_clean":"2024-03-16T19:32:07.080858+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:07.080858+0000","last_undegraded":"2024-03-16T19:32:07.080858+0000","last_fullsized":"2024-03-16T19:32:07.080858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084319+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084319+0000","last_peered":"2024-03-16T19:32:08.084319+0000","last_clean":"2024-03-16T19:32:08.084319+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084319+0000","last_undegraded":"2024-03-16T19:32:08.084319+0000","last_fullsized":"2024-03-16T19:32:08.084319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:53.021797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081088+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:07.081088+0000","last_peered":"2024-03-16T19:32:07.081088+0000","last_clean":"2024-03-16T19:32:07.081088+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:07.081088+0000","last_undegraded":"2024-03-16T19:32:07.081088+0000","last_fullsized":"2024-03-16T19:32:07.081088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":53,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079883+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:07.079883+0000","last_peered":"2024-03-16T19:32:07.079883+0000","last_clean":"2024-03-16T19:32:07.079883+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:07.079883+0000","last_undegraded":"2024-03-16T19:32:07.079883+0000","last_fullsized":"2024-03-16T19:32:07.079883+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:07:39.853290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081238+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:07.081238+0000","last_peered":"2024-03-16T19:32:07.081238+0000","last_clean":"2024-03-16T19:32:07.081238+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:07.081238+0000","last_undegraded":"2024-03-16T19:32:07.081238+0000","last_fullsized":"2024-03-16T19:32:07.081238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":178,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294600+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:05.294600+0000","last_peered":"2024-03-16T19:32:05.294600+0000","last_clean":"2024-03-16T19:32:05.294600+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:05.294600+0000","last_undegraded":"2024-03-16T19:32:05.294600+0000","last_fullsized":"2024-03-16T19:32:05.294600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084002+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:08.084002+0000","last_peered":"2024-03-16T19:32:08.084002+0000","last_clean":"2024-03-16T19:32:08.084002+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:08.084002+0000","last_undegraded":"2024-03-16T19:32:08.084002+0000","last_fullsized":"2024-03-16T19:32:08.084002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079907+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:07.079907+0000","last_peered":"2024-03-16T19:32:07.079907+0000","last_clean":"2024-03-16T19:32:07.079907+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:07.079907+0000","last_undegraded":"2024-03-16T19:32:07.079907+0000","last_fullsized":"2024-03-16T19:32:07.079907+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:59:55.019036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":146,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080905+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:07.080905+0000","last_peered":"2024-03-16T19:32:07.080905+0000","last_clean":"2024-03-16T19:32:07.080905+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:07.080905+0000","last_undegraded":"2024-03-16T19:32:07.080905+0000","last_fullsized":"2024-03-16T19:32:07.080905+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":361,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080930+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:07.080930+0000","last_peered":"2024-03-16T19:32:07.080930+0000","last_clean":"2024-03-16T19:32:07.080930+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:07.080930+0000","last_undegraded":"2024-03-16T19:32:07.080930+0000","last_fullsized":"2024-03-16T19:32:07.080930+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087541+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:08.087541+0000","last_peered":"2024-03-16T19:32:08.087541+0000","last_clean":"2024-03-16T19:32:08.087541+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:08.087541+0000","last_undegraded":"2024-03-16T19:32:08.087541+0000","last_fullsized":"2024-03-16T19:32:08.087541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:16:54.768710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080738+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:07.080738+0000","last_peered":"2024-03-16T19:32:07.080738+0000","last_clean":"2024-03-16T19:32:07.080738+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:07.080738+0000","last_undegraded":"2024-03-16T19:32:07.080738+0000","last_fullsized":"2024-03-16T19:32:07.080738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077009+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:07.077009+0000","last_peered":"2024-03-16T19:32:07.077009+0000","last_clean":"2024-03-16T19:32:07.077009+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:07.077009+0000","last_undegraded":"2024-03-16T19:32:07.077009+0000","last_fullsized":"2024-03-16T19:32:07.077009+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:44.308565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":50,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079702+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:07.079702+0000","last_peered":"2024-03-16T19:32:07.079702+0000","last_clean":"2024-03-16T19:32:07.079702+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:07.079702+0000","last_undegraded":"2024-03-16T19:32:07.079702+0000","last_fullsized":"2024-03-16T19:32:07.079702+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:20:21.014268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":182,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294258+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:05.294258+0000","last_peered":"2024-03-16T19:32:05.294258+0000","last_clean":"2024-03-16T19:32:05.294258+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:05.294258+0000","last_undegraded":"2024-03-16T19:32:05.294258+0000","last_fullsized":"2024-03-16T19:32:05.294258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084314+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:08.084314+0000","last_peered":"2024-03-16T19:32:08.084314+0000","last_clean":"2024-03-16T19:32:08.084314+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:08.084314+0000","last_undegraded":"2024-03-16T19:32:08.084314+0000","last_fullsized":"2024-03-16T19:32:08.084314+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:01.856961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":187,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294385+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:05.294385+0000","last_peered":"2024-03-16T19:32:05.294385+0000","last_clean":"2024-03-16T19:32:05.294385+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:05.294385+0000","last_undegraded":"2024-03-16T19:32:05.294385+0000","last_fullsized":"2024-03-16T19:32:05.294385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084193+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:08.084193+0000","last_peered":"2024-03-16T19:32:08.084193+0000","last_clean":"2024-03-16T19:32:08.084193+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:08.084193+0000","last_undegraded":"2024-03-16T19:32:08.084193+0000","last_fullsized":"2024-03-16T19:32:08.084193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081176+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:07.081176+0000","last_peered":"2024-03-16T19:32:07.081176+0000","last_clean":"2024-03-16T19:32:07.081176+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:07.081176+0000","last_undegraded":"2024-03-16T19:32:07.081176+0000","last_fullsized":"2024-03-16T19:32:07.081176+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":182,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294339+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:05.294339+0000","last_peered":"2024-03-16T19:32:05.294339+0000","last_clean":"2024-03-16T19:32:05.294339+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:05.294339+0000","last_undegraded":"2024-03-16T19:32:05.294339+0000","last_fullsized":"2024-03-16T19:32:05.294339+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'11","reported_seq":188,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294977+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:05.294977+0000","last_peered":"2024-03-16T19:32:05.294977+0000","last_clean":"2024-03-16T19:32:05.294977+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:05.294977+0000","last_undegraded":"2024-03-16T19:32:05.294977+0000","last_fullsized":"2024-03-16T19:32:05.294977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088741+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:08.088741+0000","last_peered":"2024-03-16T19:32:08.088741+0000","last_clean":"2024-03-16T19:32:08.088741+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:08.088741+0000","last_undegraded":"2024-03-16T19:32:08.088741+0000","last_fullsized":"2024-03-16T19:32:08.088741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:18.996236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620312+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:07.620312+0000","last_peered":"2024-03-16T19:32:07.620312+0000","last_clean":"2024-03-16T19:32:07.620312+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:07.620312+0000","last_undegraded":"2024-03-16T19:32:07.620312+0000","last_fullsized":"2024-03-16T19:32:07.620312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077386+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:07.077386+0000","last_peered":"2024-03-16T19:32:07.077386+0000","last_clean":"2024-03-16T19:32:07.077386+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:07.077386+0000","last_undegraded":"2024-03-16T19:32:07.077386+0000","last_fullsized":"2024-03-16T19:32:07.077386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:02.024100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080734+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:07.080734+0000","last_peered":"2024-03-16T19:32:07.080734+0000","last_clean":"2024-03-16T19:32:07.080734+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:07.080734+0000","last_undegraded":"2024-03-16T19:32:07.080734+0000","last_fullsized":"2024-03-16T19:32:07.080734+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083795+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:08.083795+0000","last_peered":"2024-03-16T19:32:08.083795+0000","last_clean":"2024-03-16T19:32:08.083795+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:08.083795+0000","last_undegraded":"2024-03-16T19:32:08.083795+0000","last_fullsized":"2024-03-16T19:32:08.083795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:46:49.628764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":177,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294344+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:05.294344+0000","last_peered":"2024-03-16T19:32:05.294344+0000","last_clean":"2024-03-16T19:32:05.294344+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:05.294344+0000","last_undegraded":"2024-03-16T19:32:05.294344+0000","last_fullsized":"2024-03-16T19:32:05.294344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":64,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.181279+0000","last_change":"2024-03-16T19:31:26.809671+0000","last_active":"2024-03-16T19:32:05.181279+0000","last_peered":"2024-03-16T19:32:05.181279+0000","last_clean":"2024-03-16T19:32:05.181279+0000","last_became_active":"2024-03-16T19:31:26.809344+0000","last_became_peered":"2024-03-16T19:31:26.809344+0000","last_unstale":"2024-03-16T19:32:05.181279+0000","last_undegraded":"2024-03-16T19:32:05.181279+0000","last_fullsized":"2024-03-16T19:32:05.181279+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:49.907553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":187,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.294590+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:05.294590+0000","last_peered":"2024-03-16T19:32:05.294590+0000","last_clean":"2024-03-16T19:32:05.294590+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:05.294590+0000","last_undegraded":"2024-03-16T19:32:05.294590+0000","last_fullsized":"2024-03-16T19:32:05.294590+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089208+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:08.089208+0000","last_peered":"2024-03-16T19:32:08.089208+0000","last_clean":"2024-03-16T19:32:08.089208+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:08.089208+0000","last_undegraded":"2024-03-16T19:32:08.089208+0000","last_fullsized":"2024-03-16T19:32:08.089208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:03.618618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080995+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:07.080995+0000","last_peered":"2024-03-16T19:32:07.080995+0000","last_clean":"2024-03-16T19:32:07.080995+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:07.080995+0000","last_undegraded":"2024-03-16T19:32:07.080995+0000","last_fullsized":"2024-03-16T19:32:07.080995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620022+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:07.620022+0000","last_peered":"2024-03-16T19:32:07.620022+0000","last_clean":"2024-03-16T19:32:07.620022+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:07.620022+0000","last_undegraded":"2024-03-16T19:32:07.620022+0000","last_fullsized":"2024-03-16T19:32:07.620022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084106+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:08.084106+0000","last_peered":"2024-03-16T19:32:08.084106+0000","last_clean":"2024-03-16T19:32:08.084106+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:08.084106+0000","last_undegraded":"2024-03-16T19:32:08.084106+0000","last_fullsized":"2024-03-16T19:32:08.084106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081285+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:07.081285+0000","last_peered":"2024-03-16T19:32:07.081285+0000","last_clean":"2024-03-16T19:32:07.081285+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:07.081285+0000","last_undegraded":"2024-03-16T19:32:07.081285+0000","last_fullsized":"2024-03-16T19:32:07.081285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083913+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:08.083913+0000","last_peered":"2024-03-16T19:32:08.083913+0000","last_clean":"2024-03-16T19:32:08.083913+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:08.083913+0000","last_undegraded":"2024-03-16T19:32:08.083913+0000","last_fullsized":"2024-03-16T19:32:08.083913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081056+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:07.081056+0000","last_peered":"2024-03-16T19:32:07.081056+0000","last_clean":"2024-03-16T19:32:07.081056+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:07.081056+0000","last_undegraded":"2024-03-16T19:32:07.081056+0000","last_fullsized":"2024-03-16T19:32:07.081056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":162,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077568+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:07.077568+0000","last_peered":"2024-03-16T19:32:07.077568+0000","last_clean":"2024-03-16T19:32:07.077568+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:07.077568+0000","last_undegraded":"2024-03-16T19:32:07.077568+0000","last_fullsized":"2024-03-16T19:32:07.077568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:19.317523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088482+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088482+0000","last_peered":"2024-03-16T19:32:08.088482+0000","last_clean":"2024-03-16T19:32:08.088482+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088482+0000","last_undegraded":"2024-03-16T19:32:08.088482+0000","last_fullsized":"2024-03-16T19:32:08.088482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:58:01.695714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086003+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:08.086003+0000","last_peered":"2024-03-16T19:32:08.086003+0000","last_clean":"2024-03-16T19:32:08.086003+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:08.086003+0000","last_undegraded":"2024-03-16T19:32:08.086003+0000","last_fullsized":"2024-03-16T19:32:08.086003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:17.040490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077402+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:07.077402+0000","last_peered":"2024-03-16T19:32:07.077402+0000","last_clean":"2024-03-16T19:32:07.077402+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:07.077402+0000","last_undegraded":"2024-03-16T19:32:07.077402+0000","last_fullsized":"2024-03-16T19:32:07.077402+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:28:10.116570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40494,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":80988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28824,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343016,"statfs":{"total":96636764160,"available":96607248384,"internally_reserved":0,"allocated":1847296,"data_stored":1180033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343552,"statfs":{"total":96636764160,"available":96607797248,"internally_reserved":0,"allocated":1298432,"data_stored":628375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1253376,"data_stored":589173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1290240,"data_stored":622664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122054,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29384,"kb_used_data":1212,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342456,"statfs":{"total":96636764160,"available":96606674944,"internally_reserved":0,"allocated":1241088,"data_stored":583372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28240,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343600,"statfs":{"total":96636764160,"available":96607846400,"internally_reserved":0,"allocated":1245184,"data_stored":583128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28832,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343008,"statfs":{"total":96636764160,"available":96607240192,"internally_reserved":0,"allocated":1855488,"data_stored":1180089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:09.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-16T19:32:09.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.593+0000 7f55c43b5640 1 Processor -- start 2024-03-16T19:32:09.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.593+0000 7f55c43b5640 1 -- start start 2024-03-16T19:32:09.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c43b5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc071e70 0x7f55bc158c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c43b5640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55bc0c8130 con 0x7f55bc162b20 2024-03-16T19:32:09.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c43b5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55bc15f150 con 0x7f55bc0c7c60 2024-03-16T19:32:09.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c43b5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc15f430 con 0x7f55bc071e70 2024-03-16T19:32:09.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c2151640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc071e70 0x7f55bc158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c2151640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc071e70 0x7f55bc158c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:47794/0 (socket says 172.21.15.172:47794) 2024-03-16T19:32:09.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c2151640 1 -- 172.21.15.172:0/1317373878 learned_addr learned my addr 172.21.15.172:0/1317373878 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:09.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c2151640 1 -- 172.21.15.172:0/1317373878 >> v1:172.21.15.174:6789/0 conn(0x7f55bc0c7c60 legacy=0x7f55bc162410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c114f640 1 -- 172.21.15.172:0/1317373878 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393566258 0 0) 0x7f55bc15f150 con 0x7f55bc0c7c60 2024-03-16T19:32:09.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.595+0000 7f55c2151640 1 -- 172.21.15.172:0/1317373878 >> v1:172.21.15.172:6789/0 conn(0x7f55bc162b20 legacy=0x7f55bc1697a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.596+0000 7f55c2151640 1 -- 172.21.15.172:0/1317373878 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc15f710 con 0x7f55bc071e70 2024-03-16T19:32:09.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.597+0000 7f55c2151640 1 --2- 172.21.15.172:0/1317373878 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc071e70 0x7f55bc158c70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f55b80098d0 tx=0x7f55b802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85481d6d6de9de69 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:09.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.597+0000 7f55c114f640 1 -- 172.21.15.172:0/1317373878 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b8031520 con 0x7f55bc071e70 2024-03-16T19:32:09.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.597+0000 7f55c114f640 1 -- 172.21.15.172:0/1317373878 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b8031680 con 0x7f55bc071e70 2024-03-16T19:32:09.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.597+0000 7f55c114f640 1 -- 172.21.15.172:0/1317373878 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b8031a10 con 0x7f55bc071e70 2024-03-16T19:32:09.579 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.598+0000 7f55c43b5640 1 -- 172.21.15.172:0/1317373878 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc071e70 msgr2=0x7f55bc158c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.579 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.598+0000 7f55c43b5640 1 --2- 172.21.15.172:0/1317373878 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc071e70 0x7f55bc158c70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f55b80098d0 tx=0x7f55b802b840 comp rx=0 tx=0).stop 2024-03-16T19:32:09.579 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.598+0000 7f55c43b5640 1 -- 172.21.15.172:0/1317373878 shutdown_connections 2024-03-16T19:32:09.579 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.598+0000 7f55c43b5640 1 --2- 172.21.15.172:0/1317373878 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc071e70 0x7f55bc158c70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.579 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.598+0000 7f55c43b5640 1 -- 172.21.15.172:0/1317373878 >> 172.21.15.172:0/1317373878 conn(0x7f55bc06e8d0 msgr2=0x7f55bc06ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:09.579 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.599+0000 7f55c43b5640 1 -- 172.21.15.172:0/1317373878 shutdown_connections 2024-03-16T19:32:09.580 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.599+0000 7f55c43b5640 1 -- 172.21.15.172:0/1317373878 wait complete. 2024-03-16T19:32:09.581 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.601+0000 7f55c43b5640 1 Processor -- start 2024-03-16T19:32:09.581 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.601+0000 7f55c43b5640 1 -- start start 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.602+0000 7f55c43b5640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55bc071e70 0x7f55bc0cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.602+0000 7f55c43b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55bc0c7c60 0x7f55bc0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.602+0000 7f55c2151640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55bc071e70 0x7f55bc0cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.603+0000 7f55c2151640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55bc071e70 0x7f55bc0cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57624/0 (socket says 172.21.15.172:57624) 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.603+0000 7f55c43b5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc14e4e0 0x7f55bc0cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.603+0000 7f55c1950640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55bc0c7c60 0x7f55bc0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.603+0000 7f55c1950640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55bc0c7c60 0x7f55bc0cb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:48476/0 (socket says 172.21.15.172:48476) 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.603+0000 7f55c43b5640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc1f8cc0 con 0x7f55bc071e70 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.603+0000 7f55c2151640 1 -- 172.21.15.172:0/4206997529 learned_addr learned my addr 172.21.15.172:0/4206997529 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:09.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.603+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc1f4800 con 0x7f55bc0c7c60 2024-03-16T19:32:09.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.603+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc1f5620 con 0x7f55bc14e4e0 2024-03-16T19:32:09.585 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.603+0000 7f55c2952640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc14e4e0 0x7f55bc0cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55c1950640 1 -- 172.21.15.172:0/4206997529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc14e4e0 msgr2=0x7f55bc0cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55c1950640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc14e4e0 0x7f55bc0cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55c1950640 1 -- 172.21.15.172:0/4206997529 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55bc071e70 msgr2=0x7f55bc0cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:09.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55c1950640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55bc071e70 0x7f55bc0cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:09.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55c1950640 1 -- 172.21.15.172:0/4206997529 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b80094d0 con 0x7f55bc0c7c60 2024-03-16T19:32:09.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55c2952640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc14e4e0 0x7f55bc0cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:09.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55c1950640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55bc0c7c60 0x7f55bc0cb210 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00c400 tx=0x7f55ac00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:09.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55b2ffd640 1 -- 172.21.15.172:0/4206997529 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0035f0 con 0x7f55bc0c7c60 2024-03-16T19:32:09.586 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55b2ffd640 1 -- 172.21.15.172:0/4206997529 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac003770 con 0x7f55bc0c7c60 2024-03-16T19:32:09.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55b2ffd640 1 -- 172.21.15.172:0/4206997529 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac003b20 con 0x7f55bc0c7c60 2024-03-16T19:32:09.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.604+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc1f4a10 con 0x7f55bc0c7c60 2024-03-16T19:32:09.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.605+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc1f4fd0 con 0x7f55bc0c7c60 2024-03-16T19:32:09.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.606+0000 7f55b2ffd640 1 -- 172.21.15.172:0/4206997529 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f55ac003c80 con 0x7f55bc0c7c60 2024-03-16T19:32:09.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.606+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55bc0c8450 con 0x7f55bc0c7c60 2024-03-16T19:32:09.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.606+0000 7f55b2ffd640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f55a006b3e0 0x7f55a006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:09.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.607+0000 7f55b2ffd640 1 -- 172.21.15.172:0/4206997529 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f55ac089160 con 0x7f55bc0c7c60 2024-03-16T19:32:09.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.607+0000 7f55c2151640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f55a006b3e0 0x7f55a006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:09.588 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.607+0000 7f55c2151640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f55a006b3e0 0x7f55a006d8a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f55b8000c00 tx=0x7f55b8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:09.611+0000 7f55b2ffd640 1 -- 172.21.15.172:0/4206997529 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55ac0578e0 con 0x7f55bc0c7c60 2024-03-16T19:32:10.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.051+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f55bc0688d0 con 0x7f55bc0c7c60 2024-03-16T19:32:10.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.052+0000 7f55b2ffd640 1 -- 172.21.15.172:0/4206997529 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1026 (secure 0 0 0) 0x7f55ac057470 con 0x7f55bc0c7c60 2024-03-16T19:32:10.033 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:10.033 INFO:teuthology.orchestra.run.smithi172.stdout:{"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":188,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":106,"num_osds":8,"num_up_osds":8,"osd_up_since":1710617508,"num_in_osds":8,"osd_in_since":1710617508,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":96},{"state_name":"peering","count":1}],"num_pgs":97,"num_pools":3,"num_objects":25,"data_bytes":630862,"bytes_used":233840640,"bytes_avail":772860272640,"bytes_total":773094113280,"inactive_pgs_ratio":0.010309278033673763},"fsmap":{"epoch":8,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"f","status":"up:active","gid":4297}],"up:standby":5},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-16T19:30:36.213107+0000","services":{}},"progress_events":{}} 2024-03-16T19:32:10.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f55a006b3e0 msgr2=0x7f55a006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:10.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f55a006b3e0 0x7f55a006d8a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f55b8000c00 tx=0x7f55b8030000 comp rx=0 tx=0).stop 2024-03-16T19:32:10.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55bc0c7c60 msgr2=0x7f55bc0cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:10.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55bc0c7c60 0x7f55bc0cb210 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00c400 tx=0x7f55ac00c8c0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 shutdown_connections 2024-03-16T19:32:10.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f55a006b3e0 0x7f55a006d8a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55bc14e4e0 0x7f55bc0cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55bc0c7c60 0x7f55bc0cb210 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 --2- 172.21.15.172:0/4206997529 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55bc071e70 0x7f55bc0cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 >> 172.21.15.172:0/4206997529 conn(0x7f55bc06e8d0 msgr2=0x7f55bc0ca290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:10.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 shutdown_connections 2024-03-16T19:32:10.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.054+0000 7f55c43b5640 1 -- 172.21.15.172:0/4206997529 wait complete. 2024-03-16T19:32:10.049 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-16T19:32:10.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:10.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.218+0000 7f1688b5e640 1 Processor -- start 2024-03-16T19:32:10.199 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.218+0000 7f1688b5e640 1 -- start start 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f1688b5e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1684071e70 0x7f1684157f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f1688b5e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16840c4760 0x7f1684151470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f1688b5e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16840c4c00 con 0x7f1684072250 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f1688b5e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16840c5660 con 0x7f1684071e70 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f1688b5e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1684157b00 con 0x7f16840c4760 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f1681d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16840c4760 0x7f1684151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f1682d9d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f1684072250 0x7f16840cd1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:49318/0 (socket says 172.21.15.172:49318) 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f1682d9d640 1 -- 172.21.15.172:0/387528544 learned_addr learned my addr 172.21.15.172:0/387528544 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f168259c640 1 --2- 172.21.15.172:0/387528544 >> v2:172.21.15.174:3300/0 conn(0x7f1684071e70 0x7f1684157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.219+0000 7f1681d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16840c4760 0x7f1684151470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:47820/0 (socket says 172.21.15.172:47820) 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.220+0000 7f168159a640 1 -- 172.21.15.172:0/387528544 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611486631 0 0) 0x7f16840c4c00 con 0x7f1684072250 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.220+0000 7f168259c640 1 -- 172.21.15.172:0/387528544 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16840c4760 msgr2=0x7f1684151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.220+0000 7f168259c640 1 --2- 172.21.15.172:0/387528544 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16840c4760 0x7f1684151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.220+0000 7f168259c640 1 -- 172.21.15.172:0/387528544 >> v1:172.21.15.174:6789/0 conn(0x7f1684072250 legacy=0x7f16840cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:10.202 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.220+0000 7f168259c640 1 -- 172.21.15.172:0/387528544 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168406c800 con 0x7f1684071e70 2024-03-16T19:32:10.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.221+0000 7f168259c640 1 --2- 172.21.15.172:0/387528544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 0x7f1684157f90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f16740097d0 tx=0x7f167402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21b42cb1f0f465ad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:10.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.221+0000 7f168159a640 1 -- 172.21.15.172:0/387528544 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674030800 con 0x7f1684071e70 2024-03-16T19:32:10.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.221+0000 7f168159a640 1 -- 172.21.15.172:0/387528544 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1674030960 con 0x7f1684071e70 2024-03-16T19:32:10.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.221+0000 7f168159a640 1 -- 172.21.15.172:0/387528544 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674030800 con 0x7f1684071e70 2024-03-16T19:32:10.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.222+0000 7f1688b5e640 1 -- 172.21.15.172:0/387528544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 msgr2=0x7f1684157f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:10.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.222+0000 7f1688b5e640 1 --2- 172.21.15.172:0/387528544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 0x7f1684157f90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f16740097d0 tx=0x7f167402b940 comp rx=0 tx=0).stop 2024-03-16T19:32:10.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.222+0000 7f1688b5e640 1 -- 172.21.15.172:0/387528544 shutdown_connections 2024-03-16T19:32:10.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.222+0000 7f1688b5e640 1 --2- 172.21.15.172:0/387528544 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16840c4760 0x7f1684151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.222+0000 7f1688b5e640 1 --2- 172.21.15.172:0/387528544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 0x7f1684157f90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.222+0000 7f1688b5e640 1 -- 172.21.15.172:0/387528544 >> 172.21.15.172:0/387528544 conn(0x7f168406e8d0 msgr2=0x7f168406ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:10.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.222+0000 7f1688b5e640 1 -- 172.21.15.172:0/387528544 shutdown_connections 2024-03-16T19:32:10.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.222+0000 7f1688b5e640 1 -- 172.21.15.172:0/387528544 wait complete. 2024-03-16T19:32:10.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.224+0000 7f1688b5e640 1 Processor -- start 2024-03-16T19:32:10.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.224+0000 7f1688b5e640 1 -- start start 2024-03-16T19:32:10.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.224+0000 7f1688b5e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 0x7f16841f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:10.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.224+0000 7f1688b5e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1684072250 0x7f16840c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:10.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.224+0000 7f1688b5e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16840c4760 0x7f1684146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.224+0000 7f1688b5e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16841f4340 con 0x7f16840c4760 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.224+0000 7f1688b5e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168414f5e0 con 0x7f1684071e70 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.224+0000 7f1688b5e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168406b120 con 0x7f1684072250 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f168259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 0x7f16841f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1681d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1684072250 0x7f16840c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f168259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 0x7f16841f82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:48494/0 (socket says 172.21.15.172:48494) 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f168259c640 1 -- 172.21.15.172:0/585131387 learned_addr learned my addr 172.21.15.172:0/585131387 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1682d9d640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16840c4760 0x7f1684146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1681d9b640 1 -- 172.21.15.172:0/585131387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 msgr2=0x7f16841f82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1681d9b640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 0x7f16841f82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1681d9b640 1 -- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16840c4760 msgr2=0x7f1684146d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1681d9b640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16840c4760 0x7f1684146d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1681d9b640 1 -- 172.21.15.172:0/585131387 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1674009400 con 0x7f1684072250 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1681d9b640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1684072250 0x7f16840c7b60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f167800e870 tx=0x7f167800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:10.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1672ffd640 1 -- 172.21.15.172:0/585131387 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167800ee60 con 0x7f1684072250 2024-03-16T19:32:10.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.225+0000 7f1672ffd640 1 -- 172.21.15.172:0/585131387 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167800c940 con 0x7f1684072250 2024-03-16T19:32:10.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.226+0000 7f1672ffd640 1 -- 172.21.15.172:0/585131387 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167800ccd0 con 0x7f1684072250 2024-03-16T19:32:10.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.226+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16841f8800 con 0x7f1684072250 2024-03-16T19:32:10.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.226+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16840c80a0 con 0x7f1684072250 2024-03-16T19:32:10.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.227+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16840688d0 con 0x7f1684072250 2024-03-16T19:32:10.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.227+0000 7f1672ffd640 1 -- 172.21.15.172:0/585131387 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1678013470 con 0x7f1684072250 2024-03-16T19:32:10.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.228+0000 7f1672ffd640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f166406b400 0x7f166406d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:10.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.228+0000 7f168259c640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f166406b400 0x7f166406d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:10.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.228+0000 7f1672ffd640 1 -- 172.21.15.172:0/585131387 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f1678089950 con 0x7f1684072250 2024-03-16T19:32:10.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.228+0000 7f168259c640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f166406b400 0x7f166406d8c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1674000c00 tx=0x7f1674008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:10.212 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.232+0000 7f1672ffd640 1 -- 172.21.15.172:0/585131387 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16780546a0 con 0x7f1684072250 2024-03-16T19:32:10.526 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.545+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f168406cda0 con 0x7f166406b400 2024-03-16T19:32:10.535 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.554+0000 7f1672ffd640 1 -- 172.21.15.172:0/585131387 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252318 (secure 0 0 0) 0x7f167403f040 con 0x7f166406b400 2024-03-16T19:32:10.538 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:10.539 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:10.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f166406b400 msgr2=0x7f166406d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:10.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f166406b400 0x7f166406d8c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1674000c00 tx=0x7f1674008030 comp rx=0 tx=0).stop 2024-03-16T19:32:10.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1684072250 msgr2=0x7f16840c7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:10.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1684072250 0x7f16840c7b60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f167800e870 tx=0x7f167800ed30 comp rx=0 tx=0).stop 2024-03-16T19:32:10.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 shutdown_connections 2024-03-16T19:32:10.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16840c4760 0x7f1684146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f166406b400 0x7f166406d8c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.541 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1684072250 0x7f16840c7b60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 --2- 172.21.15.172:0/585131387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1684071e70 0x7f16841f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:10.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 >> 172.21.15.172:0/585131387 conn(0x7f168406e8d0 msgr2=0x7f1684074880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:10.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 shutdown_connections 2024-03-16T19:32:10.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:10.558+0000 7f1688b5e640 1 -- 172.21.15.172:0/585131387 wait complete. 2024-03-16T19:32:10.553 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-03-16T19:32:10.231420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630862,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":245,"num_write_kb":2489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228360,"kb_used_data":11040,"kb_used_omap":16,"kb_used_meta":217199,"kb_avail":754746360,"statfs":{"total":773094113280,"available":772860272640,"internally_reserved":0,"allocated":11304960,"data_stored":5962108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":222412203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001367"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081164+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:07.081164+0000","last_peered":"2024-03-16T19:32:07.081164+0000","last_clean":"2024-03-16T19:32:07.081164+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:07.081164+0000","last_undegraded":"2024-03-16T19:32:07.081164+0000","last_fullsized":"2024-03-16T19:32:07.081164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094518+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:08.094518+0000","last_peered":"2024-03-16T19:32:08.094518+0000","last_clean":"2024-03-16T19:32:08.094518+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:08.094518+0000","last_undegraded":"2024-03-16T19:32:08.094518+0000","last_fullsized":"2024-03-16T19:32:08.094518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077201+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:07.077201+0000","last_peered":"2024-03-16T19:32:07.077201+0000","last_clean":"2024-03-16T19:32:07.077201+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:07.077201+0000","last_undegraded":"2024-03-16T19:32:07.077201+0000","last_fullsized":"2024-03-16T19:32:07.077201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:47.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083818+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:08.083818+0000","last_peered":"2024-03-16T19:32:08.083818+0000","last_clean":"2024-03-16T19:32:08.083818+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:08.083818+0000","last_undegraded":"2024-03-16T19:32:08.083818+0000","last_fullsized":"2024-03-16T19:32:08.083818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080899+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:07.080899+0000","last_peered":"2024-03-16T19:32:07.080899+0000","last_clean":"2024-03-16T19:32:07.080899+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:07.080899+0000","last_undegraded":"2024-03-16T19:32:07.080899+0000","last_fullsized":"2024-03-16T19:32:07.080899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083640+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:08.083640+0000","last_peered":"2024-03-16T19:32:08.083640+0000","last_clean":"2024-03-16T19:32:08.083640+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:08.083640+0000","last_undegraded":"2024-03-16T19:32:08.083640+0000","last_fullsized":"2024-03-16T19:32:08.083640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":187,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080786+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:07.080786+0000","last_peered":"2024-03-16T19:32:07.080786+0000","last_clean":"2024-03-16T19:32:07.080786+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:07.080786+0000","last_undegraded":"2024-03-16T19:32:07.080786+0000","last_fullsized":"2024-03-16T19:32:07.080786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086880+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:08.086880+0000","last_peered":"2024-03-16T19:32:08.086880+0000","last_clean":"2024-03-16T19:32:08.086880+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:08.086880+0000","last_undegraded":"2024-03-16T19:32:08.086880+0000","last_fullsized":"2024-03-16T19:32:08.086880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:58:31.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081177+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:07.081177+0000","last_peered":"2024-03-16T19:32:07.081177+0000","last_clean":"2024-03-16T19:32:07.081177+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:07.081177+0000","last_undegraded":"2024-03-16T19:32:07.081177+0000","last_fullsized":"2024-03-16T19:32:07.081177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620209+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:07.620209+0000","last_peered":"2024-03-16T19:32:07.620209+0000","last_clean":"2024-03-16T19:32:07.620209+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:07.620209+0000","last_undegraded":"2024-03-16T19:32:07.620209+0000","last_fullsized":"2024-03-16T19:32:07.620209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620126+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:07.620126+0000","last_peered":"2024-03-16T19:32:07.620126+0000","last_clean":"2024-03-16T19:32:07.620126+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:07.620126+0000","last_undegraded":"2024-03-16T19:32:07.620126+0000","last_fullsized":"2024-03-16T19:32:07.620126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084143+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:08.084143+0000","last_peered":"2024-03-16T19:32:08.084143+0000","last_clean":"2024-03-16T19:32:08.084143+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:08.084143+0000","last_undegraded":"2024-03-16T19:32:08.084143+0000","last_fullsized":"2024-03-16T19:32:08.084143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088886+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:08.088886+0000","last_peered":"2024-03-16T19:32:08.088886+0000","last_clean":"2024-03-16T19:32:08.088886+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:08.088886+0000","last_undegraded":"2024-03-16T19:32:08.088886+0000","last_fullsized":"2024-03-16T19:32:08.088886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:52.102388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"84'3","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077055+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:07.077055+0000","last_peered":"2024-03-16T19:32:07.077055+0000","last_clean":"2024-03-16T19:32:07.077055+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:07.077055+0000","last_undegraded":"2024-03-16T19:32:07.077055+0000","last_fullsized":"2024-03-16T19:32:07.077055+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:39.289238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094126+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:08.094126+0000","last_peered":"2024-03-16T19:32:08.094126+0000","last_clean":"2024-03-16T19:32:08.094126+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:08.094126+0000","last_undegraded":"2024-03-16T19:32:08.094126+0000","last_fullsized":"2024-03-16T19:32:08.094126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083403+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:08.083403+0000","last_peered":"2024-03-16T19:32:08.083403+0000","last_clean":"2024-03-16T19:32:08.083403+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:08.083403+0000","last_undegraded":"2024-03-16T19:32:08.083403+0000","last_fullsized":"2024-03-16T19:32:08.083403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084254+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:08.084254+0000","last_peered":"2024-03-16T19:32:08.084254+0000","last_clean":"2024-03-16T19:32:08.084254+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:08.084254+0000","last_undegraded":"2024-03-16T19:32:08.084254+0000","last_fullsized":"2024-03-16T19:32:08.084254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080010+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:07.080010+0000","last_peered":"2024-03-16T19:32:07.080010+0000","last_clean":"2024-03-16T19:32:07.080010+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:07.080010+0000","last_undegraded":"2024-03-16T19:32:07.080010+0000","last_fullsized":"2024-03-16T19:32:07.080010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:36:06.175327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081155+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:07.081155+0000","last_peered":"2024-03-16T19:32:07.081155+0000","last_clean":"2024-03-16T19:32:07.081155+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:07.081155+0000","last_undegraded":"2024-03-16T19:32:07.081155+0000","last_fullsized":"2024-03-16T19:32:07.081155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620154+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:07.620154+0000","last_peered":"2024-03-16T19:32:07.620154+0000","last_clean":"2024-03-16T19:32:07.620154+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:07.620154+0000","last_undegraded":"2024-03-16T19:32:07.620154+0000","last_fullsized":"2024-03-16T19:32:07.620154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077197+0000","last_change":"2024-03-16T19:31:51.807295+0000","last_active":"2024-03-16T19:32:07.077197+0000","last_peered":"2024-03-16T19:32:07.077197+0000","last_clean":"2024-03-16T19:32:07.077197+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:07.077197+0000","last_undegraded":"2024-03-16T19:32:07.077197+0000","last_fullsized":"2024-03-16T19:32:07.077197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:04.578640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":50,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079813+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:07.079813+0000","last_peered":"2024-03-16T19:32:07.079813+0000","last_clean":"2024-03-16T19:32:07.079813+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:07.079813+0000","last_undegraded":"2024-03-16T19:32:07.079813+0000","last_fullsized":"2024-03-16T19:32:07.079813+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:41:06.377004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088245+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:08.088245+0000","last_peered":"2024-03-16T19:32:08.088245+0000","last_clean":"2024-03-16T19:32:08.088245+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:08.088245+0000","last_undegraded":"2024-03-16T19:32:08.088245+0000","last_fullsized":"2024-03-16T19:32:08.088245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:47:08.251341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083454+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:08.083454+0000","last_peered":"2024-03-16T19:32:08.083454+0000","last_clean":"2024-03-16T19:32:08.083454+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:08.083454+0000","last_undegraded":"2024-03-16T19:32:08.083454+0000","last_fullsized":"2024-03-16T19:32:08.083454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086134+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:08.086134+0000","last_peered":"2024-03-16T19:32:08.086134+0000","last_clean":"2024-03-16T19:32:08.086134+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:08.086134+0000","last_undegraded":"2024-03-16T19:32:08.086134+0000","last_fullsized":"2024-03-16T19:32:08.086134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:10.818175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620238+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:07.620238+0000","last_peered":"2024-03-16T19:32:07.620238+0000","last_clean":"2024-03-16T19:32:07.620238+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:07.620238+0000","last_undegraded":"2024-03-16T19:32:07.620238+0000","last_fullsized":"2024-03-16T19:32:07.620238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":187,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093433+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:08.093433+0000","last_peered":"2024-03-16T19:32:08.093433+0000","last_clean":"2024-03-16T19:32:08.093433+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:08.093433+0000","last_undegraded":"2024-03-16T19:32:08.093433+0000","last_fullsized":"2024-03-16T19:32:08.093433+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087892+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:08.087892+0000","last_peered":"2024-03-16T19:32:08.087892+0000","last_clean":"2024-03-16T19:32:08.087892+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:08.087892+0000","last_undegraded":"2024-03-16T19:32:08.087892+0000","last_fullsized":"2024-03-16T19:32:08.087892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:12:08.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077141+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:07.077141+0000","last_peered":"2024-03-16T19:32:07.077141+0000","last_clean":"2024-03-16T19:32:07.077141+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:07.077141+0000","last_undegraded":"2024-03-16T19:32:07.077141+0000","last_fullsized":"2024-03-16T19:32:07.077141+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:12.646493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083749+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:08.083749+0000","last_peered":"2024-03-16T19:32:08.083749+0000","last_clean":"2024-03-16T19:32:08.083749+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:08.083749+0000","last_undegraded":"2024-03-16T19:32:08.083749+0000","last_fullsized":"2024-03-16T19:32:08.083749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079774+0000","last_change":"2024-03-16T19:31:49.974297+0000","last_active":"2024-03-16T19:32:07.079774+0000","last_peered":"2024-03-16T19:32:07.079774+0000","last_clean":"2024-03-16T19:32:07.079774+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:07.079774+0000","last_undegraded":"2024-03-16T19:32:07.079774+0000","last_fullsized":"2024-03-16T19:32:07.079774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:51:57.535372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":186,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.619981+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:07.619981+0000","last_peered":"2024-03-16T19:32:07.619981+0000","last_clean":"2024-03-16T19:32:07.619981+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:07.619981+0000","last_undegraded":"2024-03-16T19:32:07.619981+0000","last_fullsized":"2024-03-16T19:32:07.619981+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084174+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:32:08.084174+0000","last_peered":"2024-03-16T19:32:08.084174+0000","last_clean":"2024-03-16T19:32:08.084174+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:08.084174+0000","last_undegraded":"2024-03-16T19:32:08.084174+0000","last_fullsized":"2024-03-16T19:32:08.084174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089011+0000","last_change":"2024-03-16T19:31:26.808032+0000","last_active":"2024-03-16T19:32:08.089011+0000","last_peered":"2024-03-16T19:32:08.089011+0000","last_clean":"2024-03-16T19:32:08.089011+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:08.089011+0000","last_undegraded":"2024-03-16T19:32:08.089011+0000","last_fullsized":"2024-03-16T19:32:08.089011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:15.218778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077339+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:07.077339+0000","last_peered":"2024-03-16T19:32:07.077339+0000","last_clean":"2024-03-16T19:32:07.077339+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:07.077339+0000","last_undegraded":"2024-03-16T19:32:07.077339+0000","last_fullsized":"2024-03-16T19:32:07.077339+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:16:31.606469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620053+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:07.620053+0000","last_peered":"2024-03-16T19:32:07.620053+0000","last_clean":"2024-03-16T19:32:07.620053+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:07.620053+0000","last_undegraded":"2024-03-16T19:32:07.620053+0000","last_fullsized":"2024-03-16T19:32:07.620053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086078+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086078+0000","last_peered":"2024-03-16T19:32:08.086078+0000","last_clean":"2024-03-16T19:32:08.086078+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086078+0000","last_undegraded":"2024-03-16T19:32:08.086078+0000","last_fullsized":"2024-03-16T19:32:08.086078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:23:19.824904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086998+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:08.086998+0000","last_peered":"2024-03-16T19:32:08.086998+0000","last_clean":"2024-03-16T19:32:08.086998+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:08.086998+0000","last_undegraded":"2024-03-16T19:32:08.086998+0000","last_fullsized":"2024-03-16T19:32:08.086998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:45.782085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094004+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:08.094004+0000","last_peered":"2024-03-16T19:32:08.094004+0000","last_clean":"2024-03-16T19:32:08.094004+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:08.094004+0000","last_undegraded":"2024-03-16T19:32:08.094004+0000","last_fullsized":"2024-03-16T19:32:08.094004+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620263+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:07.620263+0000","last_peered":"2024-03-16T19:32:07.620263+0000","last_clean":"2024-03-16T19:32:07.620263+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:07.620263+0000","last_undegraded":"2024-03-16T19:32:07.620263+0000","last_fullsized":"2024-03-16T19:32:07.620263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083754+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:08.083754+0000","last_peered":"2024-03-16T19:32:08.083754+0000","last_clean":"2024-03-16T19:32:08.083754+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:08.083754+0000","last_undegraded":"2024-03-16T19:32:08.083754+0000","last_fullsized":"2024-03-16T19:32:08.083754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077496+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:07.077496+0000","last_peered":"2024-03-16T19:32:07.077496+0000","last_clean":"2024-03-16T19:32:07.077496+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:07.077496+0000","last_undegraded":"2024-03-16T19:32:07.077496+0000","last_fullsized":"2024-03-16T19:32:07.077496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:16.418962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093565+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:08.093565+0000","last_peered":"2024-03-16T19:32:08.093565+0000","last_clean":"2024-03-16T19:32:08.093565+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:08.093565+0000","last_undegraded":"2024-03-16T19:32:08.093565+0000","last_fullsized":"2024-03-16T19:32:08.093565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"peering","last_fresh":"2024-03-16T19:32:05.059246+0000","last_change":"2024-03-16T19:32:07.978687+0000","last_active":"2024-03-16T19:32:07.978687+0000","last_peered":"2024-03-16T19:32:05.059246+0000","last_clean":"2024-03-16T19:32:05.059246+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:32:05.059246+0000","last_undegraded":"2024-03-16T19:32:05.059246+0000","last_fullsized":"2024-03-16T19:32:05.059246+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"86'23","reported_seq":231,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620164+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:07.620164+0000","last_peered":"2024-03-16T19:32:07.620164+0000","last_clean":"2024-03-16T19:32:07.620164+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:07.620164+0000","last_undegraded":"2024-03-16T19:32:07.620164+0000","last_fullsized":"2024-03-16T19:32:07.620164+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39292,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":187,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080997+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:07.080997+0000","last_peered":"2024-03-16T19:32:07.080997+0000","last_clean":"2024-03-16T19:32:07.080997+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:07.080997+0000","last_undegraded":"2024-03-16T19:32:07.080997+0000","last_fullsized":"2024-03-16T19:32:07.080997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077290+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:07.077290+0000","last_peered":"2024-03-16T19:32:07.077290+0000","last_clean":"2024-03-16T19:32:07.077290+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:07.077290+0000","last_undegraded":"2024-03-16T19:32:07.077290+0000","last_fullsized":"2024-03-16T19:32:07.077290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:59.818393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.076992+0000","last_change":"2024-03-16T19:31:51.807895+0000","last_active":"2024-03-16T19:32:07.076992+0000","last_peered":"2024-03-16T19:32:07.076992+0000","last_clean":"2024-03-16T19:32:07.076992+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:07.076992+0000","last_undegraded":"2024-03-16T19:32:07.076992+0000","last_fullsized":"2024-03-16T19:32:07.076992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:25.988237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080958+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:07.080958+0000","last_peered":"2024-03-16T19:32:07.080958+0000","last_clean":"2024-03-16T19:32:07.080958+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:07.080958+0000","last_undegraded":"2024-03-16T19:32:07.080958+0000","last_fullsized":"2024-03-16T19:32:07.080958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":178,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620021+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:07.620021+0000","last_peered":"2024-03-16T19:32:07.620021+0000","last_clean":"2024-03-16T19:32:07.620021+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:07.620021+0000","last_undegraded":"2024-03-16T19:32:07.620021+0000","last_fullsized":"2024-03-16T19:32:07.620021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":186,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.619987+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:07.619987+0000","last_peered":"2024-03-16T19:32:07.619987+0000","last_clean":"2024-03-16T19:32:07.619987+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:07.619987+0000","last_undegraded":"2024-03-16T19:32:07.619987+0000","last_fullsized":"2024-03-16T19:32:07.619987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":56,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079696+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:07.079696+0000","last_peered":"2024-03-16T19:32:07.079696+0000","last_clean":"2024-03-16T19:32:07.079696+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:07.079696+0000","last_undegraded":"2024-03-16T19:32:07.079696+0000","last_fullsized":"2024-03-16T19:32:07.079696+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:33.634146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080858+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:07.080858+0000","last_peered":"2024-03-16T19:32:07.080858+0000","last_clean":"2024-03-16T19:32:07.080858+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:07.080858+0000","last_undegraded":"2024-03-16T19:32:07.080858+0000","last_fullsized":"2024-03-16T19:32:07.080858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084319+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084319+0000","last_peered":"2024-03-16T19:32:08.084319+0000","last_clean":"2024-03-16T19:32:08.084319+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084319+0000","last_undegraded":"2024-03-16T19:32:08.084319+0000","last_fullsized":"2024-03-16T19:32:08.084319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:53.021797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081088+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:07.081088+0000","last_peered":"2024-03-16T19:32:07.081088+0000","last_clean":"2024-03-16T19:32:07.081088+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:07.081088+0000","last_undegraded":"2024-03-16T19:32:07.081088+0000","last_fullsized":"2024-03-16T19:32:07.081088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":53,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079883+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:07.079883+0000","last_peered":"2024-03-16T19:32:07.079883+0000","last_clean":"2024-03-16T19:32:07.079883+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:07.079883+0000","last_undegraded":"2024-03-16T19:32:07.079883+0000","last_fullsized":"2024-03-16T19:32:07.079883+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:07:39.853290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081238+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:07.081238+0000","last_peered":"2024-03-16T19:32:07.081238+0000","last_clean":"2024-03-16T19:32:07.081238+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:07.081238+0000","last_undegraded":"2024-03-16T19:32:07.081238+0000","last_fullsized":"2024-03-16T19:32:07.081238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093703+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:08.093703+0000","last_peered":"2024-03-16T19:32:08.093703+0000","last_clean":"2024-03-16T19:32:08.093703+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:08.093703+0000","last_undegraded":"2024-03-16T19:32:08.093703+0000","last_fullsized":"2024-03-16T19:32:08.093703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084002+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:08.084002+0000","last_peered":"2024-03-16T19:32:08.084002+0000","last_clean":"2024-03-16T19:32:08.084002+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:08.084002+0000","last_undegraded":"2024-03-16T19:32:08.084002+0000","last_fullsized":"2024-03-16T19:32:08.084002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079907+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:07.079907+0000","last_peered":"2024-03-16T19:32:07.079907+0000","last_clean":"2024-03-16T19:32:07.079907+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:07.079907+0000","last_undegraded":"2024-03-16T19:32:07.079907+0000","last_fullsized":"2024-03-16T19:32:07.079907+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:59:55.019036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":146,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080905+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:07.080905+0000","last_peered":"2024-03-16T19:32:07.080905+0000","last_clean":"2024-03-16T19:32:07.080905+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:07.080905+0000","last_undegraded":"2024-03-16T19:32:07.080905+0000","last_fullsized":"2024-03-16T19:32:07.080905+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":361,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080930+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:07.080930+0000","last_peered":"2024-03-16T19:32:07.080930+0000","last_clean":"2024-03-16T19:32:07.080930+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:07.080930+0000","last_undegraded":"2024-03-16T19:32:07.080930+0000","last_fullsized":"2024-03-16T19:32:07.080930+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087541+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:08.087541+0000","last_peered":"2024-03-16T19:32:08.087541+0000","last_clean":"2024-03-16T19:32:08.087541+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:08.087541+0000","last_undegraded":"2024-03-16T19:32:08.087541+0000","last_fullsized":"2024-03-16T19:32:08.087541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:16:54.768710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080738+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:07.080738+0000","last_peered":"2024-03-16T19:32:07.080738+0000","last_clean":"2024-03-16T19:32:07.080738+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:07.080738+0000","last_undegraded":"2024-03-16T19:32:07.080738+0000","last_fullsized":"2024-03-16T19:32:07.080738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077009+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:07.077009+0000","last_peered":"2024-03-16T19:32:07.077009+0000","last_clean":"2024-03-16T19:32:07.077009+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:07.077009+0000","last_undegraded":"2024-03-16T19:32:07.077009+0000","last_fullsized":"2024-03-16T19:32:07.077009+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:44.308565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":50,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079702+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:07.079702+0000","last_peered":"2024-03-16T19:32:07.079702+0000","last_clean":"2024-03-16T19:32:07.079702+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:07.079702+0000","last_undegraded":"2024-03-16T19:32:07.079702+0000","last_fullsized":"2024-03-16T19:32:07.079702+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:20:21.014268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.096035+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:08.096035+0000","last_peered":"2024-03-16T19:32:08.096035+0000","last_clean":"2024-03-16T19:32:08.096035+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:08.096035+0000","last_undegraded":"2024-03-16T19:32:08.096035+0000","last_fullsized":"2024-03-16T19:32:08.096035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084314+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:08.084314+0000","last_peered":"2024-03-16T19:32:08.084314+0000","last_clean":"2024-03-16T19:32:08.084314+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:08.084314+0000","last_undegraded":"2024-03-16T19:32:08.084314+0000","last_fullsized":"2024-03-16T19:32:08.084314+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:01.856961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093299+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:08.093299+0000","last_peered":"2024-03-16T19:32:08.093299+0000","last_clean":"2024-03-16T19:32:08.093299+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:08.093299+0000","last_undegraded":"2024-03-16T19:32:08.093299+0000","last_fullsized":"2024-03-16T19:32:08.093299+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084193+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:08.084193+0000","last_peered":"2024-03-16T19:32:08.084193+0000","last_clean":"2024-03-16T19:32:08.084193+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:08.084193+0000","last_undegraded":"2024-03-16T19:32:08.084193+0000","last_fullsized":"2024-03-16T19:32:08.084193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081176+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:07.081176+0000","last_peered":"2024-03-16T19:32:07.081176+0000","last_clean":"2024-03-16T19:32:07.081176+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:07.081176+0000","last_undegraded":"2024-03-16T19:32:07.081176+0000","last_fullsized":"2024-03-16T19:32:07.081176+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093171+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:08.093171+0000","last_peered":"2024-03-16T19:32:08.093171+0000","last_clean":"2024-03-16T19:32:08.093171+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:08.093171+0000","last_undegraded":"2024-03-16T19:32:08.093171+0000","last_fullsized":"2024-03-16T19:32:08.093171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'11","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094654+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:08.094654+0000","last_peered":"2024-03-16T19:32:08.094654+0000","last_clean":"2024-03-16T19:32:08.094654+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:08.094654+0000","last_undegraded":"2024-03-16T19:32:08.094654+0000","last_fullsized":"2024-03-16T19:32:08.094654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088741+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:08.088741+0000","last_peered":"2024-03-16T19:32:08.088741+0000","last_clean":"2024-03-16T19:32:08.088741+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:08.088741+0000","last_undegraded":"2024-03-16T19:32:08.088741+0000","last_fullsized":"2024-03-16T19:32:08.088741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:18.996236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620312+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:07.620312+0000","last_peered":"2024-03-16T19:32:07.620312+0000","last_clean":"2024-03-16T19:32:07.620312+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:07.620312+0000","last_undegraded":"2024-03-16T19:32:07.620312+0000","last_fullsized":"2024-03-16T19:32:07.620312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077386+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:07.077386+0000","last_peered":"2024-03-16T19:32:07.077386+0000","last_clean":"2024-03-16T19:32:07.077386+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:07.077386+0000","last_undegraded":"2024-03-16T19:32:07.077386+0000","last_fullsized":"2024-03-16T19:32:07.077386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:02.024100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080734+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:07.080734+0000","last_peered":"2024-03-16T19:32:07.080734+0000","last_clean":"2024-03-16T19:32:07.080734+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:07.080734+0000","last_undegraded":"2024-03-16T19:32:07.080734+0000","last_fullsized":"2024-03-16T19:32:07.080734+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083795+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:08.083795+0000","last_peered":"2024-03-16T19:32:08.083795+0000","last_clean":"2024-03-16T19:32:08.083795+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:08.083795+0000","last_undegraded":"2024-03-16T19:32:08.083795+0000","last_fullsized":"2024-03-16T19:32:08.083795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:46:49.628764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094246+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:08.094246+0000","last_peered":"2024-03-16T19:32:08.094246+0000","last_clean":"2024-03-16T19:32:08.094246+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:08.094246+0000","last_undegraded":"2024-03-16T19:32:08.094246+0000","last_fullsized":"2024-03-16T19:32:08.094246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":64,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.181279+0000","last_change":"2024-03-16T19:31:26.809671+0000","last_active":"2024-03-16T19:32:05.181279+0000","last_peered":"2024-03-16T19:32:05.181279+0000","last_clean":"2024-03-16T19:32:05.181279+0000","last_became_active":"2024-03-16T19:31:26.809344+0000","last_became_peered":"2024-03-16T19:31:26.809344+0000","last_unstale":"2024-03-16T19:32:05.181279+0000","last_undegraded":"2024-03-16T19:32:05.181279+0000","last_fullsized":"2024-03-16T19:32:05.181279+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:49.907553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.095090+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:08.095090+0000","last_peered":"2024-03-16T19:32:08.095090+0000","last_clean":"2024-03-16T19:32:08.095090+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:08.095090+0000","last_undegraded":"2024-03-16T19:32:08.095090+0000","last_fullsized":"2024-03-16T19:32:08.095090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089208+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:08.089208+0000","last_peered":"2024-03-16T19:32:08.089208+0000","last_clean":"2024-03-16T19:32:08.089208+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:08.089208+0000","last_undegraded":"2024-03-16T19:32:08.089208+0000","last_fullsized":"2024-03-16T19:32:08.089208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:03.618618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080995+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:07.080995+0000","last_peered":"2024-03-16T19:32:07.080995+0000","last_clean":"2024-03-16T19:32:07.080995+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:07.080995+0000","last_undegraded":"2024-03-16T19:32:07.080995+0000","last_fullsized":"2024-03-16T19:32:07.080995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620022+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:07.620022+0000","last_peered":"2024-03-16T19:32:07.620022+0000","last_clean":"2024-03-16T19:32:07.620022+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:07.620022+0000","last_undegraded":"2024-03-16T19:32:07.620022+0000","last_fullsized":"2024-03-16T19:32:07.620022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084106+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:08.084106+0000","last_peered":"2024-03-16T19:32:08.084106+0000","last_clean":"2024-03-16T19:32:08.084106+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:08.084106+0000","last_undegraded":"2024-03-16T19:32:08.084106+0000","last_fullsized":"2024-03-16T19:32:08.084106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081285+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:07.081285+0000","last_peered":"2024-03-16T19:32:07.081285+0000","last_clean":"2024-03-16T19:32:07.081285+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:07.081285+0000","last_undegraded":"2024-03-16T19:32:07.081285+0000","last_fullsized":"2024-03-16T19:32:07.081285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083913+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:08.083913+0000","last_peered":"2024-03-16T19:32:08.083913+0000","last_clean":"2024-03-16T19:32:08.083913+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:08.083913+0000","last_undegraded":"2024-03-16T19:32:08.083913+0000","last_fullsized":"2024-03-16T19:32:08.083913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081056+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:07.081056+0000","last_peered":"2024-03-16T19:32:07.081056+0000","last_clean":"2024-03-16T19:32:07.081056+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:07.081056+0000","last_undegraded":"2024-03-16T19:32:07.081056+0000","last_fullsized":"2024-03-16T19:32:07.081056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":162,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077568+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:07.077568+0000","last_peered":"2024-03-16T19:32:07.077568+0000","last_clean":"2024-03-16T19:32:07.077568+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:07.077568+0000","last_undegraded":"2024-03-16T19:32:07.077568+0000","last_fullsized":"2024-03-16T19:32:07.077568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:19.317523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088482+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088482+0000","last_peered":"2024-03-16T19:32:08.088482+0000","last_clean":"2024-03-16T19:32:08.088482+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088482+0000","last_undegraded":"2024-03-16T19:32:08.088482+0000","last_fullsized":"2024-03-16T19:32:08.088482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:58:01.695714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086003+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:08.086003+0000","last_peered":"2024-03-16T19:32:08.086003+0000","last_clean":"2024-03-16T19:32:08.086003+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:08.086003+0000","last_undegraded":"2024-03-16T19:32:08.086003+0000","last_fullsized":"2024-03-16T19:32:08.086003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:17.040490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077402+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:07.077402+0000","last_peered":"2024-03-16T19:32:07.077402+0000","last_clean":"2024-03-16T19:32:07.077402+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:07.077402+0000","last_undegraded":"2024-03-16T19:32:07.077402+0000","last_fullsized":"2024-03-16T19:32:07.077402+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:28:10.116570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40494,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":80988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28824,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343016,"statfs":{"total":96636764160,"available":96607248384,"internally_reserved":0,"allocated":1847296,"data_stored":1180033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343552,"statfs":{"total":96636764160,"available":96607797248,"internally_reserved":0,"allocated":1298432,"data_stored":628375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1253376,"data_stored":589173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1290240,"data_stored":622664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122054,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29384,"kb_used_data":1212,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342456,"statfs":{"total":96636764160,"available":96606674944,"internally_reserved":0,"allocated":1241088,"data_stored":583372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28240,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343600,"statfs":{"total":96636764160,"available":96607846400,"internally_reserved":0,"allocated":1245184,"data_stored":583128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28832,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343008,"statfs":{"total":96636764160,"available":96607240192,"internally_reserved":0,"allocated":1855488,"data_stored":1180089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:13.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:13.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.726+0000 7f7b9f8fb640 1 Processor -- start 2024-03-16T19:32:13.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.727+0000 7f7b9f8fb640 1 -- start start 2024-03-16T19:32:13.709 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9f8fb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b9806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9f8fb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b9806fe10 0x7f7b9815de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9f8fb640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9815e340 con 0x7f7b980c8ba0 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9f8fb640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b980cbba0 con 0x7f7b9806fe10 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9f8fb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980c91f0 con 0x7f7b98154cb0 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9ce96640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7b9806fe10 0x7f7b9815de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9d697640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b9806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9d697640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b9806dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:47840/0 (socket says 172.21.15.172:47840) 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9de98640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f7b980c8ba0 0x7f7b980cabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:49322/0 (socket says 172.21.15.172:49322) 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b9d697640 1 -- 172.21.15.172:0/1747929675 learned_addr learned my addr 172.21.15.172:0/1747929675 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b8ffff640 1 -- 172.21.15.172:0/1747929675 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189696160 0 0) 0x7f7b9815e340 con 0x7f7b980c8ba0 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.728+0000 7f7b8ffff640 1 -- 172.21.15.172:0/1747929675 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b84003520 con 0x7f7b980c8ba0 2024-03-16T19:32:13.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.729+0000 7f7b9ce96640 1 -- 172.21.15.172:0/1747929675 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 msgr2=0x7f7b9806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:13.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.729+0000 7f7b9ce96640 1 --2- 172.21.15.172:0/1747929675 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b9806dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:13.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.729+0000 7f7b9ce96640 1 -- 172.21.15.172:0/1747929675 >> v1:172.21.15.174:6789/0 conn(0x7f7b980c8ba0 legacy=0x7f7b980cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:13.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.729+0000 7f7b9ce96640 1 -- 172.21.15.172:0/1747929675 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9806c180 con 0x7f7b9806fe10 2024-03-16T19:32:13.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.730+0000 7f7b9ce96640 1 --2- 172.21.15.172:0/1747929675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 0x7f7b9815de00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7b940097a0 tx=0x7f7b9402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712f93a9905227d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:13.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.730+0000 7f7b8ffff640 1 -- 172.21.15.172:0/1747929675 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b940316f0 con 0x7f7b9806fe10 2024-03-16T19:32:13.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.730+0000 7f7b8ffff640 1 -- 172.21.15.172:0/1747929675 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b94031850 con 0x7f7b9806fe10 2024-03-16T19:32:13.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.730+0000 7f7b8ffff640 1 -- 172.21.15.172:0/1747929675 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b94031bc0 con 0x7f7b9806fe10 2024-03-16T19:32:13.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.730+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1747929675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 msgr2=0x7f7b9815de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:13.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.730+0000 7f7b9f8fb640 1 --2- 172.21.15.172:0/1747929675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 0x7f7b9815de00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7b940097a0 tx=0x7f7b9402b790 comp rx=0 tx=0).stop 2024-03-16T19:32:13.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.731+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1747929675 shutdown_connections 2024-03-16T19:32:13.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.731+0000 7f7b9f8fb640 1 --2- 172.21.15.172:0/1747929675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 0x7f7b9815de00 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:13.712 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.731+0000 7f7b9f8fb640 1 --2- 172.21.15.172:0/1747929675 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b9806dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:13.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.731+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1747929675 >> 172.21.15.172:0/1747929675 conn(0x7f7b98070860 msgr2=0x7f7b98070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:13.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.731+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1747929675 shutdown_connections 2024-03-16T19:32:13.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.731+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1747929675 wait complete. 2024-03-16T19:32:13.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.732+0000 7f7b9f8fb640 1 Processor -- start 2024-03-16T19:32:13.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.732+0000 7f7b9f8fb640 1 -- start start 2024-03-16T19:32:13.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9f8fb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 0x7f7b9814f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:13.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9f8fb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b980c8ba0 0x7f7b98159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:13.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9f8fb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b98159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:13.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9f8fb640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9814ee40 con 0x7f7b9806fe10 2024-03-16T19:32:13.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9f8fb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b981593b0 con 0x7f7b980c8ba0 2024-03-16T19:32:13.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9f8fb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9815b1b0 con 0x7f7b98154cb0 2024-03-16T19:32:13.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9d697640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 0x7f7b9814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:13.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9d697640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 0x7f7b9814f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57656/0 (socket says 172.21.15.172:57656) 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9d697640 1 -- 172.21.15.172:0/1447748030 learned_addr learned my addr 172.21.15.172:0/1447748030 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9de98640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b98159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.733+0000 7f7b9ce96640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b980c8ba0 0x7f7b98159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b9de98640 1 -- 172.21.15.172:0/1447748030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b980c8ba0 msgr2=0x7f7b98159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b9de98640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b980c8ba0 0x7f7b98159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b9de98640 1 -- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 msgr2=0x7f7b9814f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b9de98640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 0x7f7b9814f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b9de98640 1 -- 172.21.15.172:0/1447748030 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b94009400 con 0x7f7b98154cb0 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b9d697640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 0x7f7b9814f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b9de98640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b98159e40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000c700 tx=0x7f7b9000cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b8dffb640 1 -- 172.21.15.172:0/1447748030 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9000ce10 con 0x7f7b98154cb0 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b8dffb640 1 -- 172.21.15.172:0/1447748030 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9000a640 con 0x7f7b98154cb0 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b98158b10 con 0x7f7b98154cb0 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.734+0000 7f7b8dffb640 1 -- 172.21.15.172:0/1447748030 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9000a9f0 con 0x7f7b98154cb0 2024-03-16T19:32:13.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.735+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b981590d0 con 0x7f7b98154cb0 2024-03-16T19:32:13.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.736+0000 7f7b8dffb640 1 -- 172.21.15.172:0/1447748030 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7b9000ab50 con 0x7f7b98154cb0 2024-03-16T19:32:13.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.736+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b98159620 con 0x7f7b98154cb0 2024-03-16T19:32:13.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.737+0000 7f7b8dffb640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b7c06b3e0 0x7f7b7c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:13.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.737+0000 7f7b8dffb640 1 -- 172.21.15.172:0/1447748030 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f7b90089410 con 0x7f7b98154cb0 2024-03-16T19:32:13.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.737+0000 7f7b9d697640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b7c06b3e0 0x7f7b7c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:13.718 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.738+0000 7f7b9d697640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b7c06b3e0 0x7f7b7c06d8a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b88006f40 tx=0x7f7b88009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:13.721 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:13.741+0000 7f7b8dffb640 1 -- 172.21.15.172:0/1447748030 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b98159620 con 0x7f7b98154cb0 2024-03-16T19:32:14.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.047+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b980c9790 con 0x7f7b7c06b3e0 2024-03-16T19:32:14.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.057+0000 7f7b8dffb640 1 -- 172.21.15.172:0/1447748030 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252318 (secure 0 0 0) 0x7f7b88014440 con 0x7f7b7c06b3e0 2024-03-16T19:32:14.040 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:14.041 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.060+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b7c06b3e0 msgr2=0x7f7b7c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.060+0000 7f7b9f8fb640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b7c06b3e0 0x7f7b7c06d8a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b88006f40 tx=0x7f7b88009240 comp rx=0 tx=0).stop 2024-03-16T19:32:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.060+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 msgr2=0x7f7b98159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.060+0000 7f7b9f8fb640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b98159e40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000c700 tx=0x7f7b9000cbc0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.061+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 shutdown_connections 2024-03-16T19:32:14.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.061+0000 7f7b9f8fb640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7b7c06b3e0 0x7f7b7c06d8a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.061+0000 7f7b9f8fb640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b98154cb0 0x7f7b98159e40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.061+0000 7f7b9f8fb640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b980c8ba0 0x7f7b98159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.061+0000 7f7b9f8fb640 1 --2- 172.21.15.172:0/1447748030 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9806fe10 0x7f7b9814f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.061+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 >> 172.21.15.172:0/1447748030 conn(0x7f7b98070860 msgr2=0x7f7b98150630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:14.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.061+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 shutdown_connections 2024-03-16T19:32:14.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.061+0000 7f7b9f8fb640 1 -- 172.21.15.172:0/1447748030 wait complete. 2024-03-16T19:32:14.055 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-03-16T19:32:12.231761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630862,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":245,"num_write_kb":2489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228360,"kb_used_data":11040,"kb_used_omap":16,"kb_used_meta":217199,"kb_avail":754746360,"statfs":{"total":773094113280,"available":772860272640,"internally_reserved":0,"allocated":11304960,"data_stored":5962108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":222412203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.172530"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081164+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:07.081164+0000","last_peered":"2024-03-16T19:32:07.081164+0000","last_clean":"2024-03-16T19:32:07.081164+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:07.081164+0000","last_undegraded":"2024-03-16T19:32:07.081164+0000","last_fullsized":"2024-03-16T19:32:07.081164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094518+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:08.094518+0000","last_peered":"2024-03-16T19:32:08.094518+0000","last_clean":"2024-03-16T19:32:08.094518+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:08.094518+0000","last_undegraded":"2024-03-16T19:32:08.094518+0000","last_fullsized":"2024-03-16T19:32:08.094518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077201+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:07.077201+0000","last_peered":"2024-03-16T19:32:07.077201+0000","last_clean":"2024-03-16T19:32:07.077201+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:07.077201+0000","last_undegraded":"2024-03-16T19:32:07.077201+0000","last_fullsized":"2024-03-16T19:32:07.077201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:47.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083818+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:08.083818+0000","last_peered":"2024-03-16T19:32:08.083818+0000","last_clean":"2024-03-16T19:32:08.083818+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:08.083818+0000","last_undegraded":"2024-03-16T19:32:08.083818+0000","last_fullsized":"2024-03-16T19:32:08.083818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080899+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:07.080899+0000","last_peered":"2024-03-16T19:32:07.080899+0000","last_clean":"2024-03-16T19:32:07.080899+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:07.080899+0000","last_undegraded":"2024-03-16T19:32:07.080899+0000","last_fullsized":"2024-03-16T19:32:07.080899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083640+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:08.083640+0000","last_peered":"2024-03-16T19:32:08.083640+0000","last_clean":"2024-03-16T19:32:08.083640+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:08.083640+0000","last_undegraded":"2024-03-16T19:32:08.083640+0000","last_fullsized":"2024-03-16T19:32:08.083640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":187,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080786+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:07.080786+0000","last_peered":"2024-03-16T19:32:07.080786+0000","last_clean":"2024-03-16T19:32:07.080786+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:07.080786+0000","last_undegraded":"2024-03-16T19:32:07.080786+0000","last_fullsized":"2024-03-16T19:32:07.080786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086880+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:08.086880+0000","last_peered":"2024-03-16T19:32:08.086880+0000","last_clean":"2024-03-16T19:32:08.086880+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:08.086880+0000","last_undegraded":"2024-03-16T19:32:08.086880+0000","last_fullsized":"2024-03-16T19:32:08.086880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:58:31.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081177+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:07.081177+0000","last_peered":"2024-03-16T19:32:07.081177+0000","last_clean":"2024-03-16T19:32:07.081177+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:07.081177+0000","last_undegraded":"2024-03-16T19:32:07.081177+0000","last_fullsized":"2024-03-16T19:32:07.081177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620209+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:07.620209+0000","last_peered":"2024-03-16T19:32:07.620209+0000","last_clean":"2024-03-16T19:32:07.620209+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:07.620209+0000","last_undegraded":"2024-03-16T19:32:07.620209+0000","last_fullsized":"2024-03-16T19:32:07.620209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620126+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:07.620126+0000","last_peered":"2024-03-16T19:32:07.620126+0000","last_clean":"2024-03-16T19:32:07.620126+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:07.620126+0000","last_undegraded":"2024-03-16T19:32:07.620126+0000","last_fullsized":"2024-03-16T19:32:07.620126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084143+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:08.084143+0000","last_peered":"2024-03-16T19:32:08.084143+0000","last_clean":"2024-03-16T19:32:08.084143+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:08.084143+0000","last_undegraded":"2024-03-16T19:32:08.084143+0000","last_fullsized":"2024-03-16T19:32:08.084143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088886+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:08.088886+0000","last_peered":"2024-03-16T19:32:08.088886+0000","last_clean":"2024-03-16T19:32:08.088886+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:08.088886+0000","last_undegraded":"2024-03-16T19:32:08.088886+0000","last_fullsized":"2024-03-16T19:32:08.088886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:52.102388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"84'3","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077055+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:07.077055+0000","last_peered":"2024-03-16T19:32:07.077055+0000","last_clean":"2024-03-16T19:32:07.077055+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:07.077055+0000","last_undegraded":"2024-03-16T19:32:07.077055+0000","last_fullsized":"2024-03-16T19:32:07.077055+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:39.289238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094126+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:08.094126+0000","last_peered":"2024-03-16T19:32:08.094126+0000","last_clean":"2024-03-16T19:32:08.094126+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:08.094126+0000","last_undegraded":"2024-03-16T19:32:08.094126+0000","last_fullsized":"2024-03-16T19:32:08.094126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083403+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:08.083403+0000","last_peered":"2024-03-16T19:32:08.083403+0000","last_clean":"2024-03-16T19:32:08.083403+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:08.083403+0000","last_undegraded":"2024-03-16T19:32:08.083403+0000","last_fullsized":"2024-03-16T19:32:08.083403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084254+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:08.084254+0000","last_peered":"2024-03-16T19:32:08.084254+0000","last_clean":"2024-03-16T19:32:08.084254+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:08.084254+0000","last_undegraded":"2024-03-16T19:32:08.084254+0000","last_fullsized":"2024-03-16T19:32:08.084254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080010+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:07.080010+0000","last_peered":"2024-03-16T19:32:07.080010+0000","last_clean":"2024-03-16T19:32:07.080010+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:07.080010+0000","last_undegraded":"2024-03-16T19:32:07.080010+0000","last_fullsized":"2024-03-16T19:32:07.080010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:36:06.175327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081155+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:07.081155+0000","last_peered":"2024-03-16T19:32:07.081155+0000","last_clean":"2024-03-16T19:32:07.081155+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:07.081155+0000","last_undegraded":"2024-03-16T19:32:07.081155+0000","last_fullsized":"2024-03-16T19:32:07.081155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620154+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:07.620154+0000","last_peered":"2024-03-16T19:32:07.620154+0000","last_clean":"2024-03-16T19:32:07.620154+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:07.620154+0000","last_undegraded":"2024-03-16T19:32:07.620154+0000","last_fullsized":"2024-03-16T19:32:07.620154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077197+0000","last_change":"2024-03-16T19:31:51.807295+0000","last_active":"2024-03-16T19:32:07.077197+0000","last_peered":"2024-03-16T19:32:07.077197+0000","last_clean":"2024-03-16T19:32:07.077197+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:07.077197+0000","last_undegraded":"2024-03-16T19:32:07.077197+0000","last_fullsized":"2024-03-16T19:32:07.077197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:04.578640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":50,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079813+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:07.079813+0000","last_peered":"2024-03-16T19:32:07.079813+0000","last_clean":"2024-03-16T19:32:07.079813+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:07.079813+0000","last_undegraded":"2024-03-16T19:32:07.079813+0000","last_fullsized":"2024-03-16T19:32:07.079813+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:41:06.377004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088245+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:08.088245+0000","last_peered":"2024-03-16T19:32:08.088245+0000","last_clean":"2024-03-16T19:32:08.088245+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:08.088245+0000","last_undegraded":"2024-03-16T19:32:08.088245+0000","last_fullsized":"2024-03-16T19:32:08.088245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:47:08.251341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083454+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:08.083454+0000","last_peered":"2024-03-16T19:32:08.083454+0000","last_clean":"2024-03-16T19:32:08.083454+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:08.083454+0000","last_undegraded":"2024-03-16T19:32:08.083454+0000","last_fullsized":"2024-03-16T19:32:08.083454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086134+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:08.086134+0000","last_peered":"2024-03-16T19:32:08.086134+0000","last_clean":"2024-03-16T19:32:08.086134+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:08.086134+0000","last_undegraded":"2024-03-16T19:32:08.086134+0000","last_fullsized":"2024-03-16T19:32:08.086134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:10.818175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620238+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:07.620238+0000","last_peered":"2024-03-16T19:32:07.620238+0000","last_clean":"2024-03-16T19:32:07.620238+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:07.620238+0000","last_undegraded":"2024-03-16T19:32:07.620238+0000","last_fullsized":"2024-03-16T19:32:07.620238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":187,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093433+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:08.093433+0000","last_peered":"2024-03-16T19:32:08.093433+0000","last_clean":"2024-03-16T19:32:08.093433+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:08.093433+0000","last_undegraded":"2024-03-16T19:32:08.093433+0000","last_fullsized":"2024-03-16T19:32:08.093433+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087892+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:08.087892+0000","last_peered":"2024-03-16T19:32:08.087892+0000","last_clean":"2024-03-16T19:32:08.087892+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:08.087892+0000","last_undegraded":"2024-03-16T19:32:08.087892+0000","last_fullsized":"2024-03-16T19:32:08.087892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:12:08.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077141+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:07.077141+0000","last_peered":"2024-03-16T19:32:07.077141+0000","last_clean":"2024-03-16T19:32:07.077141+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:07.077141+0000","last_undegraded":"2024-03-16T19:32:07.077141+0000","last_fullsized":"2024-03-16T19:32:07.077141+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:12.646493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083749+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:08.083749+0000","last_peered":"2024-03-16T19:32:08.083749+0000","last_clean":"2024-03-16T19:32:08.083749+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:08.083749+0000","last_undegraded":"2024-03-16T19:32:08.083749+0000","last_fullsized":"2024-03-16T19:32:08.083749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079774+0000","last_change":"2024-03-16T19:31:49.974297+0000","last_active":"2024-03-16T19:32:07.079774+0000","last_peered":"2024-03-16T19:32:07.079774+0000","last_clean":"2024-03-16T19:32:07.079774+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:07.079774+0000","last_undegraded":"2024-03-16T19:32:07.079774+0000","last_fullsized":"2024-03-16T19:32:07.079774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:51:57.535372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":186,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.619981+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:07.619981+0000","last_peered":"2024-03-16T19:32:07.619981+0000","last_clean":"2024-03-16T19:32:07.619981+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:07.619981+0000","last_undegraded":"2024-03-16T19:32:07.619981+0000","last_fullsized":"2024-03-16T19:32:07.619981+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084174+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:32:08.084174+0000","last_peered":"2024-03-16T19:32:08.084174+0000","last_clean":"2024-03-16T19:32:08.084174+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:08.084174+0000","last_undegraded":"2024-03-16T19:32:08.084174+0000","last_fullsized":"2024-03-16T19:32:08.084174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089011+0000","last_change":"2024-03-16T19:31:26.808032+0000","last_active":"2024-03-16T19:32:08.089011+0000","last_peered":"2024-03-16T19:32:08.089011+0000","last_clean":"2024-03-16T19:32:08.089011+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:08.089011+0000","last_undegraded":"2024-03-16T19:32:08.089011+0000","last_fullsized":"2024-03-16T19:32:08.089011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:15.218778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077339+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:07.077339+0000","last_peered":"2024-03-16T19:32:07.077339+0000","last_clean":"2024-03-16T19:32:07.077339+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:07.077339+0000","last_undegraded":"2024-03-16T19:32:07.077339+0000","last_fullsized":"2024-03-16T19:32:07.077339+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:16:31.606469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620053+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:07.620053+0000","last_peered":"2024-03-16T19:32:07.620053+0000","last_clean":"2024-03-16T19:32:07.620053+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:07.620053+0000","last_undegraded":"2024-03-16T19:32:07.620053+0000","last_fullsized":"2024-03-16T19:32:07.620053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086078+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086078+0000","last_peered":"2024-03-16T19:32:08.086078+0000","last_clean":"2024-03-16T19:32:08.086078+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086078+0000","last_undegraded":"2024-03-16T19:32:08.086078+0000","last_fullsized":"2024-03-16T19:32:08.086078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:23:19.824904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086998+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:08.086998+0000","last_peered":"2024-03-16T19:32:08.086998+0000","last_clean":"2024-03-16T19:32:08.086998+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:08.086998+0000","last_undegraded":"2024-03-16T19:32:08.086998+0000","last_fullsized":"2024-03-16T19:32:08.086998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:45.782085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094004+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:08.094004+0000","last_peered":"2024-03-16T19:32:08.094004+0000","last_clean":"2024-03-16T19:32:08.094004+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:08.094004+0000","last_undegraded":"2024-03-16T19:32:08.094004+0000","last_fullsized":"2024-03-16T19:32:08.094004+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620263+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:07.620263+0000","last_peered":"2024-03-16T19:32:07.620263+0000","last_clean":"2024-03-16T19:32:07.620263+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:07.620263+0000","last_undegraded":"2024-03-16T19:32:07.620263+0000","last_fullsized":"2024-03-16T19:32:07.620263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083754+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:08.083754+0000","last_peered":"2024-03-16T19:32:08.083754+0000","last_clean":"2024-03-16T19:32:08.083754+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:08.083754+0000","last_undegraded":"2024-03-16T19:32:08.083754+0000","last_fullsized":"2024-03-16T19:32:08.083754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077496+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:07.077496+0000","last_peered":"2024-03-16T19:32:07.077496+0000","last_clean":"2024-03-16T19:32:07.077496+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:07.077496+0000","last_undegraded":"2024-03-16T19:32:07.077496+0000","last_fullsized":"2024-03-16T19:32:07.077496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:16.418962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093565+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:08.093565+0000","last_peered":"2024-03-16T19:32:08.093565+0000","last_clean":"2024-03-16T19:32:08.093565+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:08.093565+0000","last_undegraded":"2024-03-16T19:32:08.093565+0000","last_fullsized":"2024-03-16T19:32:08.093565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"peering","last_fresh":"2024-03-16T19:32:05.059246+0000","last_change":"2024-03-16T19:32:07.978687+0000","last_active":"2024-03-16T19:32:07.978687+0000","last_peered":"2024-03-16T19:32:05.059246+0000","last_clean":"2024-03-16T19:32:05.059246+0000","last_became_active":"2024-03-16T19:29:26.362013+0000","last_became_peered":"2024-03-16T19:29:26.362013+0000","last_unstale":"2024-03-16T19:32:05.059246+0000","last_undegraded":"2024-03-16T19:32:05.059246+0000","last_fullsized":"2024-03-16T19:32:05.059246+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"86'23","reported_seq":231,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620164+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:07.620164+0000","last_peered":"2024-03-16T19:32:07.620164+0000","last_clean":"2024-03-16T19:32:07.620164+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:07.620164+0000","last_undegraded":"2024-03-16T19:32:07.620164+0000","last_fullsized":"2024-03-16T19:32:07.620164+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39292,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":187,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080997+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:07.080997+0000","last_peered":"2024-03-16T19:32:07.080997+0000","last_clean":"2024-03-16T19:32:07.080997+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:07.080997+0000","last_undegraded":"2024-03-16T19:32:07.080997+0000","last_fullsized":"2024-03-16T19:32:07.080997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077290+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:07.077290+0000","last_peered":"2024-03-16T19:32:07.077290+0000","last_clean":"2024-03-16T19:32:07.077290+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:07.077290+0000","last_undegraded":"2024-03-16T19:32:07.077290+0000","last_fullsized":"2024-03-16T19:32:07.077290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:59.818393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.076992+0000","last_change":"2024-03-16T19:31:51.807895+0000","last_active":"2024-03-16T19:32:07.076992+0000","last_peered":"2024-03-16T19:32:07.076992+0000","last_clean":"2024-03-16T19:32:07.076992+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:07.076992+0000","last_undegraded":"2024-03-16T19:32:07.076992+0000","last_fullsized":"2024-03-16T19:32:07.076992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:25.988237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080958+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:07.080958+0000","last_peered":"2024-03-16T19:32:07.080958+0000","last_clean":"2024-03-16T19:32:07.080958+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:07.080958+0000","last_undegraded":"2024-03-16T19:32:07.080958+0000","last_fullsized":"2024-03-16T19:32:07.080958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":178,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620021+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:07.620021+0000","last_peered":"2024-03-16T19:32:07.620021+0000","last_clean":"2024-03-16T19:32:07.620021+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:07.620021+0000","last_undegraded":"2024-03-16T19:32:07.620021+0000","last_fullsized":"2024-03-16T19:32:07.620021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":186,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.619987+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:07.619987+0000","last_peered":"2024-03-16T19:32:07.619987+0000","last_clean":"2024-03-16T19:32:07.619987+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:07.619987+0000","last_undegraded":"2024-03-16T19:32:07.619987+0000","last_fullsized":"2024-03-16T19:32:07.619987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":56,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079696+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:07.079696+0000","last_peered":"2024-03-16T19:32:07.079696+0000","last_clean":"2024-03-16T19:32:07.079696+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:07.079696+0000","last_undegraded":"2024-03-16T19:32:07.079696+0000","last_fullsized":"2024-03-16T19:32:07.079696+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:33.634146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080858+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:07.080858+0000","last_peered":"2024-03-16T19:32:07.080858+0000","last_clean":"2024-03-16T19:32:07.080858+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:07.080858+0000","last_undegraded":"2024-03-16T19:32:07.080858+0000","last_fullsized":"2024-03-16T19:32:07.080858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084319+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084319+0000","last_peered":"2024-03-16T19:32:08.084319+0000","last_clean":"2024-03-16T19:32:08.084319+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084319+0000","last_undegraded":"2024-03-16T19:32:08.084319+0000","last_fullsized":"2024-03-16T19:32:08.084319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:53.021797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081088+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:07.081088+0000","last_peered":"2024-03-16T19:32:07.081088+0000","last_clean":"2024-03-16T19:32:07.081088+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:07.081088+0000","last_undegraded":"2024-03-16T19:32:07.081088+0000","last_fullsized":"2024-03-16T19:32:07.081088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":53,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079883+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:07.079883+0000","last_peered":"2024-03-16T19:32:07.079883+0000","last_clean":"2024-03-16T19:32:07.079883+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:07.079883+0000","last_undegraded":"2024-03-16T19:32:07.079883+0000","last_fullsized":"2024-03-16T19:32:07.079883+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:07:39.853290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081238+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:07.081238+0000","last_peered":"2024-03-16T19:32:07.081238+0000","last_clean":"2024-03-16T19:32:07.081238+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:07.081238+0000","last_undegraded":"2024-03-16T19:32:07.081238+0000","last_fullsized":"2024-03-16T19:32:07.081238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093703+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:08.093703+0000","last_peered":"2024-03-16T19:32:08.093703+0000","last_clean":"2024-03-16T19:32:08.093703+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:08.093703+0000","last_undegraded":"2024-03-16T19:32:08.093703+0000","last_fullsized":"2024-03-16T19:32:08.093703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084002+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:08.084002+0000","last_peered":"2024-03-16T19:32:08.084002+0000","last_clean":"2024-03-16T19:32:08.084002+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:08.084002+0000","last_undegraded":"2024-03-16T19:32:08.084002+0000","last_fullsized":"2024-03-16T19:32:08.084002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079907+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:07.079907+0000","last_peered":"2024-03-16T19:32:07.079907+0000","last_clean":"2024-03-16T19:32:07.079907+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:07.079907+0000","last_undegraded":"2024-03-16T19:32:07.079907+0000","last_fullsized":"2024-03-16T19:32:07.079907+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:59:55.019036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":146,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080905+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:07.080905+0000","last_peered":"2024-03-16T19:32:07.080905+0000","last_clean":"2024-03-16T19:32:07.080905+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:07.080905+0000","last_undegraded":"2024-03-16T19:32:07.080905+0000","last_fullsized":"2024-03-16T19:32:07.080905+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":361,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080930+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:07.080930+0000","last_peered":"2024-03-16T19:32:07.080930+0000","last_clean":"2024-03-16T19:32:07.080930+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:07.080930+0000","last_undegraded":"2024-03-16T19:32:07.080930+0000","last_fullsized":"2024-03-16T19:32:07.080930+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087541+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:08.087541+0000","last_peered":"2024-03-16T19:32:08.087541+0000","last_clean":"2024-03-16T19:32:08.087541+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:08.087541+0000","last_undegraded":"2024-03-16T19:32:08.087541+0000","last_fullsized":"2024-03-16T19:32:08.087541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:16:54.768710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080738+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:07.080738+0000","last_peered":"2024-03-16T19:32:07.080738+0000","last_clean":"2024-03-16T19:32:07.080738+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:07.080738+0000","last_undegraded":"2024-03-16T19:32:07.080738+0000","last_fullsized":"2024-03-16T19:32:07.080738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077009+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:07.077009+0000","last_peered":"2024-03-16T19:32:07.077009+0000","last_clean":"2024-03-16T19:32:07.077009+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:07.077009+0000","last_undegraded":"2024-03-16T19:32:07.077009+0000","last_fullsized":"2024-03-16T19:32:07.077009+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:44.308565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":50,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.079702+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:07.079702+0000","last_peered":"2024-03-16T19:32:07.079702+0000","last_clean":"2024-03-16T19:32:07.079702+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:07.079702+0000","last_undegraded":"2024-03-16T19:32:07.079702+0000","last_fullsized":"2024-03-16T19:32:07.079702+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:20:21.014268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.096035+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:08.096035+0000","last_peered":"2024-03-16T19:32:08.096035+0000","last_clean":"2024-03-16T19:32:08.096035+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:08.096035+0000","last_undegraded":"2024-03-16T19:32:08.096035+0000","last_fullsized":"2024-03-16T19:32:08.096035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084314+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:08.084314+0000","last_peered":"2024-03-16T19:32:08.084314+0000","last_clean":"2024-03-16T19:32:08.084314+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:08.084314+0000","last_undegraded":"2024-03-16T19:32:08.084314+0000","last_fullsized":"2024-03-16T19:32:08.084314+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:01.856961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093299+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:08.093299+0000","last_peered":"2024-03-16T19:32:08.093299+0000","last_clean":"2024-03-16T19:32:08.093299+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:08.093299+0000","last_undegraded":"2024-03-16T19:32:08.093299+0000","last_fullsized":"2024-03-16T19:32:08.093299+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084193+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:08.084193+0000","last_peered":"2024-03-16T19:32:08.084193+0000","last_clean":"2024-03-16T19:32:08.084193+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:08.084193+0000","last_undegraded":"2024-03-16T19:32:08.084193+0000","last_fullsized":"2024-03-16T19:32:08.084193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081176+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:07.081176+0000","last_peered":"2024-03-16T19:32:07.081176+0000","last_clean":"2024-03-16T19:32:07.081176+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:07.081176+0000","last_undegraded":"2024-03-16T19:32:07.081176+0000","last_fullsized":"2024-03-16T19:32:07.081176+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093171+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:08.093171+0000","last_peered":"2024-03-16T19:32:08.093171+0000","last_clean":"2024-03-16T19:32:08.093171+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:08.093171+0000","last_undegraded":"2024-03-16T19:32:08.093171+0000","last_fullsized":"2024-03-16T19:32:08.093171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'11","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094654+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:08.094654+0000","last_peered":"2024-03-16T19:32:08.094654+0000","last_clean":"2024-03-16T19:32:08.094654+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:08.094654+0000","last_undegraded":"2024-03-16T19:32:08.094654+0000","last_fullsized":"2024-03-16T19:32:08.094654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088741+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:08.088741+0000","last_peered":"2024-03-16T19:32:08.088741+0000","last_clean":"2024-03-16T19:32:08.088741+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:08.088741+0000","last_undegraded":"2024-03-16T19:32:08.088741+0000","last_fullsized":"2024-03-16T19:32:08.088741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:18.996236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620312+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:07.620312+0000","last_peered":"2024-03-16T19:32:07.620312+0000","last_clean":"2024-03-16T19:32:07.620312+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:07.620312+0000","last_undegraded":"2024-03-16T19:32:07.620312+0000","last_fullsized":"2024-03-16T19:32:07.620312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077386+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:07.077386+0000","last_peered":"2024-03-16T19:32:07.077386+0000","last_clean":"2024-03-16T19:32:07.077386+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:07.077386+0000","last_undegraded":"2024-03-16T19:32:07.077386+0000","last_fullsized":"2024-03-16T19:32:07.077386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:02.024100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":189,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080734+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:07.080734+0000","last_peered":"2024-03-16T19:32:07.080734+0000","last_clean":"2024-03-16T19:32:07.080734+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:07.080734+0000","last_undegraded":"2024-03-16T19:32:07.080734+0000","last_fullsized":"2024-03-16T19:32:07.080734+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083795+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:08.083795+0000","last_peered":"2024-03-16T19:32:08.083795+0000","last_clean":"2024-03-16T19:32:08.083795+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:08.083795+0000","last_undegraded":"2024-03-16T19:32:08.083795+0000","last_fullsized":"2024-03-16T19:32:08.083795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:46:49.628764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094246+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:08.094246+0000","last_peered":"2024-03-16T19:32:08.094246+0000","last_clean":"2024-03-16T19:32:08.094246+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:08.094246+0000","last_undegraded":"2024-03-16T19:32:08.094246+0000","last_fullsized":"2024-03-16T19:32:08.094246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":64,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-16T19:32:05.181279+0000","last_change":"2024-03-16T19:31:26.809671+0000","last_active":"2024-03-16T19:32:05.181279+0000","last_peered":"2024-03-16T19:32:05.181279+0000","last_clean":"2024-03-16T19:32:05.181279+0000","last_became_active":"2024-03-16T19:31:26.809344+0000","last_became_peered":"2024-03-16T19:31:26.809344+0000","last_unstale":"2024-03-16T19:32:05.181279+0000","last_undegraded":"2024-03-16T19:32:05.181279+0000","last_fullsized":"2024-03-16T19:32:05.181279+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:49.907553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.095090+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:08.095090+0000","last_peered":"2024-03-16T19:32:08.095090+0000","last_clean":"2024-03-16T19:32:08.095090+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:08.095090+0000","last_undegraded":"2024-03-16T19:32:08.095090+0000","last_fullsized":"2024-03-16T19:32:08.095090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089208+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:08.089208+0000","last_peered":"2024-03-16T19:32:08.089208+0000","last_clean":"2024-03-16T19:32:08.089208+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:08.089208+0000","last_undegraded":"2024-03-16T19:32:08.089208+0000","last_fullsized":"2024-03-16T19:32:08.089208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:03.618618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.080995+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:07.080995+0000","last_peered":"2024-03-16T19:32:07.080995+0000","last_clean":"2024-03-16T19:32:07.080995+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:07.080995+0000","last_undegraded":"2024-03-16T19:32:07.080995+0000","last_fullsized":"2024-03-16T19:32:07.080995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":176,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.620022+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:07.620022+0000","last_peered":"2024-03-16T19:32:07.620022+0000","last_clean":"2024-03-16T19:32:07.620022+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:07.620022+0000","last_undegraded":"2024-03-16T19:32:07.620022+0000","last_fullsized":"2024-03-16T19:32:07.620022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084106+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:08.084106+0000","last_peered":"2024-03-16T19:32:08.084106+0000","last_clean":"2024-03-16T19:32:08.084106+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:08.084106+0000","last_undegraded":"2024-03-16T19:32:08.084106+0000","last_fullsized":"2024-03-16T19:32:08.084106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081285+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:07.081285+0000","last_peered":"2024-03-16T19:32:07.081285+0000","last_clean":"2024-03-16T19:32:07.081285+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:07.081285+0000","last_undegraded":"2024-03-16T19:32:07.081285+0000","last_fullsized":"2024-03-16T19:32:07.081285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083913+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:08.083913+0000","last_peered":"2024-03-16T19:32:08.083913+0000","last_clean":"2024-03-16T19:32:08.083913+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:08.083913+0000","last_undegraded":"2024-03-16T19:32:08.083913+0000","last_fullsized":"2024-03-16T19:32:08.083913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":182,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.081056+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:07.081056+0000","last_peered":"2024-03-16T19:32:07.081056+0000","last_clean":"2024-03-16T19:32:07.081056+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:07.081056+0000","last_undegraded":"2024-03-16T19:32:07.081056+0000","last_fullsized":"2024-03-16T19:32:07.081056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":162,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077568+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:07.077568+0000","last_peered":"2024-03-16T19:32:07.077568+0000","last_clean":"2024-03-16T19:32:07.077568+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:07.077568+0000","last_undegraded":"2024-03-16T19:32:07.077568+0000","last_fullsized":"2024-03-16T19:32:07.077568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:19.317523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088482+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088482+0000","last_peered":"2024-03-16T19:32:08.088482+0000","last_clean":"2024-03-16T19:32:08.088482+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088482+0000","last_undegraded":"2024-03-16T19:32:08.088482+0000","last_fullsized":"2024-03-16T19:32:08.088482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:58:01.695714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086003+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:08.086003+0000","last_peered":"2024-03-16T19:32:08.086003+0000","last_clean":"2024-03-16T19:32:08.086003+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:08.086003+0000","last_undegraded":"2024-03-16T19:32:08.086003+0000","last_fullsized":"2024-03-16T19:32:08.086003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:17.040490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-16T19:32:07.077402+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:07.077402+0000","last_peered":"2024-03-16T19:32:07.077402+0000","last_clean":"2024-03-16T19:32:07.077402+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:07.077402+0000","last_undegraded":"2024-03-16T19:32:07.077402+0000","last_fullsized":"2024-03-16T19:32:07.077402+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:28:10.116570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40494,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":80988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28824,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343016,"statfs":{"total":96636764160,"available":96607248384,"internally_reserved":0,"allocated":1847296,"data_stored":1180033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343552,"statfs":{"total":96636764160,"available":96607797248,"internally_reserved":0,"allocated":1298432,"data_stored":628375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1253376,"data_stored":589173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1290240,"data_stored":622664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122054,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29384,"kb_used_data":1212,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342456,"statfs":{"total":96636764160,"available":96606674944,"internally_reserved":0,"allocated":1241088,"data_stored":583372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28240,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343600,"statfs":{"total":96636764160,"available":96607846400,"internally_reserved":0,"allocated":1245184,"data_stored":583128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28832,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343008,"statfs":{"total":96636764160,"available":96607240192,"internally_reserved":0,"allocated":1855488,"data_stored":1180089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:14.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-16T19:32:14.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.220+0000 7fee51b81640 1 Processor -- start 2024-03-16T19:32:14.201 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.221+0000 7fee51b81640 1 -- start start 2024-03-16T19:32:14.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.221+0000 7fee51b81640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fee4c06bef0 0x7fee4c154660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.221+0000 7fee51b81640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.221+0000 7fee51b81640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c0cc720 con 0x7fee4c06bb10 2024-03-16T19:32:14.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.221+0000 7fee51b81640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c1550e0 con 0x7fee4c06bef0 2024-03-16T19:32:14.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.221+0000 7fee51b81640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0cc9c0 con 0x7fee4c0cb560 2024-03-16T19:32:14.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4b7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fee4c06bb10 0x7fee4c0cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:53266/0 (socket says 172.21.15.172:53266) 2024-03-16T19:32:14.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4b7fe640 1 -- 172.21.15.172:0/3542792456 learned_addr learned my addr 172.21.15.172:0/3542792456 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:14.203 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4affd640 1 --2- 172.21.15.172:0/3542792456 >> v2:172.21.15.174:3300/0 conn(0x7fee4c06bef0 0x7fee4c154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4bfff640 1 --2- 172.21.15.172:0/3542792456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4affd640 1 -- 172.21.15.172:0/3542792456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 msgr2=0x7fee4c154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4affd640 1 --2- 172.21.15.172:0/3542792456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4affd640 1 -- 172.21.15.172:0/3542792456 >> v1:172.21.15.172:6789/0 conn(0x7fee4c06bb10 legacy=0x7fee4c0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4a7fc640 1 -- 172.21.15.172:0/3542792456 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042505087 0 0) 0x7fee4c0cc720 con 0x7fee4c06bb10 2024-03-16T19:32:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4affd640 1 -- 172.21.15.172:0/3542792456 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c15de30 con 0x7fee4c06bef0 2024-03-16T19:32:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.222+0000 7fee4bfff640 1 --2- 172.21.15.172:0/3542792456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.223+0000 7fee4affd640 1 --2- 172.21.15.172:0/3542792456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bef0 0x7fee4c154660 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fee400097a0 tx=0x7fee4002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9939942541766c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.223+0000 7fee4a7fc640 1 -- 172.21.15.172:0/3542792456 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40030610 con 0x7fee4c06bef0 2024-03-16T19:32:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.223+0000 7fee4a7fc640 1 -- 172.21.15.172:0/3542792456 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee40030770 con 0x7fee4c06bef0 2024-03-16T19:32:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.223+0000 7fee4a7fc640 1 -- 172.21.15.172:0/3542792456 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40030b00 con 0x7fee4c06bef0 2024-03-16T19:32:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.224+0000 7fee51b81640 1 -- 172.21.15.172:0/3542792456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bef0 msgr2=0x7fee4c154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.224+0000 7fee51b81640 1 --2- 172.21.15.172:0/3542792456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bef0 0x7fee4c154660 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fee400097a0 tx=0x7fee4002b960 comp rx=0 tx=0).stop 2024-03-16T19:32:14.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.224+0000 7fee51b81640 1 -- 172.21.15.172:0/3542792456 shutdown_connections 2024-03-16T19:32:14.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.224+0000 7fee51b81640 1 --2- 172.21.15.172:0/3542792456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.224+0000 7fee51b81640 1 --2- 172.21.15.172:0/3542792456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bef0 0x7fee4c154660 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.224+0000 7fee51b81640 1 -- 172.21.15.172:0/3542792456 >> 172.21.15.172:0/3542792456 conn(0x7fee4c067250 msgr2=0x7fee4c0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:14.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.224+0000 7fee51b81640 1 -- 172.21.15.172:0/3542792456 shutdown_connections 2024-03-16T19:32:14.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.224+0000 7fee51b81640 1 -- 172.21.15.172:0/3542792456 wait complete. 2024-03-16T19:32:14.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.225+0000 7fee51b81640 1 Processor -- start 2024-03-16T19:32:14.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.225+0000 7fee51b81640 1 -- start start 2024-03-16T19:32:14.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee51b81640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bb10 0x7fee4c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee51b81640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4c06bef0 0x7fee4c1e6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee51b81640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c1e6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee51b81640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c1f07b0 con 0x7fee4c06bef0 2024-03-16T19:32:14.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee51b81640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c1f0a70 con 0x7fee4c06bb10 2024-03-16T19:32:14.208 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee51b81640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c1f0d50 con 0x7fee4c0cb560 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee4affd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4c06bef0 0x7fee4c1e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee4affd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4c06bef0 0x7fee4c1e6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57668/0 (socket says 172.21.15.172:57668) 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee4affd640 1 -- 172.21.15.172:0/1973960767 learned_addr learned my addr 172.21.15.172:0/1973960767 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee4b7fe640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bb10 0x7fee4c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee4bfff640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c1e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.226+0000 7fee4affd640 1 -- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 msgr2=0x7fee4c1e6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee4affd640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c1e6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee4affd640 1 -- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bb10 msgr2=0x7fee4c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee4affd640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bb10 0x7fee4c0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee4affd640 1 -- 172.21.15.172:0/1973960767 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44009a70 con 0x7fee4c06bef0 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee4bfff640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c1e6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee4b7fe640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bb10 0x7fee4c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee4affd640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4c06bef0 0x7fee4c1e6130 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fee4002bf70 tx=0x7fee40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee33fff640 1 -- 172.21.15.172:0/1973960767 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4003e040 con 0x7fee4c06bef0 2024-03-16T19:32:14.209 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee33fff640 1 -- 172.21.15.172:0/1973960767 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee40039cb0 con 0x7fee4c06bef0 2024-03-16T19:32:14.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee40009400 con 0x7fee4c06bef0 2024-03-16T19:32:14.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee33fff640 1 -- 172.21.15.172:0/1973960767 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40044630 con 0x7fee4c06bef0 2024-03-16T19:32:14.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.227+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c1f1310 con 0x7fee4c06bef0 2024-03-16T19:32:14.210 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.228+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c0681c0 con 0x7fee4c06bef0 2024-03-16T19:32:14.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.228+0000 7fee33fff640 1 -- 172.21.15.172:0/1973960767 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fee4004d3b0 con 0x7fee4c06bef0 2024-03-16T19:32:14.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.229+0000 7fee33fff640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee3806b3b0 0x7fee3806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.229+0000 7fee33fff640 1 -- 172.21.15.172:0/1973960767 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fee400abf60 con 0x7fee4c06bef0 2024-03-16T19:32:14.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.229+0000 7fee4b7fe640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee3806b3b0 0x7fee3806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.211 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.229+0000 7fee4b7fe640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee3806b3b0 0x7fee3806d870 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000f60 tx=0x7fee3c007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:14.215 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.234+0000 7fee33fff640 1 -- 172.21.15.172:0/1973960767 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee40076bd0 con 0x7fee4c06bef0 2024-03-16T19:32:14.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.672+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fee4c0c6cf0 con 0x7fee4c06bef0 2024-03-16T19:32:14.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.673+0000 7fee33fff640 1 -- 172.21.15.172:0/1973960767 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+949 (secure 0 0 0) 0x7fee40006e10 con 0x7fee4c06bef0 2024-03-16T19:32:14.654 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:14.654 INFO:teuthology.orchestra.run.smithi172.stdout:{"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":192,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":106,"num_osds":8,"num_up_osds":8,"osd_up_since":1710617508,"num_in_osds":8,"osd_in_since":1710617508,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":25,"data_bytes":630862,"bytes_used":233963520,"bytes_avail":772860149760,"bytes_total":773094113280},"fsmap":{"epoch":8,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"f","status":"up:active","gid":4297}],"up:standby":5},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-16T19:30:36.213107+0000","services":{}},"progress_events":{}} 2024-03-16T19:32:14.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee3806b3b0 msgr2=0x7fee3806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee3806b3b0 0x7fee3806d870 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000f60 tx=0x7fee3c007d50 comp rx=0 tx=0).stop 2024-03-16T19:32:14.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4c06bef0 msgr2=0x7fee4c1e6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4c06bef0 0x7fee4c1e6130 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fee4002bf70 tx=0x7fee40004730 comp rx=0 tx=0).stop 2024-03-16T19:32:14.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 shutdown_connections 2024-03-16T19:32:14.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee3806b3b0 0x7fee3806d870 secure :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000f60 tx=0x7fee3c007d50 comp rx=0 tx=0).stop 2024-03-16T19:32:14.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee4c0cb560 0x7fee4c1e6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4c06bef0 0x7fee4c1e6130 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 --2- 172.21.15.172:0/1973960767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c06bb10 0x7fee4c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 >> 172.21.15.172:0/1973960767 conn(0x7fee4c067250 msgr2=0x7fee4c151660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:14.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 shutdown_connections 2024-03-16T19:32:14.658 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.676+0000 7fee51b81640 1 -- 172.21.15.172:0/1973960767 wait complete. 2024-03-16T19:32:14.671 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-16T19:32:14.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:14.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.837+0000 7fb890fd7640 1 Processor -- start 2024-03-16T19:32:14.818 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.837+0000 7fb890fd7640 1 -- start start 2024-03-16T19:32:14.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.838+0000 7fb890fd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bb10 0x7fb88c0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.838+0000 7fb890fd7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88c0cc720 con 0x7fb88c06bef0 2024-03-16T19:32:14.819 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.838+0000 7fb890fd7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88c0cc9c0 con 0x7fb88c0cc260 2024-03-16T19:32:14.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.838+0000 7fb890fd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c15de30 con 0x7fb88c06bb10 2024-03-16T19:32:14.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.838+0000 7fb88a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bb10 0x7fb88c0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.838+0000 7fb88a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bb10 0x7fb88c0cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:47876/0 (socket says 172.21.15.172:47876) 2024-03-16T19:32:14.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.838+0000 7fb88a59c640 1 -- 172.21.15.172:0/760197394 learned_addr learned my addr 172.21.15.172:0/760197394 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:14.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.838+0000 7fb889d9b640 1 -- 172.21.15.172:0/760197394 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70930181 0 0) 0x7fb88c0cc9c0 con 0x7fb88c0cc260 2024-03-16T19:32:14.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.839+0000 7fb889d9b640 1 -- 172.21.15.172:0/760197394 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb878003520 con 0x7fb88c0cc260 2024-03-16T19:32:14.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.839+0000 7fb88a59c640 1 -- 172.21.15.172:0/760197394 >> v1:172.21.15.174:6789/0 conn(0x7fb88c0cc260 legacy=0x7fb88c1651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.839+0000 7fb88a59c640 1 -- 172.21.15.172:0/760197394 >> v1:172.21.15.172:6789/0 conn(0x7fb88c06bef0 legacy=0x7fb88c154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.820 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.839+0000 7fb88a59c640 1 -- 172.21.15.172:0/760197394 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c15e110 con 0x7fb88c06bb10 2024-03-16T19:32:14.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.840+0000 7fb88a59c640 1 --2- 172.21.15.172:0/760197394 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bb10 0x7fb88c0cae50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0098a0 tx=0x7fb87c02baa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40fda032124b2db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:14.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.840+0000 7fb889d9b640 1 -- 172.21.15.172:0/760197394 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c02d860 con 0x7fb88c06bb10 2024-03-16T19:32:14.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.840+0000 7fb889d9b640 1 -- 172.21.15.172:0/760197394 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c02d9c0 con 0x7fb88c06bb10 2024-03-16T19:32:14.821 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.840+0000 7fb889d9b640 1 -- 172.21.15.172:0/760197394 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c02dd50 con 0x7fb88c06bb10 2024-03-16T19:32:14.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.841+0000 7fb890fd7640 1 -- 172.21.15.172:0/760197394 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bb10 msgr2=0x7fb88c0cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.841+0000 7fb890fd7640 1 --2- 172.21.15.172:0/760197394 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bb10 0x7fb88c0cae50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0098a0 tx=0x7fb87c02baa0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.841+0000 7fb890fd7640 1 -- 172.21.15.172:0/760197394 shutdown_connections 2024-03-16T19:32:14.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.841+0000 7fb890fd7640 1 --2- 172.21.15.172:0/760197394 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bb10 0x7fb88c0cae50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.841+0000 7fb890fd7640 1 -- 172.21.15.172:0/760197394 >> 172.21.15.172:0/760197394 conn(0x7fb88c067250 msgr2=0x7fb88c0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:14.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.841+0000 7fb890fd7640 1 -- 172.21.15.172:0/760197394 shutdown_connections 2024-03-16T19:32:14.822 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.841+0000 7fb890fd7640 1 -- 172.21.15.172:0/760197394 wait complete. 2024-03-16T19:32:14.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.843+0000 7fb890fd7640 1 Processor -- start 2024-03-16T19:32:14.824 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.843+0000 7fb890fd7640 1 -- start start 2024-03-16T19:32:14.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.844+0000 7fb890fd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c06bb10 0x7fb88c14f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.844+0000 7fb890fd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bef0 0x7fb88c1e52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.844+0000 7fb890fd7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb88c0cc260 0x7fb88c15ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.826 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.844+0000 7fb881d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bef0 0x7fb88c1e52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.844+0000 7fb88ad9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb88c0cc260 0x7fb88c15ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c06bb10 0x7fb88c14f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88ad9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb88c0cc260 0x7fb88c15ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57672/0 (socket says 172.21.15.172:57672) 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb881d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bef0 0x7fb88c1e52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:47880/0 (socket says 172.21.15.172:47880) 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88ad9d640 1 -- 172.21.15.172:0/1102810337 learned_addr learned my addr 172.21.15.172:0/1102810337 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c06bb10 0x7fb88c14f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:48526/0 (socket says 172.21.15.172:48526) 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c150ba0 con 0x7fb88c0cc260 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c1e6490 con 0x7fb88c06bb10 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c1e5830 con 0x7fb88c06bef0 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88a59c640 1 -- 172.21.15.172:0/1102810337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bef0 msgr2=0x7fb88c1e52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88a59c640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bef0 0x7fb88c1e52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88a59c640 1 -- 172.21.15.172:0/1102810337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb88c0cc260 msgr2=0x7fb88c15ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88a59c640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb88c0cc260 0x7fb88c15ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:14.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88a59c640 1 -- 172.21.15.172:0/1102810337 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0094d0 con 0x7fb88c06bb10 2024-03-16T19:32:14.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb88a59c640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c06bb10 0x7fb88c14f7d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb87c033000 tx=0x7fb87c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:14.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb8837fe640 1 -- 172.21.15.172:0/1102810337 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c046070 con 0x7fb88c06bb10 2024-03-16T19:32:14.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb8837fe640 1 -- 172.21.15.172:0/1102810337 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c037690 con 0x7fb88c06bb10 2024-03-16T19:32:14.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb8837fe640 1 -- 172.21.15.172:0/1102810337 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c03b040 con 0x7fb88c06bb10 2024-03-16T19:32:14.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88c150db0 con 0x7fb88c06bb10 2024-03-16T19:32:14.828 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.845+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88c14ecf0 con 0x7fb88c06bb10 2024-03-16T19:32:14.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.847+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88c0681c0 con 0x7fb88c06bb10 2024-03-16T19:32:14.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.847+0000 7fb8837fe640 1 -- 172.21.15.172:0/1102810337 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb87c037a40 con 0x7fb88c06bb10 2024-03-16T19:32:14.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.848+0000 7fb8837fe640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3b0 0x7fb86c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:14.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.848+0000 7fb8837fe640 1 -- 172.21.15.172:0/1102810337 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fb87c032070 con 0x7fb88c06bb10 2024-03-16T19:32:14.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.848+0000 7fb881d9b640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3b0 0x7fb86c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:14.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.849+0000 7fb881d9b640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3b0 0x7fb86c06d870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb88c15d6a0 tx=0x7fb874009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:14.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:14.852+0000 7fb8837fe640 1 -- 172.21.15.172:0/1102810337 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb87c076c90 con 0x7fb88c06bb10 2024-03-16T19:32:15.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.173+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb88c15e9b0 con 0x7fb86c06b3b0 2024-03-16T19:32:15.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.184+0000 7fb8837fe640 1 -- 172.21.15.172:0/1102810337 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252351 (secure 0 0 0) 0x7fb8740050d0 con 0x7fb86c06b3b0 2024-03-16T19:32:15.167 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:15.168 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3b0 msgr2=0x7fb86c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3b0 0x7fb86c06d870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb88c15d6a0 tx=0x7fb874009040 comp rx=0 tx=0).stop 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c06bb10 msgr2=0x7fb88c14f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c06bb10 0x7fb88c14f7d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb87c033000 tx=0x7fb87c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 shutdown_connections 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb88c0cc260 0x7fb88c15ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb86c06b3b0 0x7fb86c06d870 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb88c06bef0 0x7fb88c1e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 --2- 172.21.15.172:0/1102810337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb88c06bb10 0x7fb88c14f7d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:15.170 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 >> 172.21.15.172:0/1102810337 conn(0x7fb88c067250 msgr2=0x7fb88c0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:15.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 shutdown_connections 2024-03-16T19:32:15.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:15.188+0000 7fb890fd7640 1 -- 172.21.15.172:0/1102810337 wait complete. 2024-03-16T19:32:15.186 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-03-16T19:32:14.232078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630862,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":245,"num_write_kb":2489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228480,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":217199,"kb_avail":754746240,"statfs":{"total":773094113280,"available":772860149760,"internally_reserved":0,"allocated":11427840,"data_stored":6022908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":222412203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001724"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937971+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:08.937971+0000","last_peered":"2024-03-16T19:32:08.937971+0000","last_clean":"2024-03-16T19:32:08.937971+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:08.937971+0000","last_undegraded":"2024-03-16T19:32:08.937971+0000","last_fullsized":"2024-03-16T19:32:08.937971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094518+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:08.094518+0000","last_peered":"2024-03-16T19:32:08.094518+0000","last_clean":"2024-03-16T19:32:08.094518+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:08.094518+0000","last_undegraded":"2024-03-16T19:32:08.094518+0000","last_fullsized":"2024-03-16T19:32:08.094518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351343+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:08.351343+0000","last_peered":"2024-03-16T19:32:08.351343+0000","last_clean":"2024-03-16T19:32:08.351343+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:08.351343+0000","last_undegraded":"2024-03-16T19:32:08.351343+0000","last_fullsized":"2024-03-16T19:32:08.351343+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:47.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083818+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:08.083818+0000","last_peered":"2024-03-16T19:32:08.083818+0000","last_clean":"2024-03-16T19:32:08.083818+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:08.083818+0000","last_undegraded":"2024-03-16T19:32:08.083818+0000","last_fullsized":"2024-03-16T19:32:08.083818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937685+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:08.937685+0000","last_peered":"2024-03-16T19:32:08.937685+0000","last_clean":"2024-03-16T19:32:08.937685+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:08.937685+0000","last_undegraded":"2024-03-16T19:32:08.937685+0000","last_fullsized":"2024-03-16T19:32:08.937685+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083640+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:08.083640+0000","last_peered":"2024-03-16T19:32:08.083640+0000","last_clean":"2024-03-16T19:32:08.083640+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:08.083640+0000","last_undegraded":"2024-03-16T19:32:08.083640+0000","last_fullsized":"2024-03-16T19:32:08.083640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937556+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:08.937556+0000","last_peered":"2024-03-16T19:32:08.937556+0000","last_clean":"2024-03-16T19:32:08.937556+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:08.937556+0000","last_undegraded":"2024-03-16T19:32:08.937556+0000","last_fullsized":"2024-03-16T19:32:08.937556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086880+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:08.086880+0000","last_peered":"2024-03-16T19:32:08.086880+0000","last_clean":"2024-03-16T19:32:08.086880+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:08.086880+0000","last_undegraded":"2024-03-16T19:32:08.086880+0000","last_fullsized":"2024-03-16T19:32:08.086880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:58:31.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937978+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:08.937978+0000","last_peered":"2024-03-16T19:32:08.937978+0000","last_clean":"2024-03-16T19:32:08.937978+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:08.937978+0000","last_undegraded":"2024-03-16T19:32:08.937978+0000","last_fullsized":"2024-03-16T19:32:08.937978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287814+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:09.287814+0000","last_peered":"2024-03-16T19:32:09.287814+0000","last_clean":"2024-03-16T19:32:09.287814+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:09.287814+0000","last_undegraded":"2024-03-16T19:32:09.287814+0000","last_fullsized":"2024-03-16T19:32:09.287814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287704+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:09.287704+0000","last_peered":"2024-03-16T19:32:09.287704+0000","last_clean":"2024-03-16T19:32:09.287704+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:09.287704+0000","last_undegraded":"2024-03-16T19:32:09.287704+0000","last_fullsized":"2024-03-16T19:32:09.287704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084143+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:08.084143+0000","last_peered":"2024-03-16T19:32:08.084143+0000","last_clean":"2024-03-16T19:32:08.084143+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:08.084143+0000","last_undegraded":"2024-03-16T19:32:08.084143+0000","last_fullsized":"2024-03-16T19:32:08.084143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088886+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:08.088886+0000","last_peered":"2024-03-16T19:32:08.088886+0000","last_clean":"2024-03-16T19:32:08.088886+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:08.088886+0000","last_undegraded":"2024-03-16T19:32:08.088886+0000","last_fullsized":"2024-03-16T19:32:08.088886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:52.102388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"84'3","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350888+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:08.350888+0000","last_peered":"2024-03-16T19:32:08.350888+0000","last_clean":"2024-03-16T19:32:08.350888+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:08.350888+0000","last_undegraded":"2024-03-16T19:32:08.350888+0000","last_fullsized":"2024-03-16T19:32:08.350888+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:39.289238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094126+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:08.094126+0000","last_peered":"2024-03-16T19:32:08.094126+0000","last_clean":"2024-03-16T19:32:08.094126+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:08.094126+0000","last_undegraded":"2024-03-16T19:32:08.094126+0000","last_fullsized":"2024-03-16T19:32:08.094126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083403+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:08.083403+0000","last_peered":"2024-03-16T19:32:08.083403+0000","last_clean":"2024-03-16T19:32:08.083403+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:08.083403+0000","last_undegraded":"2024-03-16T19:32:08.083403+0000","last_fullsized":"2024-03-16T19:32:08.083403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084254+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:08.084254+0000","last_peered":"2024-03-16T19:32:08.084254+0000","last_clean":"2024-03-16T19:32:08.084254+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:08.084254+0000","last_undegraded":"2024-03-16T19:32:08.084254+0000","last_fullsized":"2024-03-16T19:32:08.084254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087017+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:08.087017+0000","last_peered":"2024-03-16T19:32:08.087017+0000","last_clean":"2024-03-16T19:32:08.087017+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:08.087017+0000","last_undegraded":"2024-03-16T19:32:08.087017+0000","last_fullsized":"2024-03-16T19:32:08.087017+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:36:06.175327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937878+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:08.937878+0000","last_peered":"2024-03-16T19:32:08.937878+0000","last_clean":"2024-03-16T19:32:08.937878+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:08.937878+0000","last_undegraded":"2024-03-16T19:32:08.937878+0000","last_fullsized":"2024-03-16T19:32:08.937878+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287690+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:09.287690+0000","last_peered":"2024-03-16T19:32:09.287690+0000","last_clean":"2024-03-16T19:32:09.287690+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:09.287690+0000","last_undegraded":"2024-03-16T19:32:09.287690+0000","last_fullsized":"2024-03-16T19:32:09.287690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351462+0000","last_change":"2024-03-16T19:31:51.807295+0000","last_active":"2024-03-16T19:32:08.351462+0000","last_peered":"2024-03-16T19:32:08.351462+0000","last_clean":"2024-03-16T19:32:08.351462+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:08.351462+0000","last_undegraded":"2024-03-16T19:32:08.351462+0000","last_fullsized":"2024-03-16T19:32:08.351462+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:04.578640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.085957+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:08.085957+0000","last_peered":"2024-03-16T19:32:08.085957+0000","last_clean":"2024-03-16T19:32:08.085957+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:08.085957+0000","last_undegraded":"2024-03-16T19:32:08.085957+0000","last_fullsized":"2024-03-16T19:32:08.085957+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:41:06.377004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088245+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:08.088245+0000","last_peered":"2024-03-16T19:32:08.088245+0000","last_clean":"2024-03-16T19:32:08.088245+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:08.088245+0000","last_undegraded":"2024-03-16T19:32:08.088245+0000","last_fullsized":"2024-03-16T19:32:08.088245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:47:08.251341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083454+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:08.083454+0000","last_peered":"2024-03-16T19:32:08.083454+0000","last_clean":"2024-03-16T19:32:08.083454+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:08.083454+0000","last_undegraded":"2024-03-16T19:32:08.083454+0000","last_fullsized":"2024-03-16T19:32:08.083454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086134+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:08.086134+0000","last_peered":"2024-03-16T19:32:08.086134+0000","last_clean":"2024-03-16T19:32:08.086134+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:08.086134+0000","last_undegraded":"2024-03-16T19:32:08.086134+0000","last_fullsized":"2024-03-16T19:32:08.086134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:10.818175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287911+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:09.287911+0000","last_peered":"2024-03-16T19:32:09.287911+0000","last_clean":"2024-03-16T19:32:09.287911+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:09.287911+0000","last_undegraded":"2024-03-16T19:32:09.287911+0000","last_fullsized":"2024-03-16T19:32:09.287911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":187,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093433+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:08.093433+0000","last_peered":"2024-03-16T19:32:08.093433+0000","last_clean":"2024-03-16T19:32:08.093433+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:08.093433+0000","last_undegraded":"2024-03-16T19:32:08.093433+0000","last_fullsized":"2024-03-16T19:32:08.093433+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087892+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:08.087892+0000","last_peered":"2024-03-16T19:32:08.087892+0000","last_clean":"2024-03-16T19:32:08.087892+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:08.087892+0000","last_undegraded":"2024-03-16T19:32:08.087892+0000","last_fullsized":"2024-03-16T19:32:08.087892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:12:08.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350890+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:08.350890+0000","last_peered":"2024-03-16T19:32:08.350890+0000","last_clean":"2024-03-16T19:32:08.350890+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:08.350890+0000","last_undegraded":"2024-03-16T19:32:08.350890+0000","last_fullsized":"2024-03-16T19:32:08.350890+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:12.646493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083749+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:08.083749+0000","last_peered":"2024-03-16T19:32:08.083749+0000","last_clean":"2024-03-16T19:32:08.083749+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:08.083749+0000","last_undegraded":"2024-03-16T19:32:08.083749+0000","last_fullsized":"2024-03-16T19:32:08.083749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087224+0000","last_change":"2024-03-16T19:31:49.974297+0000","last_active":"2024-03-16T19:32:08.087224+0000","last_peered":"2024-03-16T19:32:08.087224+0000","last_clean":"2024-03-16T19:32:08.087224+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:08.087224+0000","last_undegraded":"2024-03-16T19:32:08.087224+0000","last_fullsized":"2024-03-16T19:32:08.087224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:51:57.535372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287591+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:09.287591+0000","last_peered":"2024-03-16T19:32:09.287591+0000","last_clean":"2024-03-16T19:32:09.287591+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:09.287591+0000","last_undegraded":"2024-03-16T19:32:09.287591+0000","last_fullsized":"2024-03-16T19:32:09.287591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084174+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:32:08.084174+0000","last_peered":"2024-03-16T19:32:08.084174+0000","last_clean":"2024-03-16T19:32:08.084174+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:08.084174+0000","last_undegraded":"2024-03-16T19:32:08.084174+0000","last_fullsized":"2024-03-16T19:32:08.084174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089011+0000","last_change":"2024-03-16T19:31:26.808032+0000","last_active":"2024-03-16T19:32:08.089011+0000","last_peered":"2024-03-16T19:32:08.089011+0000","last_clean":"2024-03-16T19:32:08.089011+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:08.089011+0000","last_undegraded":"2024-03-16T19:32:08.089011+0000","last_fullsized":"2024-03-16T19:32:08.089011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:15.218778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351569+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:08.351569+0000","last_peered":"2024-03-16T19:32:08.351569+0000","last_clean":"2024-03-16T19:32:08.351569+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:08.351569+0000","last_undegraded":"2024-03-16T19:32:08.351569+0000","last_fullsized":"2024-03-16T19:32:08.351569+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:16:31.606469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287611+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:09.287611+0000","last_peered":"2024-03-16T19:32:09.287611+0000","last_clean":"2024-03-16T19:32:09.287611+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:09.287611+0000","last_undegraded":"2024-03-16T19:32:09.287611+0000","last_fullsized":"2024-03-16T19:32:09.287611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086078+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086078+0000","last_peered":"2024-03-16T19:32:08.086078+0000","last_clean":"2024-03-16T19:32:08.086078+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086078+0000","last_undegraded":"2024-03-16T19:32:08.086078+0000","last_fullsized":"2024-03-16T19:32:08.086078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:23:19.824904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086998+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:08.086998+0000","last_peered":"2024-03-16T19:32:08.086998+0000","last_clean":"2024-03-16T19:32:08.086998+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:08.086998+0000","last_undegraded":"2024-03-16T19:32:08.086998+0000","last_fullsized":"2024-03-16T19:32:08.086998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:45.782085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094004+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:08.094004+0000","last_peered":"2024-03-16T19:32:08.094004+0000","last_clean":"2024-03-16T19:32:08.094004+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:08.094004+0000","last_undegraded":"2024-03-16T19:32:08.094004+0000","last_fullsized":"2024-03-16T19:32:08.094004+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287793+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:09.287793+0000","last_peered":"2024-03-16T19:32:09.287793+0000","last_clean":"2024-03-16T19:32:09.287793+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:09.287793+0000","last_undegraded":"2024-03-16T19:32:09.287793+0000","last_fullsized":"2024-03-16T19:32:09.287793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083754+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:08.083754+0000","last_peered":"2024-03-16T19:32:08.083754+0000","last_clean":"2024-03-16T19:32:08.083754+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:08.083754+0000","last_undegraded":"2024-03-16T19:32:08.083754+0000","last_fullsized":"2024-03-16T19:32:08.083754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351532+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:08.351532+0000","last_peered":"2024-03-16T19:32:08.351532+0000","last_clean":"2024-03-16T19:32:08.351532+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:08.351532+0000","last_undegraded":"2024-03-16T19:32:08.351532+0000","last_fullsized":"2024-03-16T19:32:08.351532+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:16.418962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093565+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:08.093565+0000","last_peered":"2024-03-16T19:32:08.093565+0000","last_clean":"2024-03-16T19:32:08.093565+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:08.093565+0000","last_undegraded":"2024-03-16T19:32:08.093565+0000","last_fullsized":"2024-03-16T19:32:08.093565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938817+0000","last_change":"2024-03-16T19:32:08.938817+0000","last_active":"2024-03-16T19:32:08.938817+0000","last_peered":"2024-03-16T19:32:08.938817+0000","last_clean":"2024-03-16T19:32:08.938817+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:08.938817+0000","last_undegraded":"2024-03-16T19:32:08.938817+0000","last_fullsized":"2024-03-16T19:32:08.938817+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:39:32.174855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"86'23","reported_seq":233,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287734+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:09.287734+0000","last_peered":"2024-03-16T19:32:09.287734+0000","last_clean":"2024-03-16T19:32:09.287734+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:09.287734+0000","last_undegraded":"2024-03-16T19:32:09.287734+0000","last_fullsized":"2024-03-16T19:32:09.287734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39292,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937730+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:08.937730+0000","last_peered":"2024-03-16T19:32:08.937730+0000","last_clean":"2024-03-16T19:32:08.937730+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:08.937730+0000","last_undegraded":"2024-03-16T19:32:08.937730+0000","last_fullsized":"2024-03-16T19:32:08.937730+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351564+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:08.351564+0000","last_peered":"2024-03-16T19:32:08.351564+0000","last_clean":"2024-03-16T19:32:08.351564+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:08.351564+0000","last_undegraded":"2024-03-16T19:32:08.351564+0000","last_fullsized":"2024-03-16T19:32:08.351564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:59.818393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351224+0000","last_change":"2024-03-16T19:31:51.807895+0000","last_active":"2024-03-16T19:32:08.351224+0000","last_peered":"2024-03-16T19:32:08.351224+0000","last_clean":"2024-03-16T19:32:08.351224+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:08.351224+0000","last_undegraded":"2024-03-16T19:32:08.351224+0000","last_fullsized":"2024-03-16T19:32:08.351224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:25.988237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937695+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:08.937695+0000","last_peered":"2024-03-16T19:32:08.937695+0000","last_clean":"2024-03-16T19:32:08.937695+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:08.937695+0000","last_undegraded":"2024-03-16T19:32:08.937695+0000","last_fullsized":"2024-03-16T19:32:08.937695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287610+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:09.287610+0000","last_peered":"2024-03-16T19:32:09.287610+0000","last_clean":"2024-03-16T19:32:09.287610+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:09.287610+0000","last_undegraded":"2024-03-16T19:32:09.287610+0000","last_fullsized":"2024-03-16T19:32:09.287610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287595+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:09.287595+0000","last_peered":"2024-03-16T19:32:09.287595+0000","last_clean":"2024-03-16T19:32:09.287595+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:09.287595+0000","last_undegraded":"2024-03-16T19:32:09.287595+0000","last_fullsized":"2024-03-16T19:32:09.287595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087374+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:08.087374+0000","last_peered":"2024-03-16T19:32:08.087374+0000","last_clean":"2024-03-16T19:32:08.087374+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:08.087374+0000","last_undegraded":"2024-03-16T19:32:08.087374+0000","last_fullsized":"2024-03-16T19:32:08.087374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:33.634146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937562+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:08.937562+0000","last_peered":"2024-03-16T19:32:08.937562+0000","last_clean":"2024-03-16T19:32:08.937562+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:08.937562+0000","last_undegraded":"2024-03-16T19:32:08.937562+0000","last_fullsized":"2024-03-16T19:32:08.937562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084319+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084319+0000","last_peered":"2024-03-16T19:32:08.084319+0000","last_clean":"2024-03-16T19:32:08.084319+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084319+0000","last_undegraded":"2024-03-16T19:32:08.084319+0000","last_fullsized":"2024-03-16T19:32:08.084319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:53.021797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937869+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:08.937869+0000","last_peered":"2024-03-16T19:32:08.937869+0000","last_clean":"2024-03-16T19:32:08.937869+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:08.937869+0000","last_undegraded":"2024-03-16T19:32:08.937869+0000","last_fullsized":"2024-03-16T19:32:08.937869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":55,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086917+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:08.086917+0000","last_peered":"2024-03-16T19:32:08.086917+0000","last_clean":"2024-03-16T19:32:08.086917+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:08.086917+0000","last_undegraded":"2024-03-16T19:32:08.086917+0000","last_fullsized":"2024-03-16T19:32:08.086917+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:07:39.853290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938036+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:08.938036+0000","last_peered":"2024-03-16T19:32:08.938036+0000","last_clean":"2024-03-16T19:32:08.938036+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:08.938036+0000","last_undegraded":"2024-03-16T19:32:08.938036+0000","last_fullsized":"2024-03-16T19:32:08.938036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093703+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:08.093703+0000","last_peered":"2024-03-16T19:32:08.093703+0000","last_clean":"2024-03-16T19:32:08.093703+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:08.093703+0000","last_undegraded":"2024-03-16T19:32:08.093703+0000","last_fullsized":"2024-03-16T19:32:08.093703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084002+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:08.084002+0000","last_peered":"2024-03-16T19:32:08.084002+0000","last_clean":"2024-03-16T19:32:08.084002+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:08.084002+0000","last_undegraded":"2024-03-16T19:32:08.084002+0000","last_fullsized":"2024-03-16T19:32:08.084002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086673+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:08.086673+0000","last_peered":"2024-03-16T19:32:08.086673+0000","last_clean":"2024-03-16T19:32:08.086673+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:08.086673+0000","last_undegraded":"2024-03-16T19:32:08.086673+0000","last_fullsized":"2024-03-16T19:32:08.086673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:59:55.019036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":148,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937729+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:08.937729+0000","last_peered":"2024-03-16T19:32:08.937729+0000","last_clean":"2024-03-16T19:32:08.937729+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:08.937729+0000","last_undegraded":"2024-03-16T19:32:08.937729+0000","last_fullsized":"2024-03-16T19:32:08.937729+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":363,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937413+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:08.937413+0000","last_peered":"2024-03-16T19:32:08.937413+0000","last_clean":"2024-03-16T19:32:08.937413+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:08.937413+0000","last_undegraded":"2024-03-16T19:32:08.937413+0000","last_fullsized":"2024-03-16T19:32:08.937413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087541+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:08.087541+0000","last_peered":"2024-03-16T19:32:08.087541+0000","last_clean":"2024-03-16T19:32:08.087541+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:08.087541+0000","last_undegraded":"2024-03-16T19:32:08.087541+0000","last_fullsized":"2024-03-16T19:32:08.087541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:16:54.768710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937520+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:08.937520+0000","last_peered":"2024-03-16T19:32:08.937520+0000","last_clean":"2024-03-16T19:32:08.937520+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:08.937520+0000","last_undegraded":"2024-03-16T19:32:08.937520+0000","last_fullsized":"2024-03-16T19:32:08.937520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351264+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:08.351264+0000","last_peered":"2024-03-16T19:32:08.351264+0000","last_clean":"2024-03-16T19:32:08.351264+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:08.351264+0000","last_undegraded":"2024-03-16T19:32:08.351264+0000","last_fullsized":"2024-03-16T19:32:08.351264+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:44.308565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087130+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:08.087130+0000","last_peered":"2024-03-16T19:32:08.087130+0000","last_clean":"2024-03-16T19:32:08.087130+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:08.087130+0000","last_undegraded":"2024-03-16T19:32:08.087130+0000","last_fullsized":"2024-03-16T19:32:08.087130+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:20:21.014268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.096035+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:08.096035+0000","last_peered":"2024-03-16T19:32:08.096035+0000","last_clean":"2024-03-16T19:32:08.096035+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:08.096035+0000","last_undegraded":"2024-03-16T19:32:08.096035+0000","last_fullsized":"2024-03-16T19:32:08.096035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084314+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:08.084314+0000","last_peered":"2024-03-16T19:32:08.084314+0000","last_clean":"2024-03-16T19:32:08.084314+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:08.084314+0000","last_undegraded":"2024-03-16T19:32:08.084314+0000","last_fullsized":"2024-03-16T19:32:08.084314+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:01.856961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093299+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:08.093299+0000","last_peered":"2024-03-16T19:32:08.093299+0000","last_clean":"2024-03-16T19:32:08.093299+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:08.093299+0000","last_undegraded":"2024-03-16T19:32:08.093299+0000","last_fullsized":"2024-03-16T19:32:08.093299+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084193+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:08.084193+0000","last_peered":"2024-03-16T19:32:08.084193+0000","last_clean":"2024-03-16T19:32:08.084193+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:08.084193+0000","last_undegraded":"2024-03-16T19:32:08.084193+0000","last_fullsized":"2024-03-16T19:32:08.084193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937906+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:08.937906+0000","last_peered":"2024-03-16T19:32:08.937906+0000","last_clean":"2024-03-16T19:32:08.937906+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:08.937906+0000","last_undegraded":"2024-03-16T19:32:08.937906+0000","last_fullsized":"2024-03-16T19:32:08.937906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093171+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:08.093171+0000","last_peered":"2024-03-16T19:32:08.093171+0000","last_clean":"2024-03-16T19:32:08.093171+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:08.093171+0000","last_undegraded":"2024-03-16T19:32:08.093171+0000","last_fullsized":"2024-03-16T19:32:08.093171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'11","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094654+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:08.094654+0000","last_peered":"2024-03-16T19:32:08.094654+0000","last_clean":"2024-03-16T19:32:08.094654+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:08.094654+0000","last_undegraded":"2024-03-16T19:32:08.094654+0000","last_fullsized":"2024-03-16T19:32:08.094654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088741+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:08.088741+0000","last_peered":"2024-03-16T19:32:08.088741+0000","last_clean":"2024-03-16T19:32:08.088741+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:08.088741+0000","last_undegraded":"2024-03-16T19:32:08.088741+0000","last_fullsized":"2024-03-16T19:32:08.088741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:18.996236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287797+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:09.287797+0000","last_peered":"2024-03-16T19:32:09.287797+0000","last_clean":"2024-03-16T19:32:09.287797+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:09.287797+0000","last_undegraded":"2024-03-16T19:32:09.287797+0000","last_fullsized":"2024-03-16T19:32:09.287797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350908+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:08.350908+0000","last_peered":"2024-03-16T19:32:08.350908+0000","last_clean":"2024-03-16T19:32:08.350908+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:08.350908+0000","last_undegraded":"2024-03-16T19:32:08.350908+0000","last_fullsized":"2024-03-16T19:32:08.350908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:02.024100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937470+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:08.937470+0000","last_peered":"2024-03-16T19:32:08.937470+0000","last_clean":"2024-03-16T19:32:08.937470+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:08.937470+0000","last_undegraded":"2024-03-16T19:32:08.937470+0000","last_fullsized":"2024-03-16T19:32:08.937470+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083795+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:08.083795+0000","last_peered":"2024-03-16T19:32:08.083795+0000","last_clean":"2024-03-16T19:32:08.083795+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:08.083795+0000","last_undegraded":"2024-03-16T19:32:08.083795+0000","last_fullsized":"2024-03-16T19:32:08.083795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:46:49.628764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094246+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:08.094246+0000","last_peered":"2024-03-16T19:32:08.094246+0000","last_clean":"2024-03-16T19:32:08.094246+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:08.094246+0000","last_undegraded":"2024-03-16T19:32:08.094246+0000","last_fullsized":"2024-03-16T19:32:08.094246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.102711+0000","last_change":"2024-03-16T19:32:08.101455+0000","last_active":"2024-03-16T19:32:08.102711+0000","last_peered":"2024-03-16T19:32:08.102711+0000","last_clean":"2024-03-16T19:32:08.102711+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:32:08.102711+0000","last_undegraded":"2024-03-16T19:32:08.102711+0000","last_fullsized":"2024-03-16T19:32:08.102711+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:20:10.984083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.095090+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:08.095090+0000","last_peered":"2024-03-16T19:32:08.095090+0000","last_clean":"2024-03-16T19:32:08.095090+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:08.095090+0000","last_undegraded":"2024-03-16T19:32:08.095090+0000","last_fullsized":"2024-03-16T19:32:08.095090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089208+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:08.089208+0000","last_peered":"2024-03-16T19:32:08.089208+0000","last_clean":"2024-03-16T19:32:08.089208+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:08.089208+0000","last_undegraded":"2024-03-16T19:32:08.089208+0000","last_fullsized":"2024-03-16T19:32:08.089208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:03.618618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937781+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:08.937781+0000","last_peered":"2024-03-16T19:32:08.937781+0000","last_clean":"2024-03-16T19:32:08.937781+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:08.937781+0000","last_undegraded":"2024-03-16T19:32:08.937781+0000","last_fullsized":"2024-03-16T19:32:08.937781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287721+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:09.287721+0000","last_peered":"2024-03-16T19:32:09.287721+0000","last_clean":"2024-03-16T19:32:09.287721+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:09.287721+0000","last_undegraded":"2024-03-16T19:32:09.287721+0000","last_fullsized":"2024-03-16T19:32:09.287721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084106+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:08.084106+0000","last_peered":"2024-03-16T19:32:08.084106+0000","last_clean":"2024-03-16T19:32:08.084106+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:08.084106+0000","last_undegraded":"2024-03-16T19:32:08.084106+0000","last_fullsized":"2024-03-16T19:32:08.084106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937766+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:08.937766+0000","last_peered":"2024-03-16T19:32:08.937766+0000","last_clean":"2024-03-16T19:32:08.937766+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:08.937766+0000","last_undegraded":"2024-03-16T19:32:08.937766+0000","last_fullsized":"2024-03-16T19:32:08.937766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083913+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:08.083913+0000","last_peered":"2024-03-16T19:32:08.083913+0000","last_clean":"2024-03-16T19:32:08.083913+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:08.083913+0000","last_undegraded":"2024-03-16T19:32:08.083913+0000","last_fullsized":"2024-03-16T19:32:08.083913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937841+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:08.937841+0000","last_peered":"2024-03-16T19:32:08.937841+0000","last_clean":"2024-03-16T19:32:08.937841+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:08.937841+0000","last_undegraded":"2024-03-16T19:32:08.937841+0000","last_fullsized":"2024-03-16T19:32:08.937841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":164,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351010+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:08.351010+0000","last_peered":"2024-03-16T19:32:08.351010+0000","last_clean":"2024-03-16T19:32:08.351010+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:08.351010+0000","last_undegraded":"2024-03-16T19:32:08.351010+0000","last_fullsized":"2024-03-16T19:32:08.351010+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:19.317523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088482+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088482+0000","last_peered":"2024-03-16T19:32:08.088482+0000","last_clean":"2024-03-16T19:32:08.088482+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088482+0000","last_undegraded":"2024-03-16T19:32:08.088482+0000","last_fullsized":"2024-03-16T19:32:08.088482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:58:01.695714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086003+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:08.086003+0000","last_peered":"2024-03-16T19:32:08.086003+0000","last_clean":"2024-03-16T19:32:08.086003+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:08.086003+0000","last_undegraded":"2024-03-16T19:32:08.086003+0000","last_fullsized":"2024-03-16T19:32:08.086003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:17.040490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351503+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:08.351503+0000","last_peered":"2024-03-16T19:32:08.351503+0000","last_clean":"2024-03-16T19:32:08.351503+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:08.351503+0000","last_undegraded":"2024-03-16T19:32:08.351503+0000","last_fullsized":"2024-03-16T19:32:08.351503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:28:10.116570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40494,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":80988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28840,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343000,"statfs":{"total":96636764160,"available":96607232000,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1310720,"data_stored":634440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343536,"statfs":{"total":96636764160,"available":96607780864,"internally_reserved":0,"allocated":1314816,"data_stored":634844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122055,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29408,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342432,"statfs":{"total":96636764160,"available":96606650368,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28848,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342992,"statfs":{"total":96636764160,"available":96607223808,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:18.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:18.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.359+0000 7f6a21b08640 1 Processor -- start 2024-03-16T19:32:18.339 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.360+0000 7f6a21b08640 1 -- start start 2024-03-16T19:32:18.340 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a21b08640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6a1c06bb10 0x7f6a1c0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a21b08640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c0cc260 0x7f6a1c0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a21b08640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a1c0cc720 con 0x7f6a1c06bef0 2024-03-16T19:32:18.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a21b08640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c0cc9c0 con 0x7f6a1c06bb10 2024-03-16T19:32:18.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a21b08640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c15de30 con 0x7f6a1c0cc260 2024-03-16T19:32:18.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a1bfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c0cc260 0x7f6a1c0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a1b7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6a1c06bb10 0x7f6a1c0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a1bfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c0cc260 0x7f6a1c0cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:47884/0 (socket says 172.21.15.172:47884) 2024-03-16T19:32:18.341 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a1bfff640 1 -- 172.21.15.172:0/66942518 learned_addr learned my addr 172.21.15.172:0/66942518 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:18.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a1b7fe640 1 -- 172.21.15.172:0/66942518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c0cc260 msgr2=0x7f6a1c0cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:18.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a1b7fe640 1 --2- 172.21.15.172:0/66942518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c0cc260 0x7f6a1c0cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a1b7fe640 1 -- 172.21.15.172:0/66942518 >> v1:172.21.15.174:6789/0 conn(0x7f6a1c06bef0 legacy=0x7f6a1c154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a1b7fe640 1 -- 172.21.15.172:0/66942518 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1c15e110 con 0x7f6a1c06bb10 2024-03-16T19:32:18.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.361+0000 7f6a1a7fc640 1 -- 172.21.15.172:0/66942518 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704805069 0 0) 0x7f6a1c0cc720 con 0x7f6a1c06bef0 2024-03-16T19:32:18.342 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.362+0000 7f6a1b7fe640 1 --2- 172.21.15.172:0/66942518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a1c06bb10 0x7f6a1c0cae50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0097a0 tx=0x7f6a0c02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aab993e050562ee server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:18.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.363+0000 7f6a1a7fc640 1 -- 172.21.15.172:0/66942518 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c030800 con 0x7f6a1c06bb10 2024-03-16T19:32:18.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.363+0000 7f6a1a7fc640 1 -- 172.21.15.172:0/66942518 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c030960 con 0x7f6a1c06bb10 2024-03-16T19:32:18.343 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.363+0000 7f6a1a7fc640 1 -- 172.21.15.172:0/66942518 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c030cf0 con 0x7f6a1c06bb10 2024-03-16T19:32:18.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.363+0000 7f6a21b08640 1 -- 172.21.15.172:0/66942518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a1c06bb10 msgr2=0x7f6a1c0cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.363+0000 7f6a21b08640 1 --2- 172.21.15.172:0/66942518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a1c06bb10 0x7f6a1c0cae50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0097a0 tx=0x7f6a0c02b940 comp rx=0 tx=0).stop 2024-03-16T19:32:18.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.364+0000 7f6a21b08640 1 -- 172.21.15.172:0/66942518 shutdown_connections 2024-03-16T19:32:18.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.364+0000 7f6a21b08640 1 --2- 172.21.15.172:0/66942518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c0cc260 0x7f6a1c0cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.364+0000 7f6a21b08640 1 --2- 172.21.15.172:0/66942518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a1c06bb10 0x7f6a1c0cae50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.364+0000 7f6a21b08640 1 -- 172.21.15.172:0/66942518 >> 172.21.15.172:0/66942518 conn(0x7f6a1c067250 msgr2=0x7f6a1c0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:18.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.364+0000 7f6a21b08640 1 -- 172.21.15.172:0/66942518 shutdown_connections 2024-03-16T19:32:18.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.364+0000 7f6a21b08640 1 -- 172.21.15.172:0/66942518 wait complete. 2024-03-16T19:32:18.344 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.365+0000 7f6a21b08640 1 Processor -- start 2024-03-16T19:32:18.345 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.365+0000 7f6a21b08640 1 -- start start 2024-03-16T19:32:18.347 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.366+0000 7f6a21b08640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a1c06bb10 0x7f6a1c1f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.347 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.366+0000 7f6a21b08640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c06bef0 0x7f6a1c1e7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.366+0000 7f6a21b08640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a1c0cc260 0x7f6a1c1f0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.366+0000 7f6a21b08640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c1f4ac0 con 0x7f6a1c06bb10 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.366+0000 7f6a21b08640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c1426f0 con 0x7f6a1c0cc260 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.366+0000 7f6a21b08640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c0b4dd0 con 0x7f6a1c06bef0 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.366+0000 7f6a1b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a1c06bb10 0x7f6a1c1f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a1c06bb10 0x7f6a1c1f4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57684/0 (socket says 172.21.15.172:57684) 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1b7fe640 1 -- 172.21.15.172:0/2274330403 learned_addr learned my addr 172.21.15.172:0/2274330403 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1affd640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c06bef0 0x7f6a1c1e7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1bfff640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a1c0cc260 0x7f6a1c1f0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1affd640 1 -- 172.21.15.172:0/2274330403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a1c0cc260 msgr2=0x7f6a1c1f0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1affd640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a1c0cc260 0x7f6a1c1f0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1affd640 1 -- 172.21.15.172:0/2274330403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a1c06bb10 msgr2=0x7f6a1c1f4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1affd640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a1c06bb10 0x7f6a1c1f4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1affd640 1 -- 172.21.15.172:0/2274330403 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c009400 con 0x7f6a1c06bef0 2024-03-16T19:32:18.348 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a1affd640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c06bef0 0x7f6a1c1e7090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c06c300 tx=0x7f6a1000c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:18.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a03fff640 1 -- 172.21.15.172:0/2274330403 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a100035f0 con 0x7f6a1c06bef0 2024-03-16T19:32:18.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.367+0000 7f6a03fff640 1 -- 172.21.15.172:0/2274330403 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a10003770 con 0x7f6a1c06bef0 2024-03-16T19:32:18.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.368+0000 7f6a03fff640 1 -- 172.21.15.172:0/2274330403 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a10003b20 con 0x7f6a1c06bef0 2024-03-16T19:32:18.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.368+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1c1e3c80 con 0x7f6a1c06bef0 2024-03-16T19:32:18.349 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.368+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1c1e4240 con 0x7f6a1c06bef0 2024-03-16T19:32:18.350 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.369+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1c0681c0 con 0x7f6a1c06bef0 2024-03-16T19:32:18.350 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.369+0000 7f6a03fff640 1 -- 172.21.15.172:0/2274330403 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6a10003c80 con 0x7f6a1c06bef0 2024-03-16T19:32:18.350 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.369+0000 7f6a03fff640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a0806b3b0 0x7f6a0806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.350 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.369+0000 7f6a03fff640 1 -- 172.21.15.172:0/2274330403 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f6a10089230 con 0x7f6a1c06bef0 2024-03-16T19:32:18.350 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.370+0000 7f6a1b7fe640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a0806b3b0 0x7f6a0806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.350 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.370+0000 7f6a1b7fe640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a0806b3b0 0x7f6a0806d870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0097a0 tx=0x7f6a0c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:18.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.374+0000 7f6a03fff640 1 -- 172.21.15.172:0/2274330403 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a10053e10 con 0x7f6a1c06bef0 2024-03-16T19:32:18.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.685+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a1c152220 con 0x7f6a0806b3b0 2024-03-16T19:32:18.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.695+0000 7f6a03fff640 1 -- 172.21.15.172:0/2274330403 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252328 (secure 0 0 0) 0x7f6a1c152220 con 0x7f6a0806b3b0 2024-03-16T19:32:18.677 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:18.678 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:18.679 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a0806b3b0 msgr2=0x7f6a0806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a0806b3b0 0x7f6a0806d870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0097a0 tx=0x7f6a0c008030 comp rx=0 tx=0).stop 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c06bef0 msgr2=0x7f6a1c1e7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c06bef0 0x7f6a1c1e7090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c06c300 tx=0x7f6a1000c890 comp rx=0 tx=0).stop 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 shutdown_connections 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a1c0cc260 0x7f6a1c1f0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6a0806b3b0 0x7f6a0806d870 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a1c06bef0 0x7f6a1c1e7090 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 --2- 172.21.15.172:0/2274330403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a1c06bb10 0x7f6a1c1f4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.699+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 >> 172.21.15.172:0/2274330403 conn(0x7f6a1c067250 msgr2=0x7f6a1c0cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.700+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 shutdown_connections 2024-03-16T19:32:18.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.700+0000 7f6a21b08640 1 -- 172.21.15.172:0/2274330403 wait complete. 2024-03-16T19:32:18.694 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-03-16T19:32:18.232924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630862,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":245,"num_write_kb":2489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228480,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":217199,"kb_avail":754746240,"statfs":{"total":773094113280,"available":772860149760,"internally_reserved":0,"allocated":11427840,"data_stored":6022908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":222412203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.147918"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937971+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:08.937971+0000","last_peered":"2024-03-16T19:32:08.937971+0000","last_clean":"2024-03-16T19:32:08.937971+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:08.937971+0000","last_undegraded":"2024-03-16T19:32:08.937971+0000","last_fullsized":"2024-03-16T19:32:08.937971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094518+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:08.094518+0000","last_peered":"2024-03-16T19:32:08.094518+0000","last_clean":"2024-03-16T19:32:08.094518+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:08.094518+0000","last_undegraded":"2024-03-16T19:32:08.094518+0000","last_fullsized":"2024-03-16T19:32:08.094518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351343+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:08.351343+0000","last_peered":"2024-03-16T19:32:08.351343+0000","last_clean":"2024-03-16T19:32:08.351343+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:08.351343+0000","last_undegraded":"2024-03-16T19:32:08.351343+0000","last_fullsized":"2024-03-16T19:32:08.351343+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:47.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083818+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:08.083818+0000","last_peered":"2024-03-16T19:32:08.083818+0000","last_clean":"2024-03-16T19:32:08.083818+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:08.083818+0000","last_undegraded":"2024-03-16T19:32:08.083818+0000","last_fullsized":"2024-03-16T19:32:08.083818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937685+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:08.937685+0000","last_peered":"2024-03-16T19:32:08.937685+0000","last_clean":"2024-03-16T19:32:08.937685+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:08.937685+0000","last_undegraded":"2024-03-16T19:32:08.937685+0000","last_fullsized":"2024-03-16T19:32:08.937685+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083640+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:08.083640+0000","last_peered":"2024-03-16T19:32:08.083640+0000","last_clean":"2024-03-16T19:32:08.083640+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:08.083640+0000","last_undegraded":"2024-03-16T19:32:08.083640+0000","last_fullsized":"2024-03-16T19:32:08.083640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937556+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:08.937556+0000","last_peered":"2024-03-16T19:32:08.937556+0000","last_clean":"2024-03-16T19:32:08.937556+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:08.937556+0000","last_undegraded":"2024-03-16T19:32:08.937556+0000","last_fullsized":"2024-03-16T19:32:08.937556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086880+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:08.086880+0000","last_peered":"2024-03-16T19:32:08.086880+0000","last_clean":"2024-03-16T19:32:08.086880+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:08.086880+0000","last_undegraded":"2024-03-16T19:32:08.086880+0000","last_fullsized":"2024-03-16T19:32:08.086880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:58:31.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937978+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:08.937978+0000","last_peered":"2024-03-16T19:32:08.937978+0000","last_clean":"2024-03-16T19:32:08.937978+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:08.937978+0000","last_undegraded":"2024-03-16T19:32:08.937978+0000","last_fullsized":"2024-03-16T19:32:08.937978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287814+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:09.287814+0000","last_peered":"2024-03-16T19:32:09.287814+0000","last_clean":"2024-03-16T19:32:09.287814+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:09.287814+0000","last_undegraded":"2024-03-16T19:32:09.287814+0000","last_fullsized":"2024-03-16T19:32:09.287814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287704+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:09.287704+0000","last_peered":"2024-03-16T19:32:09.287704+0000","last_clean":"2024-03-16T19:32:09.287704+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:09.287704+0000","last_undegraded":"2024-03-16T19:32:09.287704+0000","last_fullsized":"2024-03-16T19:32:09.287704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084143+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:08.084143+0000","last_peered":"2024-03-16T19:32:08.084143+0000","last_clean":"2024-03-16T19:32:08.084143+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:08.084143+0000","last_undegraded":"2024-03-16T19:32:08.084143+0000","last_fullsized":"2024-03-16T19:32:08.084143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088886+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:08.088886+0000","last_peered":"2024-03-16T19:32:08.088886+0000","last_clean":"2024-03-16T19:32:08.088886+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:08.088886+0000","last_undegraded":"2024-03-16T19:32:08.088886+0000","last_fullsized":"2024-03-16T19:32:08.088886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:52.102388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"84'3","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350888+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:08.350888+0000","last_peered":"2024-03-16T19:32:08.350888+0000","last_clean":"2024-03-16T19:32:08.350888+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:08.350888+0000","last_undegraded":"2024-03-16T19:32:08.350888+0000","last_fullsized":"2024-03-16T19:32:08.350888+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:39.289238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094126+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:08.094126+0000","last_peered":"2024-03-16T19:32:08.094126+0000","last_clean":"2024-03-16T19:32:08.094126+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:08.094126+0000","last_undegraded":"2024-03-16T19:32:08.094126+0000","last_fullsized":"2024-03-16T19:32:08.094126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083403+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:08.083403+0000","last_peered":"2024-03-16T19:32:08.083403+0000","last_clean":"2024-03-16T19:32:08.083403+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:08.083403+0000","last_undegraded":"2024-03-16T19:32:08.083403+0000","last_fullsized":"2024-03-16T19:32:08.083403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084254+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:08.084254+0000","last_peered":"2024-03-16T19:32:08.084254+0000","last_clean":"2024-03-16T19:32:08.084254+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:08.084254+0000","last_undegraded":"2024-03-16T19:32:08.084254+0000","last_fullsized":"2024-03-16T19:32:08.084254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087017+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:08.087017+0000","last_peered":"2024-03-16T19:32:08.087017+0000","last_clean":"2024-03-16T19:32:08.087017+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:08.087017+0000","last_undegraded":"2024-03-16T19:32:08.087017+0000","last_fullsized":"2024-03-16T19:32:08.087017+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:36:06.175327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937878+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:08.937878+0000","last_peered":"2024-03-16T19:32:08.937878+0000","last_clean":"2024-03-16T19:32:08.937878+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:08.937878+0000","last_undegraded":"2024-03-16T19:32:08.937878+0000","last_fullsized":"2024-03-16T19:32:08.937878+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287690+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:09.287690+0000","last_peered":"2024-03-16T19:32:09.287690+0000","last_clean":"2024-03-16T19:32:09.287690+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:09.287690+0000","last_undegraded":"2024-03-16T19:32:09.287690+0000","last_fullsized":"2024-03-16T19:32:09.287690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351462+0000","last_change":"2024-03-16T19:31:51.807295+0000","last_active":"2024-03-16T19:32:08.351462+0000","last_peered":"2024-03-16T19:32:08.351462+0000","last_clean":"2024-03-16T19:32:08.351462+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:08.351462+0000","last_undegraded":"2024-03-16T19:32:08.351462+0000","last_fullsized":"2024-03-16T19:32:08.351462+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:04.578640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.085957+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:08.085957+0000","last_peered":"2024-03-16T19:32:08.085957+0000","last_clean":"2024-03-16T19:32:08.085957+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:08.085957+0000","last_undegraded":"2024-03-16T19:32:08.085957+0000","last_fullsized":"2024-03-16T19:32:08.085957+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:41:06.377004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088245+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:08.088245+0000","last_peered":"2024-03-16T19:32:08.088245+0000","last_clean":"2024-03-16T19:32:08.088245+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:08.088245+0000","last_undegraded":"2024-03-16T19:32:08.088245+0000","last_fullsized":"2024-03-16T19:32:08.088245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:47:08.251341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083454+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:08.083454+0000","last_peered":"2024-03-16T19:32:08.083454+0000","last_clean":"2024-03-16T19:32:08.083454+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:08.083454+0000","last_undegraded":"2024-03-16T19:32:08.083454+0000","last_fullsized":"2024-03-16T19:32:08.083454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086134+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:08.086134+0000","last_peered":"2024-03-16T19:32:08.086134+0000","last_clean":"2024-03-16T19:32:08.086134+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:08.086134+0000","last_undegraded":"2024-03-16T19:32:08.086134+0000","last_fullsized":"2024-03-16T19:32:08.086134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:10.818175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287911+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:09.287911+0000","last_peered":"2024-03-16T19:32:09.287911+0000","last_clean":"2024-03-16T19:32:09.287911+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:09.287911+0000","last_undegraded":"2024-03-16T19:32:09.287911+0000","last_fullsized":"2024-03-16T19:32:09.287911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":187,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093433+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:08.093433+0000","last_peered":"2024-03-16T19:32:08.093433+0000","last_clean":"2024-03-16T19:32:08.093433+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:08.093433+0000","last_undegraded":"2024-03-16T19:32:08.093433+0000","last_fullsized":"2024-03-16T19:32:08.093433+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087892+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:08.087892+0000","last_peered":"2024-03-16T19:32:08.087892+0000","last_clean":"2024-03-16T19:32:08.087892+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:08.087892+0000","last_undegraded":"2024-03-16T19:32:08.087892+0000","last_fullsized":"2024-03-16T19:32:08.087892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:12:08.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350890+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:08.350890+0000","last_peered":"2024-03-16T19:32:08.350890+0000","last_clean":"2024-03-16T19:32:08.350890+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:08.350890+0000","last_undegraded":"2024-03-16T19:32:08.350890+0000","last_fullsized":"2024-03-16T19:32:08.350890+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:12.646493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083749+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:08.083749+0000","last_peered":"2024-03-16T19:32:08.083749+0000","last_clean":"2024-03-16T19:32:08.083749+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:08.083749+0000","last_undegraded":"2024-03-16T19:32:08.083749+0000","last_fullsized":"2024-03-16T19:32:08.083749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087224+0000","last_change":"2024-03-16T19:31:49.974297+0000","last_active":"2024-03-16T19:32:08.087224+0000","last_peered":"2024-03-16T19:32:08.087224+0000","last_clean":"2024-03-16T19:32:08.087224+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:08.087224+0000","last_undegraded":"2024-03-16T19:32:08.087224+0000","last_fullsized":"2024-03-16T19:32:08.087224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:51:57.535372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287591+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:09.287591+0000","last_peered":"2024-03-16T19:32:09.287591+0000","last_clean":"2024-03-16T19:32:09.287591+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:09.287591+0000","last_undegraded":"2024-03-16T19:32:09.287591+0000","last_fullsized":"2024-03-16T19:32:09.287591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084174+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:32:08.084174+0000","last_peered":"2024-03-16T19:32:08.084174+0000","last_clean":"2024-03-16T19:32:08.084174+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:08.084174+0000","last_undegraded":"2024-03-16T19:32:08.084174+0000","last_fullsized":"2024-03-16T19:32:08.084174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089011+0000","last_change":"2024-03-16T19:31:26.808032+0000","last_active":"2024-03-16T19:32:08.089011+0000","last_peered":"2024-03-16T19:32:08.089011+0000","last_clean":"2024-03-16T19:32:08.089011+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:08.089011+0000","last_undegraded":"2024-03-16T19:32:08.089011+0000","last_fullsized":"2024-03-16T19:32:08.089011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:15.218778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351569+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:08.351569+0000","last_peered":"2024-03-16T19:32:08.351569+0000","last_clean":"2024-03-16T19:32:08.351569+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:08.351569+0000","last_undegraded":"2024-03-16T19:32:08.351569+0000","last_fullsized":"2024-03-16T19:32:08.351569+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:16:31.606469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287611+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:09.287611+0000","last_peered":"2024-03-16T19:32:09.287611+0000","last_clean":"2024-03-16T19:32:09.287611+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:09.287611+0000","last_undegraded":"2024-03-16T19:32:09.287611+0000","last_fullsized":"2024-03-16T19:32:09.287611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086078+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086078+0000","last_peered":"2024-03-16T19:32:08.086078+0000","last_clean":"2024-03-16T19:32:08.086078+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086078+0000","last_undegraded":"2024-03-16T19:32:08.086078+0000","last_fullsized":"2024-03-16T19:32:08.086078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:23:19.824904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086998+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:08.086998+0000","last_peered":"2024-03-16T19:32:08.086998+0000","last_clean":"2024-03-16T19:32:08.086998+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:08.086998+0000","last_undegraded":"2024-03-16T19:32:08.086998+0000","last_fullsized":"2024-03-16T19:32:08.086998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:45.782085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094004+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:08.094004+0000","last_peered":"2024-03-16T19:32:08.094004+0000","last_clean":"2024-03-16T19:32:08.094004+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:08.094004+0000","last_undegraded":"2024-03-16T19:32:08.094004+0000","last_fullsized":"2024-03-16T19:32:08.094004+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287793+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:09.287793+0000","last_peered":"2024-03-16T19:32:09.287793+0000","last_clean":"2024-03-16T19:32:09.287793+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:09.287793+0000","last_undegraded":"2024-03-16T19:32:09.287793+0000","last_fullsized":"2024-03-16T19:32:09.287793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083754+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:08.083754+0000","last_peered":"2024-03-16T19:32:08.083754+0000","last_clean":"2024-03-16T19:32:08.083754+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:08.083754+0000","last_undegraded":"2024-03-16T19:32:08.083754+0000","last_fullsized":"2024-03-16T19:32:08.083754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351532+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:08.351532+0000","last_peered":"2024-03-16T19:32:08.351532+0000","last_clean":"2024-03-16T19:32:08.351532+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:08.351532+0000","last_undegraded":"2024-03-16T19:32:08.351532+0000","last_fullsized":"2024-03-16T19:32:08.351532+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:16.418962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093565+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:08.093565+0000","last_peered":"2024-03-16T19:32:08.093565+0000","last_clean":"2024-03-16T19:32:08.093565+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:08.093565+0000","last_undegraded":"2024-03-16T19:32:08.093565+0000","last_fullsized":"2024-03-16T19:32:08.093565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938817+0000","last_change":"2024-03-16T19:32:08.938817+0000","last_active":"2024-03-16T19:32:08.938817+0000","last_peered":"2024-03-16T19:32:08.938817+0000","last_clean":"2024-03-16T19:32:08.938817+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:08.938817+0000","last_undegraded":"2024-03-16T19:32:08.938817+0000","last_fullsized":"2024-03-16T19:32:08.938817+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:39:32.174855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"86'23","reported_seq":233,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287734+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:09.287734+0000","last_peered":"2024-03-16T19:32:09.287734+0000","last_clean":"2024-03-16T19:32:09.287734+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:09.287734+0000","last_undegraded":"2024-03-16T19:32:09.287734+0000","last_fullsized":"2024-03-16T19:32:09.287734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39292,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937730+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:08.937730+0000","last_peered":"2024-03-16T19:32:08.937730+0000","last_clean":"2024-03-16T19:32:08.937730+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:08.937730+0000","last_undegraded":"2024-03-16T19:32:08.937730+0000","last_fullsized":"2024-03-16T19:32:08.937730+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351564+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:08.351564+0000","last_peered":"2024-03-16T19:32:08.351564+0000","last_clean":"2024-03-16T19:32:08.351564+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:08.351564+0000","last_undegraded":"2024-03-16T19:32:08.351564+0000","last_fullsized":"2024-03-16T19:32:08.351564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:59.818393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351224+0000","last_change":"2024-03-16T19:31:51.807895+0000","last_active":"2024-03-16T19:32:08.351224+0000","last_peered":"2024-03-16T19:32:08.351224+0000","last_clean":"2024-03-16T19:32:08.351224+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:08.351224+0000","last_undegraded":"2024-03-16T19:32:08.351224+0000","last_fullsized":"2024-03-16T19:32:08.351224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:25.988237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937695+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:08.937695+0000","last_peered":"2024-03-16T19:32:08.937695+0000","last_clean":"2024-03-16T19:32:08.937695+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:08.937695+0000","last_undegraded":"2024-03-16T19:32:08.937695+0000","last_fullsized":"2024-03-16T19:32:08.937695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287610+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:09.287610+0000","last_peered":"2024-03-16T19:32:09.287610+0000","last_clean":"2024-03-16T19:32:09.287610+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:09.287610+0000","last_undegraded":"2024-03-16T19:32:09.287610+0000","last_fullsized":"2024-03-16T19:32:09.287610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287595+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:09.287595+0000","last_peered":"2024-03-16T19:32:09.287595+0000","last_clean":"2024-03-16T19:32:09.287595+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:09.287595+0000","last_undegraded":"2024-03-16T19:32:09.287595+0000","last_fullsized":"2024-03-16T19:32:09.287595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087374+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:08.087374+0000","last_peered":"2024-03-16T19:32:08.087374+0000","last_clean":"2024-03-16T19:32:08.087374+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:08.087374+0000","last_undegraded":"2024-03-16T19:32:08.087374+0000","last_fullsized":"2024-03-16T19:32:08.087374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:33.634146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937562+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:08.937562+0000","last_peered":"2024-03-16T19:32:08.937562+0000","last_clean":"2024-03-16T19:32:08.937562+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:08.937562+0000","last_undegraded":"2024-03-16T19:32:08.937562+0000","last_fullsized":"2024-03-16T19:32:08.937562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084319+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084319+0000","last_peered":"2024-03-16T19:32:08.084319+0000","last_clean":"2024-03-16T19:32:08.084319+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084319+0000","last_undegraded":"2024-03-16T19:32:08.084319+0000","last_fullsized":"2024-03-16T19:32:08.084319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:53.021797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937869+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:08.937869+0000","last_peered":"2024-03-16T19:32:08.937869+0000","last_clean":"2024-03-16T19:32:08.937869+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:08.937869+0000","last_undegraded":"2024-03-16T19:32:08.937869+0000","last_fullsized":"2024-03-16T19:32:08.937869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":55,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086917+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:08.086917+0000","last_peered":"2024-03-16T19:32:08.086917+0000","last_clean":"2024-03-16T19:32:08.086917+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:08.086917+0000","last_undegraded":"2024-03-16T19:32:08.086917+0000","last_fullsized":"2024-03-16T19:32:08.086917+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:07:39.853290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938036+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:08.938036+0000","last_peered":"2024-03-16T19:32:08.938036+0000","last_clean":"2024-03-16T19:32:08.938036+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:08.938036+0000","last_undegraded":"2024-03-16T19:32:08.938036+0000","last_fullsized":"2024-03-16T19:32:08.938036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093703+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:08.093703+0000","last_peered":"2024-03-16T19:32:08.093703+0000","last_clean":"2024-03-16T19:32:08.093703+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:08.093703+0000","last_undegraded":"2024-03-16T19:32:08.093703+0000","last_fullsized":"2024-03-16T19:32:08.093703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084002+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:08.084002+0000","last_peered":"2024-03-16T19:32:08.084002+0000","last_clean":"2024-03-16T19:32:08.084002+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:08.084002+0000","last_undegraded":"2024-03-16T19:32:08.084002+0000","last_fullsized":"2024-03-16T19:32:08.084002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086673+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:08.086673+0000","last_peered":"2024-03-16T19:32:08.086673+0000","last_clean":"2024-03-16T19:32:08.086673+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:08.086673+0000","last_undegraded":"2024-03-16T19:32:08.086673+0000","last_fullsized":"2024-03-16T19:32:08.086673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:59:55.019036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":148,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937729+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:08.937729+0000","last_peered":"2024-03-16T19:32:08.937729+0000","last_clean":"2024-03-16T19:32:08.937729+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:08.937729+0000","last_undegraded":"2024-03-16T19:32:08.937729+0000","last_fullsized":"2024-03-16T19:32:08.937729+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":363,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937413+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:08.937413+0000","last_peered":"2024-03-16T19:32:08.937413+0000","last_clean":"2024-03-16T19:32:08.937413+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:08.937413+0000","last_undegraded":"2024-03-16T19:32:08.937413+0000","last_fullsized":"2024-03-16T19:32:08.937413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087541+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:08.087541+0000","last_peered":"2024-03-16T19:32:08.087541+0000","last_clean":"2024-03-16T19:32:08.087541+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:08.087541+0000","last_undegraded":"2024-03-16T19:32:08.087541+0000","last_fullsized":"2024-03-16T19:32:08.087541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:16:54.768710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937520+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:08.937520+0000","last_peered":"2024-03-16T19:32:08.937520+0000","last_clean":"2024-03-16T19:32:08.937520+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:08.937520+0000","last_undegraded":"2024-03-16T19:32:08.937520+0000","last_fullsized":"2024-03-16T19:32:08.937520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351264+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:08.351264+0000","last_peered":"2024-03-16T19:32:08.351264+0000","last_clean":"2024-03-16T19:32:08.351264+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:08.351264+0000","last_undegraded":"2024-03-16T19:32:08.351264+0000","last_fullsized":"2024-03-16T19:32:08.351264+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:44.308565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087130+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:08.087130+0000","last_peered":"2024-03-16T19:32:08.087130+0000","last_clean":"2024-03-16T19:32:08.087130+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:08.087130+0000","last_undegraded":"2024-03-16T19:32:08.087130+0000","last_fullsized":"2024-03-16T19:32:08.087130+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:20:21.014268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.096035+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:08.096035+0000","last_peered":"2024-03-16T19:32:08.096035+0000","last_clean":"2024-03-16T19:32:08.096035+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:08.096035+0000","last_undegraded":"2024-03-16T19:32:08.096035+0000","last_fullsized":"2024-03-16T19:32:08.096035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084314+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:08.084314+0000","last_peered":"2024-03-16T19:32:08.084314+0000","last_clean":"2024-03-16T19:32:08.084314+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:08.084314+0000","last_undegraded":"2024-03-16T19:32:08.084314+0000","last_fullsized":"2024-03-16T19:32:08.084314+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:01.856961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093299+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:08.093299+0000","last_peered":"2024-03-16T19:32:08.093299+0000","last_clean":"2024-03-16T19:32:08.093299+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:08.093299+0000","last_undegraded":"2024-03-16T19:32:08.093299+0000","last_fullsized":"2024-03-16T19:32:08.093299+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084193+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:08.084193+0000","last_peered":"2024-03-16T19:32:08.084193+0000","last_clean":"2024-03-16T19:32:08.084193+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:08.084193+0000","last_undegraded":"2024-03-16T19:32:08.084193+0000","last_fullsized":"2024-03-16T19:32:08.084193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937906+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:08.937906+0000","last_peered":"2024-03-16T19:32:08.937906+0000","last_clean":"2024-03-16T19:32:08.937906+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:08.937906+0000","last_undegraded":"2024-03-16T19:32:08.937906+0000","last_fullsized":"2024-03-16T19:32:08.937906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093171+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:08.093171+0000","last_peered":"2024-03-16T19:32:08.093171+0000","last_clean":"2024-03-16T19:32:08.093171+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:08.093171+0000","last_undegraded":"2024-03-16T19:32:08.093171+0000","last_fullsized":"2024-03-16T19:32:08.093171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'11","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094654+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:08.094654+0000","last_peered":"2024-03-16T19:32:08.094654+0000","last_clean":"2024-03-16T19:32:08.094654+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:08.094654+0000","last_undegraded":"2024-03-16T19:32:08.094654+0000","last_fullsized":"2024-03-16T19:32:08.094654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088741+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:08.088741+0000","last_peered":"2024-03-16T19:32:08.088741+0000","last_clean":"2024-03-16T19:32:08.088741+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:08.088741+0000","last_undegraded":"2024-03-16T19:32:08.088741+0000","last_fullsized":"2024-03-16T19:32:08.088741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:18.996236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287797+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:09.287797+0000","last_peered":"2024-03-16T19:32:09.287797+0000","last_clean":"2024-03-16T19:32:09.287797+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:09.287797+0000","last_undegraded":"2024-03-16T19:32:09.287797+0000","last_fullsized":"2024-03-16T19:32:09.287797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350908+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:08.350908+0000","last_peered":"2024-03-16T19:32:08.350908+0000","last_clean":"2024-03-16T19:32:08.350908+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:08.350908+0000","last_undegraded":"2024-03-16T19:32:08.350908+0000","last_fullsized":"2024-03-16T19:32:08.350908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:02.024100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937470+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:08.937470+0000","last_peered":"2024-03-16T19:32:08.937470+0000","last_clean":"2024-03-16T19:32:08.937470+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:08.937470+0000","last_undegraded":"2024-03-16T19:32:08.937470+0000","last_fullsized":"2024-03-16T19:32:08.937470+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083795+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:08.083795+0000","last_peered":"2024-03-16T19:32:08.083795+0000","last_clean":"2024-03-16T19:32:08.083795+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:08.083795+0000","last_undegraded":"2024-03-16T19:32:08.083795+0000","last_fullsized":"2024-03-16T19:32:08.083795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:46:49.628764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094246+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:08.094246+0000","last_peered":"2024-03-16T19:32:08.094246+0000","last_clean":"2024-03-16T19:32:08.094246+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:08.094246+0000","last_undegraded":"2024-03-16T19:32:08.094246+0000","last_fullsized":"2024-03-16T19:32:08.094246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.102711+0000","last_change":"2024-03-16T19:32:08.101455+0000","last_active":"2024-03-16T19:32:08.102711+0000","last_peered":"2024-03-16T19:32:08.102711+0000","last_clean":"2024-03-16T19:32:08.102711+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:32:08.102711+0000","last_undegraded":"2024-03-16T19:32:08.102711+0000","last_fullsized":"2024-03-16T19:32:08.102711+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:20:10.984083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.095090+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:08.095090+0000","last_peered":"2024-03-16T19:32:08.095090+0000","last_clean":"2024-03-16T19:32:08.095090+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:08.095090+0000","last_undegraded":"2024-03-16T19:32:08.095090+0000","last_fullsized":"2024-03-16T19:32:08.095090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089208+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:08.089208+0000","last_peered":"2024-03-16T19:32:08.089208+0000","last_clean":"2024-03-16T19:32:08.089208+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:08.089208+0000","last_undegraded":"2024-03-16T19:32:08.089208+0000","last_fullsized":"2024-03-16T19:32:08.089208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:03.618618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937781+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:08.937781+0000","last_peered":"2024-03-16T19:32:08.937781+0000","last_clean":"2024-03-16T19:32:08.937781+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:08.937781+0000","last_undegraded":"2024-03-16T19:32:08.937781+0000","last_fullsized":"2024-03-16T19:32:08.937781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287721+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:09.287721+0000","last_peered":"2024-03-16T19:32:09.287721+0000","last_clean":"2024-03-16T19:32:09.287721+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:09.287721+0000","last_undegraded":"2024-03-16T19:32:09.287721+0000","last_fullsized":"2024-03-16T19:32:09.287721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084106+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:08.084106+0000","last_peered":"2024-03-16T19:32:08.084106+0000","last_clean":"2024-03-16T19:32:08.084106+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:08.084106+0000","last_undegraded":"2024-03-16T19:32:08.084106+0000","last_fullsized":"2024-03-16T19:32:08.084106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937766+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:08.937766+0000","last_peered":"2024-03-16T19:32:08.937766+0000","last_clean":"2024-03-16T19:32:08.937766+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:08.937766+0000","last_undegraded":"2024-03-16T19:32:08.937766+0000","last_fullsized":"2024-03-16T19:32:08.937766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083913+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:08.083913+0000","last_peered":"2024-03-16T19:32:08.083913+0000","last_clean":"2024-03-16T19:32:08.083913+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:08.083913+0000","last_undegraded":"2024-03-16T19:32:08.083913+0000","last_fullsized":"2024-03-16T19:32:08.083913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937841+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:08.937841+0000","last_peered":"2024-03-16T19:32:08.937841+0000","last_clean":"2024-03-16T19:32:08.937841+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:08.937841+0000","last_undegraded":"2024-03-16T19:32:08.937841+0000","last_fullsized":"2024-03-16T19:32:08.937841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":164,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351010+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:08.351010+0000","last_peered":"2024-03-16T19:32:08.351010+0000","last_clean":"2024-03-16T19:32:08.351010+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:08.351010+0000","last_undegraded":"2024-03-16T19:32:08.351010+0000","last_fullsized":"2024-03-16T19:32:08.351010+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:19.317523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088482+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088482+0000","last_peered":"2024-03-16T19:32:08.088482+0000","last_clean":"2024-03-16T19:32:08.088482+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088482+0000","last_undegraded":"2024-03-16T19:32:08.088482+0000","last_fullsized":"2024-03-16T19:32:08.088482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:58:01.695714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086003+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:08.086003+0000","last_peered":"2024-03-16T19:32:08.086003+0000","last_clean":"2024-03-16T19:32:08.086003+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:08.086003+0000","last_undegraded":"2024-03-16T19:32:08.086003+0000","last_fullsized":"2024-03-16T19:32:08.086003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:17.040490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351503+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:08.351503+0000","last_peered":"2024-03-16T19:32:08.351503+0000","last_clean":"2024-03-16T19:32:08.351503+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:08.351503+0000","last_undegraded":"2024-03-16T19:32:08.351503+0000","last_fullsized":"2024-03-16T19:32:08.351503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:28:10.116570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40494,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":80988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28840,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343000,"statfs":{"total":96636764160,"available":96607232000,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1310720,"data_stored":634440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343536,"statfs":{"total":96636764160,"available":96607780864,"internally_reserved":0,"allocated":1314816,"data_stored":634844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122056,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29408,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342432,"statfs":{"total":96636764160,"available":96606650368,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28848,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342992,"statfs":{"total":96636764160,"available":96607223808,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:18.697 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-16T19:32:18.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-16T19:32:18.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.894+0000 7f98300ec640 1 Processor -- start 2024-03-16T19:32:18.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.895+0000 7f98300ec640 1 -- start start 2024-03-16T19:32:18.876 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f98300ec640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f982815a670 0x7f982806d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f98300ec640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f98280c8e10 0x7f982815aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f98300ec640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9828073060 con 0x7f982815af90 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f98300ec640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9828072890 con 0x7f98280c8e10 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f98300ec640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98280ca2e0 con 0x7f982815a670 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f982d687640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f98280c8e10 0x7f982815aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f982d687640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f98280c8e10 0x7f982815aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46584/0 (socket says 172.21.15.172:46584) 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f982d687640 1 -- 172.21.15.172:0/350502407 learned_addr learned my addr 172.21.15.172:0/350502407 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f982e689640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f982815af90 0x7f9828150d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39950/0 (socket says 172.21.15.172:39950) 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.896+0000 7f982de88640 1 --2- 172.21.15.172:0/350502407 >> v2:172.21.15.174:3300/0 conn(0x7f982815a670 0x7f982806d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.897+0000 7f982d687640 1 -- 172.21.15.172:0/350502407 >> v2:172.21.15.174:3300/0 conn(0x7f982815a670 msgr2=0x7f982806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.897+0000 7f982ce86640 1 -- 172.21.15.172:0/350502407 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125533625 0 0) 0x7f9828073060 con 0x7f982815af90 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.897+0000 7f982d687640 1 --2- 172.21.15.172:0/350502407 >> v2:172.21.15.174:3300/0 conn(0x7f982815a670 0x7f982806d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.897+0000 7f982d687640 1 -- 172.21.15.172:0/350502407 >> v1:172.21.15.172:6789/0 conn(0x7f982815af90 legacy=0x7f9828150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.897+0000 7f982d687640 1 -- 172.21.15.172:0/350502407 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982806cf30 con 0x7f98280c8e10 2024-03-16T19:32:18.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.897+0000 7f982de88640 1 --2- 172.21.15.172:0/350502407 >> v2:172.21.15.174:3300/0 conn(0x7f982815a670 0x7f982806d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:18.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.898+0000 7f982d687640 1 --2- 172.21.15.172:0/350502407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98280c8e10 0x7f982815aa50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f98280c9fd0 tx=0x7f981802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3cbee64d3f46468 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:18.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.899+0000 7f982ce86640 1 -- 172.21.15.172:0/350502407 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818030860 con 0x7f98280c8e10 2024-03-16T19:32:18.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.899+0000 7f982ce86640 1 -- 172.21.15.172:0/350502407 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98180309c0 con 0x7f98280c8e10 2024-03-16T19:32:18.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.899+0000 7f982ce86640 1 -- 172.21.15.172:0/350502407 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818030d50 con 0x7f98280c8e10 2024-03-16T19:32:18.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.899+0000 7f98300ec640 1 -- 172.21.15.172:0/350502407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98280c8e10 msgr2=0x7f982815aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.899+0000 7f98300ec640 1 --2- 172.21.15.172:0/350502407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98280c8e10 0x7f982815aa50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f98280c9fd0 tx=0x7f981802b9a0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.899+0000 7f98300ec640 1 -- 172.21.15.172:0/350502407 shutdown_connections 2024-03-16T19:32:18.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.899+0000 7f98300ec640 1 --2- 172.21.15.172:0/350502407 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f98280c8e10 0x7f982815aa50 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.899+0000 7f98300ec640 1 --2- 172.21.15.172:0/350502407 >> v2:172.21.15.174:3300/0 conn(0x7f982815a670 0x7f982806d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.899+0000 7f98300ec640 1 -- 172.21.15.172:0/350502407 >> 172.21.15.172:0/350502407 conn(0x7f98280c9390 msgr2=0x7f98280c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:18.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.900+0000 7f98300ec640 1 -- 172.21.15.172:0/350502407 shutdown_connections 2024-03-16T19:32:18.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.900+0000 7f98300ec640 1 -- 172.21.15.172:0/350502407 wait complete. 2024-03-16T19:32:18.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.901+0000 7f98300ec640 1 Processor -- start 2024-03-16T19:32:18.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.901+0000 7f98300ec640 1 -- start start 2024-03-16T19:32:18.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.902+0000 7f98300ec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98280c8e10 0x7f9828157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.902+0000 7f98300ec640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982815a670 0x7f98280c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.902+0000 7f98300ec640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f982815af90 0x7f9828145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.902+0000 7f98300ec640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982814c840 con 0x7f982815af90 2024-03-16T19:32:18.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.902+0000 7f98300ec640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982814bd50 con 0x7f982815a670 2024-03-16T19:32:18.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.902+0000 7f98300ec640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982814b460 con 0x7f98280c8e10 2024-03-16T19:32:18.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982e689640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f982815af90 0x7f9828145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982de88640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98280c8e10 0x7f9828157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982de88640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98280c8e10 0x7f9828157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44674/0 (socket says 172.21.15.172:44674) 2024-03-16T19:32:18.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982de88640 1 -- 172.21.15.172:0/3565164366 learned_addr learned my addr 172.21.15.172:0/3565164366 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:18.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982d687640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982815a670 0x7f98280c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982de88640 1 -- 172.21.15.172:0/3565164366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982815a670 msgr2=0x7f98280c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982de88640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982815a670 0x7f98280c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982de88640 1 -- 172.21.15.172:0/3565164366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f982815af90 msgr2=0x7f9828145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982de88640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f982815af90 0x7f9828145d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982de88640 1 -- 172.21.15.172:0/3565164366 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824009400 con 0x7f98280c8e10 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982d687640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982815a670 0x7f98280c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f982de88640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98280c8e10 0x7f9828157ff0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f982400ec10 tx=0x7f982400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f981e7fc640 1 -- 172.21.15.172:0/3565164366 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824009760 con 0x7f98280c8e10 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f981e7fc640 1 -- 172.21.15.172:0/3565164366 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982400cc30 con 0x7f98280c8e10 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f981e7fc640 1 -- 172.21.15.172:0/3565164366 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824013470 con 0x7f98280c8e10 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98180094d0 con 0x7f98280c8e10 2024-03-16T19:32:18.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.903+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982814b700 con 0x7f98280c8e10 2024-03-16T19:32:18.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.904+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98280688d0 con 0x7f98280c8e10 2024-03-16T19:32:18.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.905+0000 7f981e7fc640 1 -- 172.21.15.172:0/3565164366 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f98240135d0 con 0x7f98280c8e10 2024-03-16T19:32:18.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.906+0000 7f981e7fc640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980c06b3c0 0x7f980c06d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:18.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.906+0000 7f981e7fc640 1 -- 172.21.15.172:0/3565164366 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f982408a250 con 0x7f98280c8e10 2024-03-16T19:32:18.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.906+0000 7f982d687640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980c06b3c0 0x7f980c06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:18.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.907+0000 7f982d687640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980c06b3c0 0x7f980c06d880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9818036000 tx=0x7f9818008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:18.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:18.910+0000 7f981e7fc640 1 -- 172.21.15.172:0/3565164366 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9824054f50 con 0x7f98280c8e10 2024-03-16T19:32:19.184 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.205+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) v1 -- 0x7f98280cbb00 con 0x7f98280c8e10 2024-03-16T19:32:19.188 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.209+0000 7f981e7fc640 1 -- 172.21.15.172:0/3565164366 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v11) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9824058610 con 0x7f98280c8e10 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980c06b3c0 msgr2=0x7f980c06d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980c06b3c0 0x7f980c06d880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9818036000 tx=0x7f9818008030 comp rx=0 tx=0).stop 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98280c8e10 msgr2=0x7f9828157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98280c8e10 0x7f9828157ff0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f982400ec10 tx=0x7f982400c520 comp rx=0 tx=0).stop 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 shutdown_connections 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f982815af90 0x7f9828145d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982815a670 0x7f98280c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f980c06b3c0 0x7f980c06d880 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 --2- 172.21.15.172:0/3565164366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98280c8e10 0x7f9828157ff0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 >> 172.21.15.172:0/3565164366 conn(0x7f98280c9390 msgr2=0x7f98280c9770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:19.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 shutdown_connections 2024-03-16T19:32:19.193 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.212+0000 7f98300ec640 1 -- 172.21.15.172:0/3565164366 wait complete. 2024-03-16T19:32:19.208 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-16T19:32:19.217 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-16T19:32:19.217 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:32:19.217 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T19:32:19.237 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-16T19:32:19.237 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T19:32:19.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:32:19.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.503+0000 7fddcf2d0640 1 Processor -- start 2024-03-16T19:32:19.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.503+0000 7fddcf2d0640 1 -- start start 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcf2d0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fddc8154cb0 0x7fddc806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcf2d0640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fddc806fe10 0x7fddc815de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcf2d0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc815e340 con 0x7fddc80c8ba0 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcf2d0640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc80cbba0 con 0x7fddc806fe10 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcf2d0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc80c91f0 con 0x7fddc8154cb0 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcd06c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fddc8154cb0 0x7fddc806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcd86d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fddc80c8ba0 0x7fddc80cabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:37370/0 (socket says 172.21.15.172:37370) 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcd06c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fddc8154cb0 0x7fddc806dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:54110/0 (socket says 172.21.15.172:54110) 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcd86d640 1 -- 172.21.15.172:0/3734720955 learned_addr learned my addr 172.21.15.172:0/3734720955 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:19.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcc86b640 1 --2- 172.21.15.172:0/3734720955 >> v2:172.21.15.172:3300/0 conn(0x7fddc806fe10 0x7fddc815de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcd06c640 1 -- 172.21.15.172:0/3734720955 >> v2:172.21.15.172:3300/0 conn(0x7fddc806fe10 msgr2=0x7fddc815de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:19.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcd06c640 1 --2- 172.21.15.172:0/3734720955 >> v2:172.21.15.172:3300/0 conn(0x7fddc806fe10 0x7fddc815de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcd06c640 1 -- 172.21.15.172:0/3734720955 >> v1:172.21.15.174:6789/0 conn(0x7fddc80c8ba0 legacy=0x7fddc80cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddcd06c640 1 -- 172.21.15.172:0/3734720955 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc806c180 con 0x7fddc8154cb0 2024-03-16T19:32:19.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.504+0000 7fddbffff640 1 -- 172.21.15.172:0/3734720955 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818307379 0 0) 0x7fddc815e340 con 0x7fddc80c8ba0 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.505+0000 7fddcd06c640 1 --2- 172.21.15.172:0/3734720955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc8154cb0 0x7fddc806dc20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fddc80c8860 tx=0x7fddc402b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9489f52e0859cd34 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.505+0000 7fddbffff640 1 -- 172.21.15.172:0/3734720955 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4031520 con 0x7fddc8154cb0 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.505+0000 7fddbffff640 1 -- 172.21.15.172:0/3734720955 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc4031680 con 0x7fddc8154cb0 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.505+0000 7fddbffff640 1 -- 172.21.15.172:0/3734720955 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4031a10 con 0x7fddc8154cb0 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.506+0000 7fddcf2d0640 1 -- 172.21.15.172:0/3734720955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc8154cb0 msgr2=0x7fddc806dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.506+0000 7fddcf2d0640 1 --2- 172.21.15.172:0/3734720955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc8154cb0 0x7fddc806dc20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fddc80c8860 tx=0x7fddc402b7c0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.506+0000 7fddcf2d0640 1 -- 172.21.15.172:0/3734720955 shutdown_connections 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.506+0000 7fddcf2d0640 1 --2- 172.21.15.172:0/3734720955 >> v2:172.21.15.172:3300/0 conn(0x7fddc806fe10 0x7fddc815de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.506+0000 7fddcf2d0640 1 --2- 172.21.15.172:0/3734720955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc8154cb0 0x7fddc806dc20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.506+0000 7fddcf2d0640 1 -- 172.21.15.172:0/3734720955 >> 172.21.15.172:0/3734720955 conn(0x7fddc8070860 msgr2=0x7fddc8070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.506+0000 7fddcf2d0640 1 -- 172.21.15.172:0/3734720955 shutdown_connections 2024-03-16T19:32:19.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.506+0000 7fddcf2d0640 1 -- 172.21.15.172:0/3734720955 wait complete. 2024-03-16T19:32:19.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.507+0000 7fddcf2d0640 1 Processor -- start 2024-03-16T19:32:19.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.508+0000 7fddcf2d0640 1 -- start start 2024-03-16T19:32:19.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.508+0000 7fddcf2d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc806fe10 0x7fddc815a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.508+0000 7fddcf2d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fddc80c8ba0 0x7fddc815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.508+0000 7fddcf2d0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc8154cb0 0x7fddc80c3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.508+0000 7fddcf2d0640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80ce100 con 0x7fddc8154cb0 2024-03-16T19:32:19.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.508+0000 7fddcf2d0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80ce350 con 0x7fddc806fe10 2024-03-16T19:32:19.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.508+0000 7fddcf2d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80ce630 con 0x7fddc80c8ba0 2024-03-16T19:32:19.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcc86b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fddc80c8ba0 0x7fddc815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcd06c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc806fe10 0x7fddc815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcc86b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fddc80c8ba0 0x7fddc815ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44686/0 (socket says 172.21.15.172:44686) 2024-03-16T19:32:19.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcc86b640 1 -- 172.21.15.172:0/428763687 learned_addr learned my addr 172.21.15.172:0/428763687 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:19.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcd06c640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc806fe10 0x7fddc815a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:54122/0 (socket says 172.21.15.172:54122) 2024-03-16T19:32:19.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcd86d640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc8154cb0 0x7fddc80c3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.489 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcc86b640 1 -- 172.21.15.172:0/428763687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc806fe10 msgr2=0x7fddc815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcc86b640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc806fe10 0x7fddc815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcc86b640 1 -- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc8154cb0 msgr2=0x7fddc80c3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcc86b640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc8154cb0 0x7fddc80c3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcc86b640 1 -- 172.21.15.172:0/428763687 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc4009400 con 0x7fddc80c8ba0 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcc86b640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fddc80c8ba0 0x7fddc815ae10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fddb800e9b0 tx=0x7fddb800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddbdffb640 1 -- 172.21.15.172:0/428763687 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8009670 con 0x7fddc80c8ba0 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddbdffb640 1 -- 172.21.15.172:0/428763687 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb800c9a0 con 0x7fddc80c8ba0 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc80ce910 con 0x7fddc80c8ba0 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.509+0000 7fddbdffb640 1 -- 172.21.15.172:0/428763687 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb800cd50 con 0x7fddc80c8ba0 2024-03-16T19:32:19.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.510+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc80ceed0 con 0x7fddc80c8ba0 2024-03-16T19:32:19.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.511+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc80b4540 con 0x7fddc80c8ba0 2024-03-16T19:32:19.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.511+0000 7fddbdffb640 1 -- 172.21.15.172:0/428763687 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fddb8013470 con 0x7fddc80c8ba0 2024-03-16T19:32:19.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.512+0000 7fddbdffb640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fddac06b3b0 0x7fddac06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.512+0000 7fddbdffb640 1 -- 172.21.15.172:0/428763687 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fddb8089ec0 con 0x7fddc80c8ba0 2024-03-16T19:32:19.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.512+0000 7fddcd06c640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fddac06b3b0 0x7fddac06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.512+0000 7fddcd06c640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fddac06b3b0 0x7fddac06d870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fddc4035000 tx=0x7fddc4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:19.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.516+0000 7fddbdffb640 1 -- 172.21.15.172:0/428763687 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddb8054b30 con 0x7fddc80c8ba0 2024-03-16T19:32:19.772 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.792+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddc80ce770 con 0x7fddc80c8ba0 2024-03-16T19:32:19.773 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.793+0000 7fddbdffb640 1 -- 172.21.15.172:0/428763687 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16884 (secure 0 0 0) 0x7fddc80ce770 con 0x7fddc80c8ba0 2024-03-16T19:32:19.773 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:19.774 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":106,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:32:08.075358+0000","last_up_change":"2024-03-16T19:31:48.956759+0000","last_in_change":"2024-03-16T19:31:48.956759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":58,"up_from":88,"up_thru":105,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6825","nonce":3572014952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6827","nonce":3572014952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6831","nonce":3572014952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6829","nonce":3572014952}]},"public_addr":"172.21.15.172:6825/3572014952","cluster_addr":"172.21.15.172:6827/3572014952","heartbeat_back_addr":"172.21.15.172:6831/3572014952","heartbeat_front_addr":"172.21.15.172:6829/3572014952","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:32:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fddac06b3b0 msgr2=0x7fddac06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fddac06b3b0 0x7fddac06d870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fddc4035000 tx=0x7fddc4030000 comp rx=0 tx=0).stop 2024-03-16T19:32:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fddc80c8ba0 msgr2=0x7fddc815ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fddc80c8ba0 0x7fddc815ae10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fddb800e9b0 tx=0x7fddb800ee70 comp rx=0 tx=0).stop 2024-03-16T19:32:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 shutdown_connections 2024-03-16T19:32:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fddc8154cb0 0x7fddc80c3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fddac06b3b0 0x7fddac06d870 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fddc80c8ba0 0x7fddc815ae10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 --2- 172.21.15.172:0/428763687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddc806fe10 0x7fddc815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 >> 172.21.15.172:0/428763687 conn(0x7fddc8070860 msgr2=0x7fddc8152a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.795+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 shutdown_connections 2024-03-16T19:32:19.777 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.796+0000 7fddcf2d0640 1 -- 172.21.15.172:0/428763687 wait complete. 2024-03-16T19:32:19.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T19:29:14.242742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T19:29:24.121132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T19:29:24.651099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T19:32:19.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T19:32:19.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.960+0000 7f33889d7640 1 Processor -- start 2024-03-16T19:32:19.940 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.960+0000 7f33889d7640 1 -- start start 2024-03-16T19:32:19.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.961+0000 7f33889d7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3384072080 0x7f338415b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.961+0000 7f33889d7640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338415aae0 con 0x7f3384155060 2024-03-16T19:32:19.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.961+0000 7f33889d7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338415ef50 con 0x7f33840cd540 2024-03-16T19:32:19.941 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.961+0000 7f33889d7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33840672d0 con 0x7f3384072080 2024-03-16T19:32:19.942 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.961+0000 7f3382d9d640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3384155060 0x7f3384165380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39956/0 (socket says 172.21.15.172:39956) 2024-03-16T19:32:19.942 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.961+0000 7f338259c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3384072080 0x7f338415b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.942 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.961+0000 7f3382d9d640 1 -- 172.21.15.172:0/3570691255 learned_addr learned my addr 172.21.15.172:0/3570691255 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:19.942 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.962+0000 7f338159a640 1 -- 172.21.15.172:0/3570691255 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058092620 0 0) 0x7f338415aae0 con 0x7f3384155060 2024-03-16T19:32:19.942 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.962+0000 7f338259c640 1 -- 172.21.15.172:0/3570691255 >> v1:172.21.15.174:6789/0 conn(0x7f33840cd540 legacy=0x7f3384154950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.942 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.962+0000 7f338259c640 1 -- 172.21.15.172:0/3570691255 >> v1:172.21.15.172:6789/0 conn(0x7f3384155060 legacy=0x7f3384165380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.942 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.962+0000 7f338259c640 1 -- 172.21.15.172:0/3570691255 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3384066d50 con 0x7f3384072080 2024-03-16T19:32:19.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.963+0000 7f338259c640 1 --2- 172.21.15.172:0/3570691255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3384072080 0x7f338415b530 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3374009720 tx=0x7f337402b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=528574c92557f858 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:19.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.963+0000 7f338159a640 1 -- 172.21.15.172:0/3570691255 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374030710 con 0x7f3384072080 2024-03-16T19:32:19.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.963+0000 7f338159a640 1 -- 172.21.15.172:0/3570691255 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337402e050 con 0x7f3384072080 2024-03-16T19:32:19.943 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.963+0000 7f338159a640 1 -- 172.21.15.172:0/3570691255 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374030a90 con 0x7f3384072080 2024-03-16T19:32:19.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.964+0000 7f33889d7640 1 -- 172.21.15.172:0/3570691255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3384072080 msgr2=0x7f338415b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.964+0000 7f33889d7640 1 --2- 172.21.15.172:0/3570691255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3384072080 0x7f338415b530 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3374009720 tx=0x7f337402b730 comp rx=0 tx=0).stop 2024-03-16T19:32:19.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.964+0000 7f33889d7640 1 -- 172.21.15.172:0/3570691255 shutdown_connections 2024-03-16T19:32:19.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.964+0000 7f33889d7640 1 --2- 172.21.15.172:0/3570691255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3384072080 0x7f338415b530 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.964+0000 7f33889d7640 1 -- 172.21.15.172:0/3570691255 >> 172.21.15.172:0/3570691255 conn(0x7f33840cc890 msgr2=0x7f33840ccca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:19.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.964+0000 7f33889d7640 1 -- 172.21.15.172:0/3570691255 shutdown_connections 2024-03-16T19:32:19.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.964+0000 7f33889d7640 1 -- 172.21.15.172:0/3570691255 wait complete. 2024-03-16T19:32:19.945 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.966+0000 7f33889d7640 1 Processor -- start 2024-03-16T19:32:19.946 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.966+0000 7f33889d7640 1 -- start start 2024-03-16T19:32:19.948 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.967+0000 7f33889d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3384072080 0x7f33840c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.948 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.967+0000 7f33889d7640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33840cd540 0x7f3384159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.948 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.967+0000 7f338259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3384072080 0x7f33840c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.948 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.967+0000 7f338259c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3384072080 0x7f33840c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44688/0 (socket says 172.21.15.172:44688) 2024-03-16T19:32:19.948 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.967+0000 7f3381d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33840cd540 0x7f3384159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.948 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.967+0000 7f3381d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33840cd540 0x7f3384159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46606/0 (socket says 172.21.15.172:46606) 2024-03-16T19:32:19.948 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.967+0000 7f338259c640 1 -- 172.21.15.172:0/2104189852 learned_addr learned my addr 172.21.15.172:0/2104189852 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f33889d7640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33841f3480 0x7f33840c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33841f0e60 con 0x7f33840cd540 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33841f02e0 con 0x7f33841f3480 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33841f05c0 con 0x7f3384072080 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f3382d9d640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33841f3480 0x7f33840c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f338259c640 1 -- 172.21.15.172:0/2104189852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33841f3480 msgr2=0x7f33840c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f338259c640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33841f3480 0x7f33840c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f338259c640 1 -- 172.21.15.172:0/2104189852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33840cd540 msgr2=0x7f3384159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f338259c640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33840cd540 0x7f3384159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f338259c640 1 -- 172.21.15.172:0/2104189852 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3374009400 con 0x7f3384072080 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f338259c640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3384072080 0x7f33840c8b10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3374009720 tx=0x7f33740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:19.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.968+0000 7f3372ffd640 1 -- 172.21.15.172:0/2104189852 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374032910 con 0x7f3384072080 2024-03-16T19:32:19.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.969+0000 7f3372ffd640 1 -- 172.21.15.172:0/2104189852 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3374032a70 con 0x7f3384072080 2024-03-16T19:32:19.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.969+0000 7f3372ffd640 1 -- 172.21.15.172:0/2104189852 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374032e20 con 0x7f3384072080 2024-03-16T19:32:19.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.969+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33841f08a0 con 0x7f3384072080 2024-03-16T19:32:19.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.969+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33840c7350 con 0x7f3384072080 2024-03-16T19:32:19.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.970+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33840688d0 con 0x7f3384072080 2024-03-16T19:32:19.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.970+0000 7f3372ffd640 1 -- 172.21.15.172:0/2104189852 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3374039040 con 0x7f3384072080 2024-03-16T19:32:19.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.971+0000 7f3372ffd640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f336406b3b0 0x7f336406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:19.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.971+0000 7f3372ffd640 1 -- 172.21.15.172:0/2104189852 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f33740ab7a0 con 0x7f3384072080 2024-03-16T19:32:19.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.971+0000 7f3381d9b640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f336406b3b0 0x7f336406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:19.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.972+0000 7f3381d9b640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f336406b3b0 0x7f336406d870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f33841545d0 tx=0x7f3378009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:19.955 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:19.975+0000 7f3372ffd640 1 -- 172.21.15.172:0/2104189852 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33740764a0 con 0x7f3384072080 2024-03-16T19:32:20.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.247+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f338414eb50 con 0x7f3384072080 2024-03-16T19:32:20.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.247+0000 7f3372ffd640 1 -- 172.21.15.172:0/2104189852 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f33740370a0 con 0x7f3384072080 2024-03-16T19:32:20.227 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 1 2024-03-16T19:32:20.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f336406b3b0 msgr2=0x7f336406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f336406b3b0 0x7f336406d870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f33841545d0 tx=0x7f3378009040 comp rx=0 tx=0).stop 2024-03-16T19:32:20.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3384072080 msgr2=0x7f33840c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3384072080 0x7f33840c8b10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3374009720 tx=0x7f33740046c0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 shutdown_connections 2024-03-16T19:32:20.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33841f3480 0x7f33840c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f33840cd540 0x7f3384159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.230 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f336406b3b0 0x7f336406d870 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 --2- 172.21.15.172:0/2104189852 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3384072080 0x7f33840c8b10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 >> 172.21.15.172:0/2104189852 conn(0x7f33840cc890 msgr2=0x7f338414fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:20.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 shutdown_connections 2024-03-16T19:32:20.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.250+0000 7f33889d7640 1 -- 172.21.15.172:0/2104189852 wait complete. 2024-03-16T19:32:20.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T19:32:20.350 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.370+0000 7f8cd684e640 1 Processor -- start 2024-03-16T19:32:20.350 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.370+0000 7f8cd684e640 1 -- start start 2024-03-16T19:32:20.351 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8cd684e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8cd00b5f10 0x7f8cd00caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.351 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8cd684e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd015b990 con 0x7f8cd0154e00 2024-03-16T19:32:20.351 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8cd684e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd00cc090 con 0x7f8cd00c7800 2024-03-16T19:32:20.351 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8cd684e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd015b570 con 0x7f8cd00b5f10 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8ccffff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f8cd0154e00 0x7f8cd006afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39972/0 (socket says 172.21.15.172:39972) 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8ccffff640 1 -- 172.21.15.172:0/957431006 learned_addr learned my addr 172.21.15.172:0/957431006 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8ccf7fe640 1 --2- 172.21.15.172:0/957431006 >> v2:172.21.15.174:3300/0 conn(0x7f8cd00b5f10 0x7f8cd00caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8cceffd640 1 -- 172.21.15.172:0/957431006 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120622110 0 0) 0x7f8cd015b990 con 0x7f8cd0154e00 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8cceffd640 1 -- 172.21.15.172:0/957431006 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb8003520 con 0x7f8cd0154e00 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8cceffd640 1 -- 172.21.15.172:0/957431006 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028502781 0 0) 0x7f8cd00cc090 con 0x7f8cd00c7800 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8cceffd640 1 -- 172.21.15.172:0/957431006 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd015b990 con 0x7f8cd00c7800 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.371+0000 7f8ccf7fe640 1 -- 172.21.15.172:0/957431006 >> v1:172.21.15.174:6789/0 conn(0x7f8cd00c7800 legacy=0x7f8cd0165260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.372+0000 7f8ccf7fe640 1 -- 172.21.15.172:0/957431006 >> v1:172.21.15.172:6789/0 conn(0x7f8cd0154e00 legacy=0x7f8cd006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.372+0000 7f8ccf7fe640 1 -- 172.21.15.172:0/957431006 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd015abd0 con 0x7f8cd00b5f10 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.373+0000 7f8ccf7fe640 1 --2- 172.21.15.172:0/957431006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 0x7f8cd00caed0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40097a0 tx=0x7f8cc402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3a58cd164ed52a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.373+0000 7f8cceffd640 1 -- 172.21.15.172:0/957431006 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc402f810 con 0x7f8cd00b5f10 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.373+0000 7f8cceffd640 1 -- 172.21.15.172:0/957431006 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc402f970 con 0x7f8cd00b5f10 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.373+0000 7f8cceffd640 1 -- 172.21.15.172:0/957431006 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc402fd00 con 0x7f8cd00b5f10 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.373+0000 7f8cd684e640 1 -- 172.21.15.172:0/957431006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 msgr2=0x7f8cd00caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.373+0000 7f8cd684e640 1 --2- 172.21.15.172:0/957431006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 0x7f8cd00caed0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40097a0 tx=0x7f8cc402b9f0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.374+0000 7f8cd684e640 1 -- 172.21.15.172:0/957431006 shutdown_connections 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.374+0000 7f8cd684e640 1 --2- 172.21.15.172:0/957431006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 0x7f8cd00caed0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.374+0000 7f8cd684e640 1 -- 172.21.15.172:0/957431006 >> 172.21.15.172:0/957431006 conn(0x7f8cd0066c20 msgr2=0x7f8cd0067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.374+0000 7f8cd684e640 1 -- 172.21.15.172:0/957431006 shutdown_connections 2024-03-16T19:32:20.354 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.374+0000 7f8cd684e640 1 -- 172.21.15.172:0/957431006 wait complete. 2024-03-16T19:32:20.355 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.375+0000 7f8cd684e640 1 Processor -- start 2024-03-16T19:32:20.355 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.375+0000 7f8cd684e640 1 -- start start 2024-03-16T19:32:20.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8cd684e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 0x7f8cd0156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8cd684e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd00c7800 0x7f8cd014c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8cd684e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8cd0154e00 0x7f8cd0148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8cd684e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01f06e0 con 0x7f8cd00c7800 2024-03-16T19:32:20.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8cd684e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01f0420 con 0x7f8cd00b5f10 2024-03-16T19:32:20.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8cd684e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0157340 con 0x7f8cd0154e00 2024-03-16T19:32:20.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccf7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd00c7800 0x7f8cd014c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccf7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd00c7800 0x7f8cd014c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46608/0 (socket says 172.21.15.172:46608) 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccf7fe640 1 -- 172.21.15.172:0/26928317 learned_addr learned my addr 172.21.15.172:0/26928317 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccffff640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 0x7f8cd0156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8cd4deb640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8cd0154e00 0x7f8cd0148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccffff640 1 -- 172.21.15.172:0/26928317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8cd0154e00 msgr2=0x7f8cd0148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccffff640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8cd0154e00 0x7f8cd0148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccffff640 1 -- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd00c7800 msgr2=0x7f8cd014c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccffff640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd00c7800 0x7f8cd014c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccffff640 1 -- 172.21.15.172:0/26928317 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc4009400 con 0x7f8cd00b5f10 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.376+0000 7f8ccf7fe640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd00c7800 0x7f8cd014c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.377+0000 7f8ccffff640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 0x7f8cd0156e00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000c390 tx=0x7f8cc000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.377+0000 7f8cccff9640 1 -- 172.21.15.172:0/26928317 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0005ce0 con 0x7f8cd00b5f10 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.377+0000 7f8cccff9640 1 -- 172.21.15.172:0/26928317 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0005e60 con 0x7f8cd00b5f10 2024-03-16T19:32:20.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.377+0000 7f8cccff9640 1 -- 172.21.15.172:0/26928317 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0007310 con 0x7f8cd00b5f10 2024-03-16T19:32:20.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.377+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd006b850 con 0x7f8cd00b5f10 2024-03-16T19:32:20.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.377+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd0159c90 con 0x7f8cd00b5f10 2024-03-16T19:32:20.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.378+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd00c83a0 con 0x7f8cd00b5f10 2024-03-16T19:32:20.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.379+0000 7f8cccff9640 1 -- 172.21.15.172:0/26928317 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8cc0007470 con 0x7f8cd00b5f10 2024-03-16T19:32:20.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.379+0000 7f8cccff9640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8cbc06b3e0 0x7f8cbc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.379+0000 7f8cccff9640 1 -- 172.21.15.172:0/26928317 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f8cc00891b0 con 0x7f8cd00b5f10 2024-03-16T19:32:20.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.379+0000 7f8ccf7fe640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8cbc06b3e0 0x7f8cbc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.380+0000 7f8ccf7fe640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8cbc06b3e0 0x7f8cbc06d8a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0068a80 tx=0x7f8cc4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:20.363 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.383+0000 7f8cccff9640 1 -- 172.21.15.172:0/26928317 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8cc0053e20 con 0x7f8cd00b5f10 2024-03-16T19:32:20.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.684+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8cd00672a0 con 0x7f8cd00b5f10 2024-03-16T19:32:20.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.684+0000 7f8cccff9640 1 -- 172.21.15.172:0/26928317 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8cc00574e0 con 0x7f8cd00b5f10 2024-03-16T19:32:20.664 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 32 2024-03-16T19:32:20.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8cbc06b3e0 msgr2=0x7f8cbc06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8cbc06b3e0 0x7f8cbc06d8a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0068a80 tx=0x7f8cc4008030 comp rx=0 tx=0).stop 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 msgr2=0x7f8cd0156e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 0x7f8cd0156e00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000c390 tx=0x7f8cc000c850 comp rx=0 tx=0).stop 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 shutdown_connections 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8cbc06b3e0 0x7f8cbc06d8a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8cd0154e00 0x7f8cd0148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cd00c7800 0x7f8cd014c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 --2- 172.21.15.172:0/26928317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8cd00b5f10 0x7f8cd0156e00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 >> 172.21.15.172:0/26928317 conn(0x7f8cd0066c20 msgr2=0x7f8cd00c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 shutdown_connections 2024-03-16T19:32:20.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.688+0000 7f8cd684e640 1 -- 172.21.15.172:0/26928317 wait complete. 2024-03-16T19:32:20.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T19:32:20.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.852+0000 7fada759e640 1 Processor -- start 2024-03-16T19:32:20.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.852+0000 7fada759e640 1 -- start start 2024-03-16T19:32:20.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80c8570 0x7fada8162df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada759e640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada80c79d0 con 0x7fada8151a30 2024-03-16T19:32:20.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada759e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada806c0c0 con 0x7fada80b2f00 2024-03-16T19:32:20.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada659c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fada8151a30 0x7fada8072890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39976/0 (socket says 172.21.15.172:39976) 2024-03-16T19:32:20.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada659c640 1 -- 172.21.15.172:0/1532535072 learned_addr learned my addr 172.21.15.172:0/1532535072 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada759e640 1 -- 172.21.15.172:0/1532535072 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada806cb50 con 0x7fada80c8570 2024-03-16T19:32:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada6d9d640 1 --2- 172.21.15.172:0/1532535072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80c8570 0x7fada8162df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada559a640 1 -- 172.21.15.172:0/1532535072 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051687002 0 0) 0x7fada806c0c0 con 0x7fada80b2f00 2024-03-16T19:32:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada559a640 1 -- 172.21.15.172:0/1532535072 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad90003520 con 0x7fada80b2f00 2024-03-16T19:32:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada559a640 1 -- 172.21.15.172:0/1532535072 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911288301 0 0) 0x7fada80c79d0 con 0x7fada8151a30 2024-03-16T19:32:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada559a640 1 -- 172.21.15.172:0/1532535072 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada806c0c0 con 0x7fada8151a30 2024-03-16T19:32:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada6d9d640 1 -- 172.21.15.172:0/1532535072 >> v1:172.21.15.174:6789/0 conn(0x7fada80b2f00 legacy=0x7fada80c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada6d9d640 1 -- 172.21.15.172:0/1532535072 >> v1:172.21.15.172:6789/0 conn(0x7fada8151a30 legacy=0x7fada8072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.854+0000 7fada6d9d640 1 -- 172.21.15.172:0/1532535072 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8157750 con 0x7fada80c8570 2024-03-16T19:32:20.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.855+0000 7fada6d9d640 1 --2- 172.21.15.172:0/1532535072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80c8570 0x7fada8162df0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fada0009dc0 tx=0x7fada002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d00a35835c592866 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:20.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.855+0000 7fada559a640 1 -- 172.21.15.172:0/1532535072 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada002ccd0 con 0x7fada80c8570 2024-03-16T19:32:20.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.855+0000 7fada559a640 1 -- 172.21.15.172:0/1532535072 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada002ce30 con 0x7fada80c8570 2024-03-16T19:32:20.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.855+0000 7fada559a640 1 -- 172.21.15.172:0/1532535072 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada0034660 con 0x7fada80c8570 2024-03-16T19:32:20.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.856+0000 7fada759e640 1 -- 172.21.15.172:0/1532535072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80c8570 msgr2=0x7fada8162df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.856+0000 7fada759e640 1 --2- 172.21.15.172:0/1532535072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80c8570 0x7fada8162df0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fada0009dc0 tx=0x7fada002bf20 comp rx=0 tx=0).stop 2024-03-16T19:32:20.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.856+0000 7fada759e640 1 -- 172.21.15.172:0/1532535072 shutdown_connections 2024-03-16T19:32:20.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.856+0000 7fada759e640 1 --2- 172.21.15.172:0/1532535072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80c8570 0x7fada8162df0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.856+0000 7fada759e640 1 -- 172.21.15.172:0/1532535072 >> 172.21.15.172:0/1532535072 conn(0x7fada80c9ad0 msgr2=0x7fada80c9ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:20.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.856+0000 7fada759e640 1 -- 172.21.15.172:0/1532535072 shutdown_connections 2024-03-16T19:32:20.837 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.856+0000 7fada759e640 1 -- 172.21.15.172:0/1532535072 wait complete. 2024-03-16T19:32:20.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.858+0000 7fada759e640 1 Processor -- start 2024-03-16T19:32:20.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.858+0000 7fada759e640 1 -- start start 2024-03-16T19:32:20.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.858+0000 7fada759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80b2f00 0x7fada81ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.858+0000 7fada759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada80c8570 0x7fada8144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada759e640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada8151a30 0x7fada8142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada759e640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada8156880 con 0x7fada8151a30 2024-03-16T19:32:20.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada759e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada81561c0 con 0x7fada80c8570 2024-03-16T19:32:20.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada759e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada8155380 con 0x7fada80b2f00 2024-03-16T19:32:20.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada6d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada8151a30 0x7fada8142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada659c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80b2f00 0x7fada81ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada6d9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada8151a30 0x7fada8142f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46612/0 (socket says 172.21.15.172:46612) 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada659c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80b2f00 0x7fada81ee5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44716/0 (socket says 172.21.15.172:44716) 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada6d9d640 1 -- 172.21.15.172:0/2213796359 learned_addr learned my addr 172.21.15.172:0/2213796359 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada5d9b640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada80c8570 0x7fada8144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada659c640 1 -- 172.21.15.172:0/2213796359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada80c8570 msgr2=0x7fada8144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada659c640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada80c8570 0x7fada8144d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada659c640 1 -- 172.21.15.172:0/2213796359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada8151a30 msgr2=0x7fada8142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada659c640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada8151a30 0x7fada8142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada659c640 1 -- 172.21.15.172:0/2213796359 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada0009a70 con 0x7fada80b2f00 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fada659c640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80b2f00 0x7fada81ee5b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fad9800c3e0 tx=0x7fad9800c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fad96ffd640 1 -- 172.21.15.172:0/2213796359 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad980074e0 con 0x7fada80b2f00 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fad96ffd640 1 -- 172.21.15.172:0/2213796359 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad98007640 con 0x7fada80b2f00 2024-03-16T19:32:20.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.859+0000 7fad96ffd640 1 -- 172.21.15.172:0/2213796359 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98004a00 con 0x7fada80b2f00 2024-03-16T19:32:20.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.860+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada8155640 con 0x7fada80b2f00 2024-03-16T19:32:20.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.860+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada8155c00 con 0x7fada80b2f00 2024-03-16T19:32:20.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.861+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada80758f0 con 0x7fada80b2f00 2024-03-16T19:32:20.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.861+0000 7fad96ffd640 1 -- 172.21.15.172:0/2213796359 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fad98004b60 con 0x7fada80b2f00 2024-03-16T19:32:20.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.862+0000 7fad96ffd640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8806b3e0 0x7fad8806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:20.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.862+0000 7fad96ffd640 1 -- 172.21.15.172:0/2213796359 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fad980898a0 con 0x7fada80b2f00 2024-03-16T19:32:20.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.862+0000 7fada5d9b640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8806b3e0 0x7fad8806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:20.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.862+0000 7fada5d9b640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8806b3e0 0x7fad8806d8a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fada8073450 tx=0x7fad9c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:20.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:20.866+0000 7fad96ffd640 1 -- 172.21.15.172:0/2213796359 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad980545a0 con 0x7fada80b2f00 2024-03-16T19:32:21.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.156+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fada806c810 con 0x7fada80b2f00 2024-03-16T19:32:21.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.157+0000 7fad96ffd640 1 -- 172.21.15.172:0/2213796359 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7fad98057c60 con 0x7fada80b2f00 2024-03-16T19:32:21.137 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 64 2024-03-16T19:32:21.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8806b3e0 msgr2=0x7fad8806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:21.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8806b3e0 0x7fad8806d8a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fada8073450 tx=0x7fad9c009040 comp rx=0 tx=0).stop 2024-03-16T19:32:21.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80b2f00 msgr2=0x7fada81ee5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:21.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80b2f00 0x7fada81ee5b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fad9800c3e0 tx=0x7fad9800c8a0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 shutdown_connections 2024-03-16T19:32:21.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fada8151a30 0x7fada8142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fada80c8570 0x7fada8144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fad8806b3e0 0x7fad8806d8a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 --2- 172.21.15.172:0/2213796359 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fada80b2f00 0x7fada81ee5b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 >> 172.21.15.172:0/2213796359 conn(0x7fada80c9ad0 msgr2=0x7fada814f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:21.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 shutdown_connections 2024-03-16T19:32:21.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.160+0000 7fada759e640 1 -- 172.21.15.172:0/2213796359 wait complete. 2024-03-16T19:32:21.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T19:32:21.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99f5df4640 1 Processor -- start 2024-03-16T19:32:21.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99f5df4640 1 -- start start 2024-03-16T19:32:21.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99f5df4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f00c7590 0x7f99f01648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:21.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99f5df4640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f99f0073180 0x7f99f0164e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:21.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99f5df4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f00c7d20 con 0x7f99f00c71b0 2024-03-16T19:32:21.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99f5df4640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f00c7fc0 con 0x7f99f0073180 2024-03-16T19:32:21.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99f5df4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f00c82a0 con 0x7f99f00c7590 2024-03-16T19:32:21.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99effff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f99f0073180 0x7f99f0164e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:21.314 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99ef7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f99f00c71b0 0x7f99f006ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39986/0 (socket says 172.21.15.172:39986) 2024-03-16T19:32:21.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.323+0000 7f99effff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f99f0073180 0x7f99f0164e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46624/0 (socket says 172.21.15.172:46624) 2024-03-16T19:32:21.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.324+0000 7f99ef7fe640 1 -- 172.21.15.172:0/2420127963 learned_addr learned my addr 172.21.15.172:0/2420127963 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:21.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.324+0000 7f99eeffd640 1 --2- 172.21.15.172:0/2420127963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f00c7590 0x7f99f01648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:21.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.334+0000 7f99effff640 1 -- 172.21.15.172:0/2420127963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f00c7590 msgr2=0x7f99f01648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:21.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.334+0000 7f99effff640 1 --2- 172.21.15.172:0/2420127963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f00c7590 0x7f99f01648e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.334+0000 7f99effff640 1 -- 172.21.15.172:0/2420127963 >> v1:172.21.15.172:6789/0 conn(0x7f99f00c71b0 legacy=0x7f99f006ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:21.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.334+0000 7f99effff640 1 -- 172.21.15.172:0/2420127963 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00c8580 con 0x7f99f0073180 2024-03-16T19:32:21.315 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.335+0000 7f99effff640 1 --2- 172.21.15.172:0/2420127963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f0073180 0x7f99f0164e20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f99e8007d60 tx=0x7f99e802d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d9cab059b00148 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:21.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.337+0000 7f99ee7fc640 1 -- 172.21.15.172:0/2420127963 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e802ddf0 con 0x7f99f0073180 2024-03-16T19:32:21.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.337+0000 7f99ee7fc640 1 -- 172.21.15.172:0/2420127963 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e8035440 con 0x7f99f0073180 2024-03-16T19:32:21.316 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.337+0000 7f99ee7fc640 1 -- 172.21.15.172:0/2420127963 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e80357d0 con 0x7f99f0073180 2024-03-16T19:32:21.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.338+0000 7f99f5df4640 1 -- 172.21.15.172:0/2420127963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f0073180 msgr2=0x7f99f0164e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:21.317 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.338+0000 7f99f5df4640 1 --2- 172.21.15.172:0/2420127963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f0073180 0x7f99f0164e20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f99e8007d60 tx=0x7f99e802d040 comp rx=0 tx=0).stop 2024-03-16T19:32:21.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.338+0000 7f99f5df4640 1 -- 172.21.15.172:0/2420127963 shutdown_connections 2024-03-16T19:32:21.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.338+0000 7f99f5df4640 1 --2- 172.21.15.172:0/2420127963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f00c7590 0x7f99f01648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.338+0000 7f99f5df4640 1 --2- 172.21.15.172:0/2420127963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f0073180 0x7f99f0164e20 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.338+0000 7f99f5df4640 1 -- 172.21.15.172:0/2420127963 >> 172.21.15.172:0/2420127963 conn(0x7f99f00cc030 msgr2=0x7f99f00cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:21.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.338+0000 7f99f5df4640 1 -- 172.21.15.172:0/2420127963 shutdown_connections 2024-03-16T19:32:21.318 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.338+0000 7f99f5df4640 1 -- 172.21.15.172:0/2420127963 wait complete. 2024-03-16T19:32:21.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.340+0000 7f99f5df4640 1 Processor -- start 2024-03-16T19:32:21.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.340+0000 7f99f5df4640 1 -- start start 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99f5df4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f0073180 0x7f99f01f1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99f5df4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f00c71b0 0x7f99f02052a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99f5df4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f00c7590 0x7f99f0201a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99eeffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f00c71b0 0x7f99f02052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99eeffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f00c71b0 0x7f99f02052a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46636/0 (socket says 172.21.15.172:46636) 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f0073180 0x7f99f01f1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99eeffd640 1 -- 172.21.15.172:0/1459797616 learned_addr learned my addr 172.21.15.172:0/1459797616 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f00b1180 con 0x7f99f00c71b0 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0204b70 con 0x7f99f00c7590 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0204e10 con 0x7f99f0073180 2024-03-16T19:32:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.341+0000 7f99effff640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f00c7590 0x7f99f0201a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99eeffd640 1 -- 172.21.15.172:0/1459797616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f0073180 msgr2=0x7f99f01f1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99eeffd640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f0073180 0x7f99f01f1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99eeffd640 1 -- 172.21.15.172:0/1459797616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f00c7590 msgr2=0x7f99f0201a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99eeffd640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f00c7590 0x7f99f0201a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99eeffd640 1 -- 172.21.15.172:0/1459797616 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e80079c0 con 0x7f99f00c71b0 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99eeffd640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f00c71b0 0x7f99f02052a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f99e400e9b0 tx=0x7f99e400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99d7fff640 1 -- 172.21.15.172:0/1459797616 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e4009670 con 0x7f99f00c71b0 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99d7fff640 1 -- 172.21.15.172:0/1459797616 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e400c9a0 con 0x7f99f00c71b0 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99d7fff640 1 -- 172.21.15.172:0/1459797616 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e400cd50 con 0x7f99f00c71b0 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f0169f50 con 0x7f99f00c71b0 2024-03-16T19:32:21.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.342+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f016a4d0 con 0x7f99f00c71b0 2024-03-16T19:32:21.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.344+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d0005290 con 0x7f99f00c71b0 2024-03-16T19:32:21.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.344+0000 7f99d7fff640 1 -- 172.21.15.172:0/1459797616 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f99e4013470 con 0x7f99f00c71b0 2024-03-16T19:32:21.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.344+0000 7f99d7fff640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99dc06b3b0 0x7f99dc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:21.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.345+0000 7f99d7fff640 1 -- 172.21.15.172:0/1459797616 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f99e4089dd0 con 0x7f99f00c71b0 2024-03-16T19:32:21.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.345+0000 7f99ef7fe640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99dc06b3b0 0x7f99dc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:21.325 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.345+0000 7f99ef7fe640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99dc06b3b0 0x7f99dc06d870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f99e0007fd0 tx=0x7f99e0007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:21.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.351+0000 7f99d7fff640 1 -- 172.21.15.172:0/1459797616 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99e4054a40 con 0x7f99f00c71b0 2024-03-16T19:32:21.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.626+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f99d0005810 con 0x7f99f00c71b0 2024-03-16T19:32:21.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.627+0000 7f99d7fff640 1 -- 172.21.15.172:0/1459797616 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v106) v1 ==== 80+0+0 (secure 0 0 0) 0x7f99e4017080 con 0x7f99f00c71b0 2024-03-16T19:32:21.607 INFO:teuthology.orchestra.run.smithi172.stderr:listed 0 entries 2024-03-16T19:32:21.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99dc06b3b0 msgr2=0x7f99dc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:21.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99dc06b3b0 0x7f99dc06d870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f99e0007fd0 tx=0x7f99e0007e80 comp rx=0 tx=0).stop 2024-03-16T19:32:21.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f00c71b0 msgr2=0x7f99f02052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:21.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f00c71b0 0x7f99f02052a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f99e400e9b0 tx=0x7f99e400ee70 comp rx=0 tx=0).stop 2024-03-16T19:32:21.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 shutdown_connections 2024-03-16T19:32:21.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f00c7590 0x7f99f0201a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99f00c71b0 0x7f99f02052a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f99dc06b3b0 0x7f99dc06d870 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 --2- 172.21.15.172:0/1459797616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99f0073180 0x7f99f01f1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:21.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 >> 172.21.15.172:0/1459797616 conn(0x7f99f00cc030 msgr2=0x7f99f0164bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:21.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 shutdown_connections 2024-03-16T19:32:21.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:21.629+0000 7f99f5df4640 1 -- 172.21.15.172:0/1459797616 wait complete. 2024-03-16T19:32:21.620 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-16T19:32:21.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:32:21.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-16T19:32:21.660 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi172.front.sepia.ceph.com 2024-03-16T19:32:21.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:32:21.660 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-16T19:32:21.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:32:21.858 DEBUG:teuthology.orchestra.run.smithi172:> ip netns list 2024-03-16T19:32:21.875 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-16T19:32:21.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:32:21.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T19:32:21.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T19:32:21.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo ip link delete ceph-brx 2024-03-16T19:32:22.043 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-16T19:32:22.053 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-16T19:32:22.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.206+0000 7f1237edf640 1 Processor -- start 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.206+0000 7f1237edf640 1 -- start start 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.207+0000 7f1237edf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c9650 0x7f123006bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.207+0000 7f1237edf640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123006b630 con 0x7f123016aeb0 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.207+0000 7f1237edf640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12300c1b70 con 0x7f12300732d0 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.207+0000 7f1237edf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12300c25b0 con 0x7f12300c9650 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.207+0000 7f1235c7b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f123016aeb0 0x7f123016b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39994/0 (socket says 172.21.15.172:39994) 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.207+0000 7f1235c7b640 1 -- 172.21.15.172:0/3828424263 learned_addr learned my addr 172.21.15.172:0/3828424263 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.207+0000 7f123647c640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c9650 0x7f123006bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.220 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.208+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919255064 0 0) 0x7f123006b630 con 0x7f123016aeb0 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f121c003520 con 0x7f123016aeb0 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694197843 0 0) 0x7f12300c1b70 con 0x7f12300732d0 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123006b630 con 0x7f12300732d0 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 <== mon.0 v1:172.21.15.172:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428133561 0 0) 0x7f121c003520 con 0x7f123016aeb0 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c9650 msgr2=0x7f123006bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c9650 0x7f123006bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 >> v1:172.21.15.174:6789/0 conn(0x7f12300732d0 legacy=0x7f12301648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 --> v1:172.21.15.172:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12300c1e50 con 0x7f123016aeb0 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 <== mon.0 v1:172.21.15.172:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f1224001670 con 0x7f123016aeb0 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f123647c640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c9650 0x7f123006bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 >> v1:172.21.15.172:6789/0 conn(0x7f123016aeb0 legacy=0x7f123016b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f121c007200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c9650 0x7f121c009800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f121c009dd0 0x7f121c00c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.221 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c001230 con 0x7f121c009dd0 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12300c25b0 con 0x7f121c004e10 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121c00cac0 con 0x7f12300c9650 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f123647c640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f121c009dd0 0x7f121c00c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f1235c7b640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f121c007200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.238+0000 7f123547a640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c9650 0x7f121c009800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.239+0000 7f1235c7b640 1 -- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c9650 msgr2=0x7f121c009800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.239+0000 7f1235c7b640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c9650 0x7f121c009800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.239+0000 7f1235c7b640 1 -- 172.21.15.172:0/3828424263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f121c009dd0 msgr2=0x7f121c00c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.239+0000 7f1235c7b640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f121c009dd0 0x7f121c00c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.239+0000 7f1235c7b640 1 -- 172.21.15.172:0/3828424263 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1228009a80 con 0x7f121c004e10 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.239+0000 7f123647c640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f121c009dd0 0x7f121c00c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:22.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.240+0000 7f1235c7b640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f121c007200 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f122400c160 tx=0x7f121c0037d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc5694545ef18004 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:22.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.240+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224005ad0 con 0x7f121c004e10 2024-03-16T19:32:22.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.240+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1224005c50 con 0x7f121c004e10 2024-03-16T19:32:22.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.240+0000 7f1234c79640 1 -- 172.21.15.172:0/3828424263 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122402c850 con 0x7f121c004e10 2024-03-16T19:32:22.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.241+0000 7f1237edf640 1 -- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 msgr2=0x7f121c007200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.241+0000 7f1237edf640 1 --2- 172.21.15.172:0/3828424263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f121c007200 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f122400c160 tx=0x7f121c0037d0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.241+0000 7f123647c640 1 -- 172.21.15.172:0/3828424263 reap_dead start 2024-03-16T19:32:22.225 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.241+0000 7f1237edf640 1 -- 172.21.15.172:0/3828424263 shutdown_connections 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.241+0000 7f1237edf640 1 -- 172.21.15.172:0/3828424263 >> 172.21.15.172:0/3828424263 conn(0x7f12300c8b30 msgr2=0x7f12300c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.241+0000 7f1237edf640 1 -- 172.21.15.172:0/3828424263 shutdown_connections 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.241+0000 7f1237edf640 1 -- 172.21.15.172:0/3828424263 wait complete. 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.243+0000 7f1237edf640 1 Processor -- start 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.243+0000 7f1237edf640 1 -- start start 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f1237edf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f12300c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f1237edf640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12300c4dc0 0x7f123015fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f1237edf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c5e10 0x7f123015f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f1237edf640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12300c56c0 con 0x7f12300c4dc0 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f1237edf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1230165b70 con 0x7f121c004e10 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f1237edf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12300c5980 con 0x7f12300c5e10 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f123547a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12300c4dc0 0x7f123015fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f1235c7b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f12300c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f1235c7b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f12300c3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:54220/0 (socket says 172.21.15.172:54220) 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f1235c7b640 1 -- 172.21.15.172:0/3074636271 learned_addr learned my addr 172.21.15.172:0/3074636271 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.244+0000 7f123647c640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c5e10 0x7f123015f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f1235c7b640 1 -- 172.21.15.172:0/3074636271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c5e10 msgr2=0x7f123015f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f1235c7b640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c5e10 0x7f123015f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f1235c7b640 1 -- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12300c4dc0 msgr2=0x7f123015fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f1235c7b640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12300c4dc0 0x7f123015fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f1235c7b640 1 -- 172.21.15.172:0/3074636271 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12302009f0 con 0x7f121c004e10 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f123547a640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12300c4dc0 0x7f123015fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f1235c7b640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f12300c3e40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f122400c160 tx=0x7f1224001980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f12227fc640 1 -- 172.21.15.172:0/3074636271 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224007270 con 0x7f121c004e10 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f12227fc640 1 -- 172.21.15.172:0/3074636271 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12240073d0 con 0x7f121c004e10 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1230165dc0 con 0x7f121c004e10 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f12227fc640 1 -- 172.21.15.172:0/3074636271 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224007760 con 0x7f121c004e10 2024-03-16T19:32:22.227 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.245+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1230166380 con 0x7f121c004e10 2024-03-16T19:32:22.228 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.246+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12300681c0 con 0x7f121c004e10 2024-03-16T19:32:22.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.247+0000 7f12227fc640 1 -- 172.21.15.172:0/3074636271 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1224032070 con 0x7f121c004e10 2024-03-16T19:32:22.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.247+0000 7f12227fc640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f121406b3b0 0x7f121406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.247+0000 7f12227fc640 1 -- 172.21.15.172:0/3074636271 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f12240ab760 con 0x7f121c004e10 2024-03-16T19:32:22.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.247+0000 7f123547a640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f121406b3b0 0x7f121406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.248+0000 7f123547a640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f121406b3b0 0x7f121406d870 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f12300cc000 tx=0x7f122c013040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:22.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.251+0000 7f12227fc640 1 -- 172.21.15.172:0/3074636271 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1224076340 con 0x7f121c004e10 2024-03-16T19:32:22.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.515+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12300c2130 con 0x7f121406b3b0 2024-03-16T19:32:22.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.521+0000 7f12227fc640 1 -- 172.21.15.172:0/3074636271 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252328 (secure 0 0 0) 0x7f122c017040 con 0x7f121406b3b0 2024-03-16T19:32:22.503 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:22.505 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:22.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f121406b3b0 msgr2=0x7f121406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f121406b3b0 0x7f121406d870 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f12300cc000 tx=0x7f122c013040 comp rx=0 tx=0).stop 2024-03-16T19:32:22.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 msgr2=0x7f12300c3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f12300c3e40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f122400c160 tx=0x7f1224001980 comp rx=0 tx=0).stop 2024-03-16T19:32:22.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 shutdown_connections 2024-03-16T19:32:22.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f121406b3b0 0x7f121406d870 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12300c5e10 0x7f123015f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.506 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12300c4dc0 0x7f123015fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 --2- 172.21.15.172:0/3074636271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f121c004e10 0x7f12300c3e40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 >> 172.21.15.172:0/3074636271 conn(0x7f12300c8b30 msgr2=0x7f1230160cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:22.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 shutdown_connections 2024-03-16T19:32:22.507 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.525+0000 7f1237edf640 1 -- 172.21.15.172:0/3074636271 wait complete. 2024-03-16T19:32:22.517 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-03-16T19:32:22.233797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630862,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":245,"num_write_kb":2489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228480,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":217199,"kb_avail":754746240,"statfs":{"total":773094113280,"available":772860149760,"internally_reserved":0,"allocated":11427840,"data_stored":6022908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":222412203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002377"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937971+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:08.937971+0000","last_peered":"2024-03-16T19:32:08.937971+0000","last_clean":"2024-03-16T19:32:08.937971+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:08.937971+0000","last_undegraded":"2024-03-16T19:32:08.937971+0000","last_fullsized":"2024-03-16T19:32:08.937971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094518+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:08.094518+0000","last_peered":"2024-03-16T19:32:08.094518+0000","last_clean":"2024-03-16T19:32:08.094518+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:08.094518+0000","last_undegraded":"2024-03-16T19:32:08.094518+0000","last_fullsized":"2024-03-16T19:32:08.094518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351343+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:08.351343+0000","last_peered":"2024-03-16T19:32:08.351343+0000","last_clean":"2024-03-16T19:32:08.351343+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:08.351343+0000","last_undegraded":"2024-03-16T19:32:08.351343+0000","last_fullsized":"2024-03-16T19:32:08.351343+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:47.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083818+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:08.083818+0000","last_peered":"2024-03-16T19:32:08.083818+0000","last_clean":"2024-03-16T19:32:08.083818+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:08.083818+0000","last_undegraded":"2024-03-16T19:32:08.083818+0000","last_fullsized":"2024-03-16T19:32:08.083818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937685+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:08.937685+0000","last_peered":"2024-03-16T19:32:08.937685+0000","last_clean":"2024-03-16T19:32:08.937685+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:08.937685+0000","last_undegraded":"2024-03-16T19:32:08.937685+0000","last_fullsized":"2024-03-16T19:32:08.937685+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083640+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:08.083640+0000","last_peered":"2024-03-16T19:32:08.083640+0000","last_clean":"2024-03-16T19:32:08.083640+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:08.083640+0000","last_undegraded":"2024-03-16T19:32:08.083640+0000","last_fullsized":"2024-03-16T19:32:08.083640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937556+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:08.937556+0000","last_peered":"2024-03-16T19:32:08.937556+0000","last_clean":"2024-03-16T19:32:08.937556+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:08.937556+0000","last_undegraded":"2024-03-16T19:32:08.937556+0000","last_fullsized":"2024-03-16T19:32:08.937556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086880+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:08.086880+0000","last_peered":"2024-03-16T19:32:08.086880+0000","last_clean":"2024-03-16T19:32:08.086880+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:08.086880+0000","last_undegraded":"2024-03-16T19:32:08.086880+0000","last_fullsized":"2024-03-16T19:32:08.086880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:58:31.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937978+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:08.937978+0000","last_peered":"2024-03-16T19:32:08.937978+0000","last_clean":"2024-03-16T19:32:08.937978+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:08.937978+0000","last_undegraded":"2024-03-16T19:32:08.937978+0000","last_fullsized":"2024-03-16T19:32:08.937978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287814+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:09.287814+0000","last_peered":"2024-03-16T19:32:09.287814+0000","last_clean":"2024-03-16T19:32:09.287814+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:09.287814+0000","last_undegraded":"2024-03-16T19:32:09.287814+0000","last_fullsized":"2024-03-16T19:32:09.287814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287704+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:09.287704+0000","last_peered":"2024-03-16T19:32:09.287704+0000","last_clean":"2024-03-16T19:32:09.287704+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:09.287704+0000","last_undegraded":"2024-03-16T19:32:09.287704+0000","last_fullsized":"2024-03-16T19:32:09.287704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084143+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:08.084143+0000","last_peered":"2024-03-16T19:32:08.084143+0000","last_clean":"2024-03-16T19:32:08.084143+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:08.084143+0000","last_undegraded":"2024-03-16T19:32:08.084143+0000","last_fullsized":"2024-03-16T19:32:08.084143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088886+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:08.088886+0000","last_peered":"2024-03-16T19:32:08.088886+0000","last_clean":"2024-03-16T19:32:08.088886+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:08.088886+0000","last_undegraded":"2024-03-16T19:32:08.088886+0000","last_fullsized":"2024-03-16T19:32:08.088886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:52.102388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"84'3","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350888+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:08.350888+0000","last_peered":"2024-03-16T19:32:08.350888+0000","last_clean":"2024-03-16T19:32:08.350888+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:08.350888+0000","last_undegraded":"2024-03-16T19:32:08.350888+0000","last_fullsized":"2024-03-16T19:32:08.350888+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:39.289238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094126+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:08.094126+0000","last_peered":"2024-03-16T19:32:08.094126+0000","last_clean":"2024-03-16T19:32:08.094126+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:08.094126+0000","last_undegraded":"2024-03-16T19:32:08.094126+0000","last_fullsized":"2024-03-16T19:32:08.094126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083403+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:08.083403+0000","last_peered":"2024-03-16T19:32:08.083403+0000","last_clean":"2024-03-16T19:32:08.083403+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:08.083403+0000","last_undegraded":"2024-03-16T19:32:08.083403+0000","last_fullsized":"2024-03-16T19:32:08.083403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084254+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:08.084254+0000","last_peered":"2024-03-16T19:32:08.084254+0000","last_clean":"2024-03-16T19:32:08.084254+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:08.084254+0000","last_undegraded":"2024-03-16T19:32:08.084254+0000","last_fullsized":"2024-03-16T19:32:08.084254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087017+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:08.087017+0000","last_peered":"2024-03-16T19:32:08.087017+0000","last_clean":"2024-03-16T19:32:08.087017+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:08.087017+0000","last_undegraded":"2024-03-16T19:32:08.087017+0000","last_fullsized":"2024-03-16T19:32:08.087017+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:36:06.175327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937878+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:08.937878+0000","last_peered":"2024-03-16T19:32:08.937878+0000","last_clean":"2024-03-16T19:32:08.937878+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:08.937878+0000","last_undegraded":"2024-03-16T19:32:08.937878+0000","last_fullsized":"2024-03-16T19:32:08.937878+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287690+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:09.287690+0000","last_peered":"2024-03-16T19:32:09.287690+0000","last_clean":"2024-03-16T19:32:09.287690+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:09.287690+0000","last_undegraded":"2024-03-16T19:32:09.287690+0000","last_fullsized":"2024-03-16T19:32:09.287690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351462+0000","last_change":"2024-03-16T19:31:51.807295+0000","last_active":"2024-03-16T19:32:08.351462+0000","last_peered":"2024-03-16T19:32:08.351462+0000","last_clean":"2024-03-16T19:32:08.351462+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:08.351462+0000","last_undegraded":"2024-03-16T19:32:08.351462+0000","last_fullsized":"2024-03-16T19:32:08.351462+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:04.578640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.085957+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:08.085957+0000","last_peered":"2024-03-16T19:32:08.085957+0000","last_clean":"2024-03-16T19:32:08.085957+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:08.085957+0000","last_undegraded":"2024-03-16T19:32:08.085957+0000","last_fullsized":"2024-03-16T19:32:08.085957+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:41:06.377004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088245+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:08.088245+0000","last_peered":"2024-03-16T19:32:08.088245+0000","last_clean":"2024-03-16T19:32:08.088245+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:08.088245+0000","last_undegraded":"2024-03-16T19:32:08.088245+0000","last_fullsized":"2024-03-16T19:32:08.088245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:47:08.251341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083454+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:08.083454+0000","last_peered":"2024-03-16T19:32:08.083454+0000","last_clean":"2024-03-16T19:32:08.083454+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:08.083454+0000","last_undegraded":"2024-03-16T19:32:08.083454+0000","last_fullsized":"2024-03-16T19:32:08.083454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086134+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:08.086134+0000","last_peered":"2024-03-16T19:32:08.086134+0000","last_clean":"2024-03-16T19:32:08.086134+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:08.086134+0000","last_undegraded":"2024-03-16T19:32:08.086134+0000","last_fullsized":"2024-03-16T19:32:08.086134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:10.818175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287911+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:09.287911+0000","last_peered":"2024-03-16T19:32:09.287911+0000","last_clean":"2024-03-16T19:32:09.287911+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:09.287911+0000","last_undegraded":"2024-03-16T19:32:09.287911+0000","last_fullsized":"2024-03-16T19:32:09.287911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":187,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093433+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:08.093433+0000","last_peered":"2024-03-16T19:32:08.093433+0000","last_clean":"2024-03-16T19:32:08.093433+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:08.093433+0000","last_undegraded":"2024-03-16T19:32:08.093433+0000","last_fullsized":"2024-03-16T19:32:08.093433+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087892+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:08.087892+0000","last_peered":"2024-03-16T19:32:08.087892+0000","last_clean":"2024-03-16T19:32:08.087892+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:08.087892+0000","last_undegraded":"2024-03-16T19:32:08.087892+0000","last_fullsized":"2024-03-16T19:32:08.087892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:12:08.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350890+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:08.350890+0000","last_peered":"2024-03-16T19:32:08.350890+0000","last_clean":"2024-03-16T19:32:08.350890+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:08.350890+0000","last_undegraded":"2024-03-16T19:32:08.350890+0000","last_fullsized":"2024-03-16T19:32:08.350890+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:12.646493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083749+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:08.083749+0000","last_peered":"2024-03-16T19:32:08.083749+0000","last_clean":"2024-03-16T19:32:08.083749+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:08.083749+0000","last_undegraded":"2024-03-16T19:32:08.083749+0000","last_fullsized":"2024-03-16T19:32:08.083749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087224+0000","last_change":"2024-03-16T19:31:49.974297+0000","last_active":"2024-03-16T19:32:08.087224+0000","last_peered":"2024-03-16T19:32:08.087224+0000","last_clean":"2024-03-16T19:32:08.087224+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:08.087224+0000","last_undegraded":"2024-03-16T19:32:08.087224+0000","last_fullsized":"2024-03-16T19:32:08.087224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:51:57.535372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287591+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:09.287591+0000","last_peered":"2024-03-16T19:32:09.287591+0000","last_clean":"2024-03-16T19:32:09.287591+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:09.287591+0000","last_undegraded":"2024-03-16T19:32:09.287591+0000","last_fullsized":"2024-03-16T19:32:09.287591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084174+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:32:08.084174+0000","last_peered":"2024-03-16T19:32:08.084174+0000","last_clean":"2024-03-16T19:32:08.084174+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:08.084174+0000","last_undegraded":"2024-03-16T19:32:08.084174+0000","last_fullsized":"2024-03-16T19:32:08.084174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089011+0000","last_change":"2024-03-16T19:31:26.808032+0000","last_active":"2024-03-16T19:32:08.089011+0000","last_peered":"2024-03-16T19:32:08.089011+0000","last_clean":"2024-03-16T19:32:08.089011+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:08.089011+0000","last_undegraded":"2024-03-16T19:32:08.089011+0000","last_fullsized":"2024-03-16T19:32:08.089011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:15.218778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351569+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:08.351569+0000","last_peered":"2024-03-16T19:32:08.351569+0000","last_clean":"2024-03-16T19:32:08.351569+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:08.351569+0000","last_undegraded":"2024-03-16T19:32:08.351569+0000","last_fullsized":"2024-03-16T19:32:08.351569+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:16:31.606469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287611+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:09.287611+0000","last_peered":"2024-03-16T19:32:09.287611+0000","last_clean":"2024-03-16T19:32:09.287611+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:09.287611+0000","last_undegraded":"2024-03-16T19:32:09.287611+0000","last_fullsized":"2024-03-16T19:32:09.287611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086078+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086078+0000","last_peered":"2024-03-16T19:32:08.086078+0000","last_clean":"2024-03-16T19:32:08.086078+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086078+0000","last_undegraded":"2024-03-16T19:32:08.086078+0000","last_fullsized":"2024-03-16T19:32:08.086078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:23:19.824904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086998+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:08.086998+0000","last_peered":"2024-03-16T19:32:08.086998+0000","last_clean":"2024-03-16T19:32:08.086998+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:08.086998+0000","last_undegraded":"2024-03-16T19:32:08.086998+0000","last_fullsized":"2024-03-16T19:32:08.086998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:45.782085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094004+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:08.094004+0000","last_peered":"2024-03-16T19:32:08.094004+0000","last_clean":"2024-03-16T19:32:08.094004+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:08.094004+0000","last_undegraded":"2024-03-16T19:32:08.094004+0000","last_fullsized":"2024-03-16T19:32:08.094004+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287793+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:09.287793+0000","last_peered":"2024-03-16T19:32:09.287793+0000","last_clean":"2024-03-16T19:32:09.287793+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:09.287793+0000","last_undegraded":"2024-03-16T19:32:09.287793+0000","last_fullsized":"2024-03-16T19:32:09.287793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083754+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:08.083754+0000","last_peered":"2024-03-16T19:32:08.083754+0000","last_clean":"2024-03-16T19:32:08.083754+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:08.083754+0000","last_undegraded":"2024-03-16T19:32:08.083754+0000","last_fullsized":"2024-03-16T19:32:08.083754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351532+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:08.351532+0000","last_peered":"2024-03-16T19:32:08.351532+0000","last_clean":"2024-03-16T19:32:08.351532+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:08.351532+0000","last_undegraded":"2024-03-16T19:32:08.351532+0000","last_fullsized":"2024-03-16T19:32:08.351532+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:16.418962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093565+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:08.093565+0000","last_peered":"2024-03-16T19:32:08.093565+0000","last_clean":"2024-03-16T19:32:08.093565+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:08.093565+0000","last_undegraded":"2024-03-16T19:32:08.093565+0000","last_fullsized":"2024-03-16T19:32:08.093565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938817+0000","last_change":"2024-03-16T19:32:08.938817+0000","last_active":"2024-03-16T19:32:08.938817+0000","last_peered":"2024-03-16T19:32:08.938817+0000","last_clean":"2024-03-16T19:32:08.938817+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:08.938817+0000","last_undegraded":"2024-03-16T19:32:08.938817+0000","last_fullsized":"2024-03-16T19:32:08.938817+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:39:32.174855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"86'23","reported_seq":233,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287734+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:09.287734+0000","last_peered":"2024-03-16T19:32:09.287734+0000","last_clean":"2024-03-16T19:32:09.287734+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:09.287734+0000","last_undegraded":"2024-03-16T19:32:09.287734+0000","last_fullsized":"2024-03-16T19:32:09.287734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39292,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937730+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:08.937730+0000","last_peered":"2024-03-16T19:32:08.937730+0000","last_clean":"2024-03-16T19:32:08.937730+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:08.937730+0000","last_undegraded":"2024-03-16T19:32:08.937730+0000","last_fullsized":"2024-03-16T19:32:08.937730+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351564+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:08.351564+0000","last_peered":"2024-03-16T19:32:08.351564+0000","last_clean":"2024-03-16T19:32:08.351564+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:08.351564+0000","last_undegraded":"2024-03-16T19:32:08.351564+0000","last_fullsized":"2024-03-16T19:32:08.351564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:59.818393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351224+0000","last_change":"2024-03-16T19:31:51.807895+0000","last_active":"2024-03-16T19:32:08.351224+0000","last_peered":"2024-03-16T19:32:08.351224+0000","last_clean":"2024-03-16T19:32:08.351224+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:08.351224+0000","last_undegraded":"2024-03-16T19:32:08.351224+0000","last_fullsized":"2024-03-16T19:32:08.351224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:25.988237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937695+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:08.937695+0000","last_peered":"2024-03-16T19:32:08.937695+0000","last_clean":"2024-03-16T19:32:08.937695+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:08.937695+0000","last_undegraded":"2024-03-16T19:32:08.937695+0000","last_fullsized":"2024-03-16T19:32:08.937695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287610+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:09.287610+0000","last_peered":"2024-03-16T19:32:09.287610+0000","last_clean":"2024-03-16T19:32:09.287610+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:09.287610+0000","last_undegraded":"2024-03-16T19:32:09.287610+0000","last_fullsized":"2024-03-16T19:32:09.287610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287595+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:09.287595+0000","last_peered":"2024-03-16T19:32:09.287595+0000","last_clean":"2024-03-16T19:32:09.287595+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:09.287595+0000","last_undegraded":"2024-03-16T19:32:09.287595+0000","last_fullsized":"2024-03-16T19:32:09.287595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087374+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:08.087374+0000","last_peered":"2024-03-16T19:32:08.087374+0000","last_clean":"2024-03-16T19:32:08.087374+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:08.087374+0000","last_undegraded":"2024-03-16T19:32:08.087374+0000","last_fullsized":"2024-03-16T19:32:08.087374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:33.634146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937562+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:08.937562+0000","last_peered":"2024-03-16T19:32:08.937562+0000","last_clean":"2024-03-16T19:32:08.937562+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:08.937562+0000","last_undegraded":"2024-03-16T19:32:08.937562+0000","last_fullsized":"2024-03-16T19:32:08.937562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084319+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084319+0000","last_peered":"2024-03-16T19:32:08.084319+0000","last_clean":"2024-03-16T19:32:08.084319+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084319+0000","last_undegraded":"2024-03-16T19:32:08.084319+0000","last_fullsized":"2024-03-16T19:32:08.084319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:53.021797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937869+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:08.937869+0000","last_peered":"2024-03-16T19:32:08.937869+0000","last_clean":"2024-03-16T19:32:08.937869+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:08.937869+0000","last_undegraded":"2024-03-16T19:32:08.937869+0000","last_fullsized":"2024-03-16T19:32:08.937869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":55,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086917+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:08.086917+0000","last_peered":"2024-03-16T19:32:08.086917+0000","last_clean":"2024-03-16T19:32:08.086917+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:08.086917+0000","last_undegraded":"2024-03-16T19:32:08.086917+0000","last_fullsized":"2024-03-16T19:32:08.086917+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:07:39.853290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938036+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:08.938036+0000","last_peered":"2024-03-16T19:32:08.938036+0000","last_clean":"2024-03-16T19:32:08.938036+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:08.938036+0000","last_undegraded":"2024-03-16T19:32:08.938036+0000","last_fullsized":"2024-03-16T19:32:08.938036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093703+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:08.093703+0000","last_peered":"2024-03-16T19:32:08.093703+0000","last_clean":"2024-03-16T19:32:08.093703+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:08.093703+0000","last_undegraded":"2024-03-16T19:32:08.093703+0000","last_fullsized":"2024-03-16T19:32:08.093703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084002+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:08.084002+0000","last_peered":"2024-03-16T19:32:08.084002+0000","last_clean":"2024-03-16T19:32:08.084002+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:08.084002+0000","last_undegraded":"2024-03-16T19:32:08.084002+0000","last_fullsized":"2024-03-16T19:32:08.084002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086673+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:08.086673+0000","last_peered":"2024-03-16T19:32:08.086673+0000","last_clean":"2024-03-16T19:32:08.086673+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:08.086673+0000","last_undegraded":"2024-03-16T19:32:08.086673+0000","last_fullsized":"2024-03-16T19:32:08.086673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:59:55.019036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":148,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937729+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:08.937729+0000","last_peered":"2024-03-16T19:32:08.937729+0000","last_clean":"2024-03-16T19:32:08.937729+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:08.937729+0000","last_undegraded":"2024-03-16T19:32:08.937729+0000","last_fullsized":"2024-03-16T19:32:08.937729+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":363,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937413+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:08.937413+0000","last_peered":"2024-03-16T19:32:08.937413+0000","last_clean":"2024-03-16T19:32:08.937413+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:08.937413+0000","last_undegraded":"2024-03-16T19:32:08.937413+0000","last_fullsized":"2024-03-16T19:32:08.937413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087541+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:08.087541+0000","last_peered":"2024-03-16T19:32:08.087541+0000","last_clean":"2024-03-16T19:32:08.087541+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:08.087541+0000","last_undegraded":"2024-03-16T19:32:08.087541+0000","last_fullsized":"2024-03-16T19:32:08.087541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:16:54.768710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937520+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:08.937520+0000","last_peered":"2024-03-16T19:32:08.937520+0000","last_clean":"2024-03-16T19:32:08.937520+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:08.937520+0000","last_undegraded":"2024-03-16T19:32:08.937520+0000","last_fullsized":"2024-03-16T19:32:08.937520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351264+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:08.351264+0000","last_peered":"2024-03-16T19:32:08.351264+0000","last_clean":"2024-03-16T19:32:08.351264+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:08.351264+0000","last_undegraded":"2024-03-16T19:32:08.351264+0000","last_fullsized":"2024-03-16T19:32:08.351264+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:44.308565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087130+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:08.087130+0000","last_peered":"2024-03-16T19:32:08.087130+0000","last_clean":"2024-03-16T19:32:08.087130+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:08.087130+0000","last_undegraded":"2024-03-16T19:32:08.087130+0000","last_fullsized":"2024-03-16T19:32:08.087130+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:20:21.014268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.096035+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:08.096035+0000","last_peered":"2024-03-16T19:32:08.096035+0000","last_clean":"2024-03-16T19:32:08.096035+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:08.096035+0000","last_undegraded":"2024-03-16T19:32:08.096035+0000","last_fullsized":"2024-03-16T19:32:08.096035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084314+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:08.084314+0000","last_peered":"2024-03-16T19:32:08.084314+0000","last_clean":"2024-03-16T19:32:08.084314+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:08.084314+0000","last_undegraded":"2024-03-16T19:32:08.084314+0000","last_fullsized":"2024-03-16T19:32:08.084314+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:01.856961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093299+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:08.093299+0000","last_peered":"2024-03-16T19:32:08.093299+0000","last_clean":"2024-03-16T19:32:08.093299+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:08.093299+0000","last_undegraded":"2024-03-16T19:32:08.093299+0000","last_fullsized":"2024-03-16T19:32:08.093299+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084193+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:08.084193+0000","last_peered":"2024-03-16T19:32:08.084193+0000","last_clean":"2024-03-16T19:32:08.084193+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:08.084193+0000","last_undegraded":"2024-03-16T19:32:08.084193+0000","last_fullsized":"2024-03-16T19:32:08.084193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937906+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:08.937906+0000","last_peered":"2024-03-16T19:32:08.937906+0000","last_clean":"2024-03-16T19:32:08.937906+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:08.937906+0000","last_undegraded":"2024-03-16T19:32:08.937906+0000","last_fullsized":"2024-03-16T19:32:08.937906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093171+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:08.093171+0000","last_peered":"2024-03-16T19:32:08.093171+0000","last_clean":"2024-03-16T19:32:08.093171+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:08.093171+0000","last_undegraded":"2024-03-16T19:32:08.093171+0000","last_fullsized":"2024-03-16T19:32:08.093171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'11","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094654+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:08.094654+0000","last_peered":"2024-03-16T19:32:08.094654+0000","last_clean":"2024-03-16T19:32:08.094654+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:08.094654+0000","last_undegraded":"2024-03-16T19:32:08.094654+0000","last_fullsized":"2024-03-16T19:32:08.094654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088741+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:08.088741+0000","last_peered":"2024-03-16T19:32:08.088741+0000","last_clean":"2024-03-16T19:32:08.088741+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:08.088741+0000","last_undegraded":"2024-03-16T19:32:08.088741+0000","last_fullsized":"2024-03-16T19:32:08.088741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:18.996236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287797+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:09.287797+0000","last_peered":"2024-03-16T19:32:09.287797+0000","last_clean":"2024-03-16T19:32:09.287797+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:09.287797+0000","last_undegraded":"2024-03-16T19:32:09.287797+0000","last_fullsized":"2024-03-16T19:32:09.287797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350908+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:08.350908+0000","last_peered":"2024-03-16T19:32:08.350908+0000","last_clean":"2024-03-16T19:32:08.350908+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:08.350908+0000","last_undegraded":"2024-03-16T19:32:08.350908+0000","last_fullsized":"2024-03-16T19:32:08.350908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:02.024100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937470+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:08.937470+0000","last_peered":"2024-03-16T19:32:08.937470+0000","last_clean":"2024-03-16T19:32:08.937470+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:08.937470+0000","last_undegraded":"2024-03-16T19:32:08.937470+0000","last_fullsized":"2024-03-16T19:32:08.937470+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083795+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:08.083795+0000","last_peered":"2024-03-16T19:32:08.083795+0000","last_clean":"2024-03-16T19:32:08.083795+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:08.083795+0000","last_undegraded":"2024-03-16T19:32:08.083795+0000","last_fullsized":"2024-03-16T19:32:08.083795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:46:49.628764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094246+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:08.094246+0000","last_peered":"2024-03-16T19:32:08.094246+0000","last_clean":"2024-03-16T19:32:08.094246+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:08.094246+0000","last_undegraded":"2024-03-16T19:32:08.094246+0000","last_fullsized":"2024-03-16T19:32:08.094246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.102711+0000","last_change":"2024-03-16T19:32:08.101455+0000","last_active":"2024-03-16T19:32:08.102711+0000","last_peered":"2024-03-16T19:32:08.102711+0000","last_clean":"2024-03-16T19:32:08.102711+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:32:08.102711+0000","last_undegraded":"2024-03-16T19:32:08.102711+0000","last_fullsized":"2024-03-16T19:32:08.102711+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:20:10.984083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.095090+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:08.095090+0000","last_peered":"2024-03-16T19:32:08.095090+0000","last_clean":"2024-03-16T19:32:08.095090+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:08.095090+0000","last_undegraded":"2024-03-16T19:32:08.095090+0000","last_fullsized":"2024-03-16T19:32:08.095090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089208+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:08.089208+0000","last_peered":"2024-03-16T19:32:08.089208+0000","last_clean":"2024-03-16T19:32:08.089208+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:08.089208+0000","last_undegraded":"2024-03-16T19:32:08.089208+0000","last_fullsized":"2024-03-16T19:32:08.089208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:03.618618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937781+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:08.937781+0000","last_peered":"2024-03-16T19:32:08.937781+0000","last_clean":"2024-03-16T19:32:08.937781+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:08.937781+0000","last_undegraded":"2024-03-16T19:32:08.937781+0000","last_fullsized":"2024-03-16T19:32:08.937781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287721+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:09.287721+0000","last_peered":"2024-03-16T19:32:09.287721+0000","last_clean":"2024-03-16T19:32:09.287721+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:09.287721+0000","last_undegraded":"2024-03-16T19:32:09.287721+0000","last_fullsized":"2024-03-16T19:32:09.287721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084106+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:08.084106+0000","last_peered":"2024-03-16T19:32:08.084106+0000","last_clean":"2024-03-16T19:32:08.084106+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:08.084106+0000","last_undegraded":"2024-03-16T19:32:08.084106+0000","last_fullsized":"2024-03-16T19:32:08.084106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937766+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:08.937766+0000","last_peered":"2024-03-16T19:32:08.937766+0000","last_clean":"2024-03-16T19:32:08.937766+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:08.937766+0000","last_undegraded":"2024-03-16T19:32:08.937766+0000","last_fullsized":"2024-03-16T19:32:08.937766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083913+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:08.083913+0000","last_peered":"2024-03-16T19:32:08.083913+0000","last_clean":"2024-03-16T19:32:08.083913+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:08.083913+0000","last_undegraded":"2024-03-16T19:32:08.083913+0000","last_fullsized":"2024-03-16T19:32:08.083913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937841+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:08.937841+0000","last_peered":"2024-03-16T19:32:08.937841+0000","last_clean":"2024-03-16T19:32:08.937841+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:08.937841+0000","last_undegraded":"2024-03-16T19:32:08.937841+0000","last_fullsized":"2024-03-16T19:32:08.937841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":164,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351010+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:08.351010+0000","last_peered":"2024-03-16T19:32:08.351010+0000","last_clean":"2024-03-16T19:32:08.351010+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:08.351010+0000","last_undegraded":"2024-03-16T19:32:08.351010+0000","last_fullsized":"2024-03-16T19:32:08.351010+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:19.317523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088482+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088482+0000","last_peered":"2024-03-16T19:32:08.088482+0000","last_clean":"2024-03-16T19:32:08.088482+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088482+0000","last_undegraded":"2024-03-16T19:32:08.088482+0000","last_fullsized":"2024-03-16T19:32:08.088482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:58:01.695714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086003+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:08.086003+0000","last_peered":"2024-03-16T19:32:08.086003+0000","last_clean":"2024-03-16T19:32:08.086003+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:08.086003+0000","last_undegraded":"2024-03-16T19:32:08.086003+0000","last_fullsized":"2024-03-16T19:32:08.086003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:17.040490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351503+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:08.351503+0000","last_peered":"2024-03-16T19:32:08.351503+0000","last_clean":"2024-03-16T19:32:08.351503+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:08.351503+0000","last_undegraded":"2024-03-16T19:32:08.351503+0000","last_fullsized":"2024-03-16T19:32:08.351503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:28:10.116570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40494,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":80988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28840,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343000,"statfs":{"total":96636764160,"available":96607232000,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1310720,"data_stored":634440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343536,"statfs":{"total":96636764160,"available":96607780864,"internally_reserved":0,"allocated":1314816,"data_stored":634844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122056,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29408,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342432,"statfs":{"total":96636764160,"available":96606650368,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28848,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342992,"statfs":{"total":96636764160,"available":96607223808,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:22.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:22.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.686+0000 7f4ce8272640 1 Processor -- start 2024-03-16T19:32:22.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.686+0000 7f4ce8272640 1 -- start start 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce8272640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 0x7f4ce0154660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce8272640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4ce00cb560 0x7f4ce0154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce8272640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce00cc720 con 0x7f4ce006bb10 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce8272640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce01550e0 con 0x7f4ce00cb560 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce8272640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce00cc9c0 con 0x7f4ce006bef0 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce600e640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4ce006bb10 0x7f4ce00cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40004/0 (socket says 172.21.15.172:40004) 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce600e640 1 -- 172.21.15.172:0/3767086551 learned_addr learned my addr 172.21.15.172:0/3767086551 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce680f640 1 --2- 172.21.15.172:0/3767086551 >> v2:172.21.15.172:3300/0 conn(0x7f4ce00cb560 0x7f4ce0154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce580d640 1 --2- 172.21.15.172:0/3767086551 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 0x7f4ce0154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce680f640 1 -- 172.21.15.172:0/3767086551 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 msgr2=0x7f4ce0154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:22.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce680f640 1 --2- 172.21.15.172:0/3767086551 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 0x7f4ce0154660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce680f640 1 -- 172.21.15.172:0/3767086551 >> v1:172.21.15.172:6789/0 conn(0x7f4ce006bb10 legacy=0x7f4ce00cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.687+0000 7f4ce680f640 1 -- 172.21.15.172:0/3767086551 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce015de30 con 0x7f4ce00cb560 2024-03-16T19:32:22.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.688+0000 7f4ce680f640 1 --2- 172.21.15.172:0/3767086551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 0x7f4ce0154ba0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009d90 tx=0x7f4cd802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc1b059833c1aad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:22.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.689+0000 7f4ce500c640 1 -- 172.21.15.172:0/3767086551 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd802cce0 con 0x7f4ce00cb560 2024-03-16T19:32:22.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.689+0000 7f4ce500c640 1 -- 172.21.15.172:0/3767086551 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd802ce40 con 0x7f4ce00cb560 2024-03-16T19:32:22.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.689+0000 7f4ce500c640 1 -- 172.21.15.172:0/3767086551 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8034660 con 0x7f4ce00cb560 2024-03-16T19:32:22.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.689+0000 7f4ce8272640 1 -- 172.21.15.172:0/3767086551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 msgr2=0x7f4ce0154ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.689+0000 7f4ce8272640 1 --2- 172.21.15.172:0/3767086551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 0x7f4ce0154ba0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009d90 tx=0x7f4cd802bf30 comp rx=0 tx=0).stop 2024-03-16T19:32:22.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.690+0000 7f4ce8272640 1 -- 172.21.15.172:0/3767086551 shutdown_connections 2024-03-16T19:32:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.690+0000 7f4ce8272640 1 --2- 172.21.15.172:0/3767086551 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 0x7f4ce0154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.690+0000 7f4ce8272640 1 --2- 172.21.15.172:0/3767086551 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 0x7f4ce0154ba0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.690+0000 7f4ce8272640 1 -- 172.21.15.172:0/3767086551 >> 172.21.15.172:0/3767086551 conn(0x7f4ce0067250 msgr2=0x7f4ce00d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.690+0000 7f4ce8272640 1 -- 172.21.15.172:0/3767086551 shutdown_connections 2024-03-16T19:32:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.690+0000 7f4ce8272640 1 -- 172.21.15.172:0/3767086551 wait complete. 2024-03-16T19:32:22.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.691+0000 7f4ce8272640 1 Processor -- start 2024-03-16T19:32:22.671 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.691+0000 7f4ce8272640 1 -- start start 2024-03-16T19:32:22.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce8272640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce006bb10 0x7f4ce014dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce8272640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 0x7f4ce00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce8272640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 0x7f4ce00b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce8272640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce015ca70 con 0x7f4ce00cb560 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce8272640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce00b41f0 con 0x7f4ce006bb10 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce8272640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce00b5c30 con 0x7f4ce006bef0 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce680f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 0x7f4ce00b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce680f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 0x7f4ce00b4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46676/0 (socket says 172.21.15.172:46676) 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce680f640 1 -- 172.21.15.172:0/3899173644 learned_addr learned my addr 172.21.15.172:0/3899173644 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce600e640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce006bb10 0x7f4ce014dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce580d640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 0x7f4ce00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce680f640 1 -- 172.21.15.172:0/3899173644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 msgr2=0x7f4ce00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce680f640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 0x7f4ce00b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce680f640 1 -- 172.21.15.172:0/3899173644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce006bb10 msgr2=0x7f4ce014dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce680f640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce006bb10 0x7f4ce014dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.692+0000 7f4ce680f640 1 -- 172.21.15.172:0/3899173644 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd8009a70 con 0x7f4ce00cb560 2024-03-16T19:32:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.693+0000 7f4ce680f640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 0x7f4ce00b4bc0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009eb0 tx=0x7f4cd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.693+0000 7f4cd6ffd640 1 -- 172.21.15.172:0/3899173644 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd803d600 con 0x7f4ce00cb560 2024-03-16T19:32:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.693+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce015dbe0 con 0x7f4ce00cb560 2024-03-16T19:32:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.693+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce015ccc0 con 0x7f4ce00cb560 2024-03-16T19:32:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.693+0000 7f4cd6ffd640 1 -- 172.21.15.172:0/3899173644 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd8009450 con 0x7f4ce00cb560 2024-03-16T19:32:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.693+0000 7f4cd6ffd640 1 -- 172.21.15.172:0/3899173644 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd803d960 con 0x7f4ce00cb560 2024-03-16T19:32:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.694+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc8005290 con 0x7f4ce00cb560 2024-03-16T19:32:22.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.695+0000 7f4cd6ffd640 1 -- 172.21.15.172:0/3899173644 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4cd803b040 con 0x7f4ce00cb560 2024-03-16T19:32:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.695+0000 7f4cd6ffd640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4cc406b3e0 0x7f4cc406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.696+0000 7f4cd6ffd640 1 -- 172.21.15.172:0/3899173644 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f4cd80abe00 con 0x7f4ce00cb560 2024-03-16T19:32:22.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.696+0000 7f4ce600e640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4cc406b3e0 0x7f4cc406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:22.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.697+0000 7f4ce600e640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4cc406b3e0 0x7f4cc406d8a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc007fd0 tx=0x7f4cdc007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:22.682 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.703+0000 7f4cd6ffd640 1 -- 172.21.15.172:0/3899173644 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cd8076a80 con 0x7f4ce00cb560 2024-03-16T19:32:22.947 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.968+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4cc8000bc0 con 0x7f4cc406b3e0 2024-03-16T19:32:22.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.977+0000 7f4cd6ffd640 1 -- 172.21.15.172:0/3899173644 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252328 (secure 0 0 0) 0x7f4cc8000bc0 con 0x7f4cc406b3e0 2024-03-16T19:32:22.959 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:22.960 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:22.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4cc406b3e0 msgr2=0x7f4cc406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4cc406b3e0 0x7f4cc406d8a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc007fd0 tx=0x7f4cdc007e80 comp rx=0 tx=0).stop 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 msgr2=0x7f4ce00b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 0x7f4ce00b4bc0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009eb0 tx=0x7f4cd8004a40 comp rx=0 tx=0).stop 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 shutdown_connections 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce00cb560 0x7f4ce00b4bc0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4cc406b3e0 0x7f4cc406d8a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ce006bef0 0x7f4ce00b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 --2- 172.21.15.172:0/3899173644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce006bb10 0x7f4ce014dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 >> 172.21.15.172:0/3899173644 conn(0x7f4ce0067250 msgr2=0x7f4ce01530f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 shutdown_connections 2024-03-16T19:32:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:22.981+0000 7f4ce8272640 1 -- 172.21.15.172:0/3899173644 wait complete. 2024-03-16T19:32:22.974 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-03-16T19:32:22.233797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630862,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":245,"num_write_kb":2489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228480,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":217199,"kb_avail":754746240,"statfs":{"total":773094113280,"available":772860149760,"internally_reserved":0,"allocated":11427840,"data_stored":6022908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":222412203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002377"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937971+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:08.937971+0000","last_peered":"2024-03-16T19:32:08.937971+0000","last_clean":"2024-03-16T19:32:08.937971+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:08.937971+0000","last_undegraded":"2024-03-16T19:32:08.937971+0000","last_fullsized":"2024-03-16T19:32:08.937971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094518+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:08.094518+0000","last_peered":"2024-03-16T19:32:08.094518+0000","last_clean":"2024-03-16T19:32:08.094518+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:08.094518+0000","last_undegraded":"2024-03-16T19:32:08.094518+0000","last_fullsized":"2024-03-16T19:32:08.094518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351343+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:08.351343+0000","last_peered":"2024-03-16T19:32:08.351343+0000","last_clean":"2024-03-16T19:32:08.351343+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:08.351343+0000","last_undegraded":"2024-03-16T19:32:08.351343+0000","last_fullsized":"2024-03-16T19:32:08.351343+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:47.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083818+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:08.083818+0000","last_peered":"2024-03-16T19:32:08.083818+0000","last_clean":"2024-03-16T19:32:08.083818+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:08.083818+0000","last_undegraded":"2024-03-16T19:32:08.083818+0000","last_fullsized":"2024-03-16T19:32:08.083818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937685+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:08.937685+0000","last_peered":"2024-03-16T19:32:08.937685+0000","last_clean":"2024-03-16T19:32:08.937685+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:08.937685+0000","last_undegraded":"2024-03-16T19:32:08.937685+0000","last_fullsized":"2024-03-16T19:32:08.937685+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083640+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:08.083640+0000","last_peered":"2024-03-16T19:32:08.083640+0000","last_clean":"2024-03-16T19:32:08.083640+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:08.083640+0000","last_undegraded":"2024-03-16T19:32:08.083640+0000","last_fullsized":"2024-03-16T19:32:08.083640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937556+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:08.937556+0000","last_peered":"2024-03-16T19:32:08.937556+0000","last_clean":"2024-03-16T19:32:08.937556+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:08.937556+0000","last_undegraded":"2024-03-16T19:32:08.937556+0000","last_fullsized":"2024-03-16T19:32:08.937556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086880+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:08.086880+0000","last_peered":"2024-03-16T19:32:08.086880+0000","last_clean":"2024-03-16T19:32:08.086880+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:08.086880+0000","last_undegraded":"2024-03-16T19:32:08.086880+0000","last_fullsized":"2024-03-16T19:32:08.086880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:58:31.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937978+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:08.937978+0000","last_peered":"2024-03-16T19:32:08.937978+0000","last_clean":"2024-03-16T19:32:08.937978+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:08.937978+0000","last_undegraded":"2024-03-16T19:32:08.937978+0000","last_fullsized":"2024-03-16T19:32:08.937978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287814+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:09.287814+0000","last_peered":"2024-03-16T19:32:09.287814+0000","last_clean":"2024-03-16T19:32:09.287814+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:09.287814+0000","last_undegraded":"2024-03-16T19:32:09.287814+0000","last_fullsized":"2024-03-16T19:32:09.287814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287704+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:09.287704+0000","last_peered":"2024-03-16T19:32:09.287704+0000","last_clean":"2024-03-16T19:32:09.287704+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:09.287704+0000","last_undegraded":"2024-03-16T19:32:09.287704+0000","last_fullsized":"2024-03-16T19:32:09.287704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084143+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:08.084143+0000","last_peered":"2024-03-16T19:32:08.084143+0000","last_clean":"2024-03-16T19:32:08.084143+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:08.084143+0000","last_undegraded":"2024-03-16T19:32:08.084143+0000","last_fullsized":"2024-03-16T19:32:08.084143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088886+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:08.088886+0000","last_peered":"2024-03-16T19:32:08.088886+0000","last_clean":"2024-03-16T19:32:08.088886+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:08.088886+0000","last_undegraded":"2024-03-16T19:32:08.088886+0000","last_fullsized":"2024-03-16T19:32:08.088886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:52.102388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"84'3","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350888+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:08.350888+0000","last_peered":"2024-03-16T19:32:08.350888+0000","last_clean":"2024-03-16T19:32:08.350888+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:08.350888+0000","last_undegraded":"2024-03-16T19:32:08.350888+0000","last_fullsized":"2024-03-16T19:32:08.350888+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:39.289238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094126+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:08.094126+0000","last_peered":"2024-03-16T19:32:08.094126+0000","last_clean":"2024-03-16T19:32:08.094126+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:08.094126+0000","last_undegraded":"2024-03-16T19:32:08.094126+0000","last_fullsized":"2024-03-16T19:32:08.094126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083403+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:08.083403+0000","last_peered":"2024-03-16T19:32:08.083403+0000","last_clean":"2024-03-16T19:32:08.083403+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:08.083403+0000","last_undegraded":"2024-03-16T19:32:08.083403+0000","last_fullsized":"2024-03-16T19:32:08.083403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084254+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:08.084254+0000","last_peered":"2024-03-16T19:32:08.084254+0000","last_clean":"2024-03-16T19:32:08.084254+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:08.084254+0000","last_undegraded":"2024-03-16T19:32:08.084254+0000","last_fullsized":"2024-03-16T19:32:08.084254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087017+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:08.087017+0000","last_peered":"2024-03-16T19:32:08.087017+0000","last_clean":"2024-03-16T19:32:08.087017+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:08.087017+0000","last_undegraded":"2024-03-16T19:32:08.087017+0000","last_fullsized":"2024-03-16T19:32:08.087017+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:36:06.175327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937878+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:08.937878+0000","last_peered":"2024-03-16T19:32:08.937878+0000","last_clean":"2024-03-16T19:32:08.937878+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:08.937878+0000","last_undegraded":"2024-03-16T19:32:08.937878+0000","last_fullsized":"2024-03-16T19:32:08.937878+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287690+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:09.287690+0000","last_peered":"2024-03-16T19:32:09.287690+0000","last_clean":"2024-03-16T19:32:09.287690+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:09.287690+0000","last_undegraded":"2024-03-16T19:32:09.287690+0000","last_fullsized":"2024-03-16T19:32:09.287690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351462+0000","last_change":"2024-03-16T19:31:51.807295+0000","last_active":"2024-03-16T19:32:08.351462+0000","last_peered":"2024-03-16T19:32:08.351462+0000","last_clean":"2024-03-16T19:32:08.351462+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:08.351462+0000","last_undegraded":"2024-03-16T19:32:08.351462+0000","last_fullsized":"2024-03-16T19:32:08.351462+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:04.578640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.085957+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:08.085957+0000","last_peered":"2024-03-16T19:32:08.085957+0000","last_clean":"2024-03-16T19:32:08.085957+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:08.085957+0000","last_undegraded":"2024-03-16T19:32:08.085957+0000","last_fullsized":"2024-03-16T19:32:08.085957+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:41:06.377004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088245+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:08.088245+0000","last_peered":"2024-03-16T19:32:08.088245+0000","last_clean":"2024-03-16T19:32:08.088245+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:08.088245+0000","last_undegraded":"2024-03-16T19:32:08.088245+0000","last_fullsized":"2024-03-16T19:32:08.088245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:47:08.251341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083454+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:08.083454+0000","last_peered":"2024-03-16T19:32:08.083454+0000","last_clean":"2024-03-16T19:32:08.083454+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:08.083454+0000","last_undegraded":"2024-03-16T19:32:08.083454+0000","last_fullsized":"2024-03-16T19:32:08.083454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086134+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:08.086134+0000","last_peered":"2024-03-16T19:32:08.086134+0000","last_clean":"2024-03-16T19:32:08.086134+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:08.086134+0000","last_undegraded":"2024-03-16T19:32:08.086134+0000","last_fullsized":"2024-03-16T19:32:08.086134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:10.818175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287911+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:09.287911+0000","last_peered":"2024-03-16T19:32:09.287911+0000","last_clean":"2024-03-16T19:32:09.287911+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:09.287911+0000","last_undegraded":"2024-03-16T19:32:09.287911+0000","last_fullsized":"2024-03-16T19:32:09.287911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":187,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093433+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:08.093433+0000","last_peered":"2024-03-16T19:32:08.093433+0000","last_clean":"2024-03-16T19:32:08.093433+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:08.093433+0000","last_undegraded":"2024-03-16T19:32:08.093433+0000","last_fullsized":"2024-03-16T19:32:08.093433+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087892+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:08.087892+0000","last_peered":"2024-03-16T19:32:08.087892+0000","last_clean":"2024-03-16T19:32:08.087892+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:08.087892+0000","last_undegraded":"2024-03-16T19:32:08.087892+0000","last_fullsized":"2024-03-16T19:32:08.087892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:12:08.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350890+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:08.350890+0000","last_peered":"2024-03-16T19:32:08.350890+0000","last_clean":"2024-03-16T19:32:08.350890+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:08.350890+0000","last_undegraded":"2024-03-16T19:32:08.350890+0000","last_fullsized":"2024-03-16T19:32:08.350890+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:12.646493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083749+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:08.083749+0000","last_peered":"2024-03-16T19:32:08.083749+0000","last_clean":"2024-03-16T19:32:08.083749+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:08.083749+0000","last_undegraded":"2024-03-16T19:32:08.083749+0000","last_fullsized":"2024-03-16T19:32:08.083749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087224+0000","last_change":"2024-03-16T19:31:49.974297+0000","last_active":"2024-03-16T19:32:08.087224+0000","last_peered":"2024-03-16T19:32:08.087224+0000","last_clean":"2024-03-16T19:32:08.087224+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:08.087224+0000","last_undegraded":"2024-03-16T19:32:08.087224+0000","last_fullsized":"2024-03-16T19:32:08.087224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:51:57.535372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287591+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:09.287591+0000","last_peered":"2024-03-16T19:32:09.287591+0000","last_clean":"2024-03-16T19:32:09.287591+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:09.287591+0000","last_undegraded":"2024-03-16T19:32:09.287591+0000","last_fullsized":"2024-03-16T19:32:09.287591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084174+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:32:08.084174+0000","last_peered":"2024-03-16T19:32:08.084174+0000","last_clean":"2024-03-16T19:32:08.084174+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:08.084174+0000","last_undegraded":"2024-03-16T19:32:08.084174+0000","last_fullsized":"2024-03-16T19:32:08.084174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089011+0000","last_change":"2024-03-16T19:31:26.808032+0000","last_active":"2024-03-16T19:32:08.089011+0000","last_peered":"2024-03-16T19:32:08.089011+0000","last_clean":"2024-03-16T19:32:08.089011+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:08.089011+0000","last_undegraded":"2024-03-16T19:32:08.089011+0000","last_fullsized":"2024-03-16T19:32:08.089011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:15.218778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351569+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:08.351569+0000","last_peered":"2024-03-16T19:32:08.351569+0000","last_clean":"2024-03-16T19:32:08.351569+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:08.351569+0000","last_undegraded":"2024-03-16T19:32:08.351569+0000","last_fullsized":"2024-03-16T19:32:08.351569+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:16:31.606469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287611+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:09.287611+0000","last_peered":"2024-03-16T19:32:09.287611+0000","last_clean":"2024-03-16T19:32:09.287611+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:09.287611+0000","last_undegraded":"2024-03-16T19:32:09.287611+0000","last_fullsized":"2024-03-16T19:32:09.287611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086078+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086078+0000","last_peered":"2024-03-16T19:32:08.086078+0000","last_clean":"2024-03-16T19:32:08.086078+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086078+0000","last_undegraded":"2024-03-16T19:32:08.086078+0000","last_fullsized":"2024-03-16T19:32:08.086078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:23:19.824904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086998+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:08.086998+0000","last_peered":"2024-03-16T19:32:08.086998+0000","last_clean":"2024-03-16T19:32:08.086998+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:08.086998+0000","last_undegraded":"2024-03-16T19:32:08.086998+0000","last_fullsized":"2024-03-16T19:32:08.086998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:45.782085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094004+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:08.094004+0000","last_peered":"2024-03-16T19:32:08.094004+0000","last_clean":"2024-03-16T19:32:08.094004+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:08.094004+0000","last_undegraded":"2024-03-16T19:32:08.094004+0000","last_fullsized":"2024-03-16T19:32:08.094004+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287793+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:09.287793+0000","last_peered":"2024-03-16T19:32:09.287793+0000","last_clean":"2024-03-16T19:32:09.287793+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:09.287793+0000","last_undegraded":"2024-03-16T19:32:09.287793+0000","last_fullsized":"2024-03-16T19:32:09.287793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083754+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:08.083754+0000","last_peered":"2024-03-16T19:32:08.083754+0000","last_clean":"2024-03-16T19:32:08.083754+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:08.083754+0000","last_undegraded":"2024-03-16T19:32:08.083754+0000","last_fullsized":"2024-03-16T19:32:08.083754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351532+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:08.351532+0000","last_peered":"2024-03-16T19:32:08.351532+0000","last_clean":"2024-03-16T19:32:08.351532+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:08.351532+0000","last_undegraded":"2024-03-16T19:32:08.351532+0000","last_fullsized":"2024-03-16T19:32:08.351532+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:16.418962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093565+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:08.093565+0000","last_peered":"2024-03-16T19:32:08.093565+0000","last_clean":"2024-03-16T19:32:08.093565+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:08.093565+0000","last_undegraded":"2024-03-16T19:32:08.093565+0000","last_fullsized":"2024-03-16T19:32:08.093565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938817+0000","last_change":"2024-03-16T19:32:08.938817+0000","last_active":"2024-03-16T19:32:08.938817+0000","last_peered":"2024-03-16T19:32:08.938817+0000","last_clean":"2024-03-16T19:32:08.938817+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:08.938817+0000","last_undegraded":"2024-03-16T19:32:08.938817+0000","last_fullsized":"2024-03-16T19:32:08.938817+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:39:32.174855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"86'23","reported_seq":233,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287734+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:09.287734+0000","last_peered":"2024-03-16T19:32:09.287734+0000","last_clean":"2024-03-16T19:32:09.287734+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:09.287734+0000","last_undegraded":"2024-03-16T19:32:09.287734+0000","last_fullsized":"2024-03-16T19:32:09.287734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39292,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937730+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:08.937730+0000","last_peered":"2024-03-16T19:32:08.937730+0000","last_clean":"2024-03-16T19:32:08.937730+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:08.937730+0000","last_undegraded":"2024-03-16T19:32:08.937730+0000","last_fullsized":"2024-03-16T19:32:08.937730+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351564+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:08.351564+0000","last_peered":"2024-03-16T19:32:08.351564+0000","last_clean":"2024-03-16T19:32:08.351564+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:08.351564+0000","last_undegraded":"2024-03-16T19:32:08.351564+0000","last_fullsized":"2024-03-16T19:32:08.351564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:59.818393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351224+0000","last_change":"2024-03-16T19:31:51.807895+0000","last_active":"2024-03-16T19:32:08.351224+0000","last_peered":"2024-03-16T19:32:08.351224+0000","last_clean":"2024-03-16T19:32:08.351224+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:08.351224+0000","last_undegraded":"2024-03-16T19:32:08.351224+0000","last_fullsized":"2024-03-16T19:32:08.351224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:25.988237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937695+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:08.937695+0000","last_peered":"2024-03-16T19:32:08.937695+0000","last_clean":"2024-03-16T19:32:08.937695+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:08.937695+0000","last_undegraded":"2024-03-16T19:32:08.937695+0000","last_fullsized":"2024-03-16T19:32:08.937695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287610+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:09.287610+0000","last_peered":"2024-03-16T19:32:09.287610+0000","last_clean":"2024-03-16T19:32:09.287610+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:09.287610+0000","last_undegraded":"2024-03-16T19:32:09.287610+0000","last_fullsized":"2024-03-16T19:32:09.287610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287595+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:09.287595+0000","last_peered":"2024-03-16T19:32:09.287595+0000","last_clean":"2024-03-16T19:32:09.287595+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:09.287595+0000","last_undegraded":"2024-03-16T19:32:09.287595+0000","last_fullsized":"2024-03-16T19:32:09.287595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087374+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:08.087374+0000","last_peered":"2024-03-16T19:32:08.087374+0000","last_clean":"2024-03-16T19:32:08.087374+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:08.087374+0000","last_undegraded":"2024-03-16T19:32:08.087374+0000","last_fullsized":"2024-03-16T19:32:08.087374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:33.634146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937562+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:08.937562+0000","last_peered":"2024-03-16T19:32:08.937562+0000","last_clean":"2024-03-16T19:32:08.937562+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:08.937562+0000","last_undegraded":"2024-03-16T19:32:08.937562+0000","last_fullsized":"2024-03-16T19:32:08.937562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084319+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084319+0000","last_peered":"2024-03-16T19:32:08.084319+0000","last_clean":"2024-03-16T19:32:08.084319+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084319+0000","last_undegraded":"2024-03-16T19:32:08.084319+0000","last_fullsized":"2024-03-16T19:32:08.084319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:53.021797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937869+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:08.937869+0000","last_peered":"2024-03-16T19:32:08.937869+0000","last_clean":"2024-03-16T19:32:08.937869+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:08.937869+0000","last_undegraded":"2024-03-16T19:32:08.937869+0000","last_fullsized":"2024-03-16T19:32:08.937869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":55,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086917+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:08.086917+0000","last_peered":"2024-03-16T19:32:08.086917+0000","last_clean":"2024-03-16T19:32:08.086917+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:08.086917+0000","last_undegraded":"2024-03-16T19:32:08.086917+0000","last_fullsized":"2024-03-16T19:32:08.086917+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:07:39.853290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938036+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:08.938036+0000","last_peered":"2024-03-16T19:32:08.938036+0000","last_clean":"2024-03-16T19:32:08.938036+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:08.938036+0000","last_undegraded":"2024-03-16T19:32:08.938036+0000","last_fullsized":"2024-03-16T19:32:08.938036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093703+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:08.093703+0000","last_peered":"2024-03-16T19:32:08.093703+0000","last_clean":"2024-03-16T19:32:08.093703+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:08.093703+0000","last_undegraded":"2024-03-16T19:32:08.093703+0000","last_fullsized":"2024-03-16T19:32:08.093703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084002+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:08.084002+0000","last_peered":"2024-03-16T19:32:08.084002+0000","last_clean":"2024-03-16T19:32:08.084002+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:08.084002+0000","last_undegraded":"2024-03-16T19:32:08.084002+0000","last_fullsized":"2024-03-16T19:32:08.084002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086673+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:08.086673+0000","last_peered":"2024-03-16T19:32:08.086673+0000","last_clean":"2024-03-16T19:32:08.086673+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:08.086673+0000","last_undegraded":"2024-03-16T19:32:08.086673+0000","last_fullsized":"2024-03-16T19:32:08.086673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:59:55.019036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":148,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937729+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:08.937729+0000","last_peered":"2024-03-16T19:32:08.937729+0000","last_clean":"2024-03-16T19:32:08.937729+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:08.937729+0000","last_undegraded":"2024-03-16T19:32:08.937729+0000","last_fullsized":"2024-03-16T19:32:08.937729+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":363,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937413+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:08.937413+0000","last_peered":"2024-03-16T19:32:08.937413+0000","last_clean":"2024-03-16T19:32:08.937413+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:08.937413+0000","last_undegraded":"2024-03-16T19:32:08.937413+0000","last_fullsized":"2024-03-16T19:32:08.937413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087541+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:08.087541+0000","last_peered":"2024-03-16T19:32:08.087541+0000","last_clean":"2024-03-16T19:32:08.087541+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:08.087541+0000","last_undegraded":"2024-03-16T19:32:08.087541+0000","last_fullsized":"2024-03-16T19:32:08.087541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:16:54.768710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937520+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:08.937520+0000","last_peered":"2024-03-16T19:32:08.937520+0000","last_clean":"2024-03-16T19:32:08.937520+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:08.937520+0000","last_undegraded":"2024-03-16T19:32:08.937520+0000","last_fullsized":"2024-03-16T19:32:08.937520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351264+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:08.351264+0000","last_peered":"2024-03-16T19:32:08.351264+0000","last_clean":"2024-03-16T19:32:08.351264+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:08.351264+0000","last_undegraded":"2024-03-16T19:32:08.351264+0000","last_fullsized":"2024-03-16T19:32:08.351264+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:44.308565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087130+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:08.087130+0000","last_peered":"2024-03-16T19:32:08.087130+0000","last_clean":"2024-03-16T19:32:08.087130+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:08.087130+0000","last_undegraded":"2024-03-16T19:32:08.087130+0000","last_fullsized":"2024-03-16T19:32:08.087130+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:20:21.014268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.096035+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:08.096035+0000","last_peered":"2024-03-16T19:32:08.096035+0000","last_clean":"2024-03-16T19:32:08.096035+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:08.096035+0000","last_undegraded":"2024-03-16T19:32:08.096035+0000","last_fullsized":"2024-03-16T19:32:08.096035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084314+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:08.084314+0000","last_peered":"2024-03-16T19:32:08.084314+0000","last_clean":"2024-03-16T19:32:08.084314+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:08.084314+0000","last_undegraded":"2024-03-16T19:32:08.084314+0000","last_fullsized":"2024-03-16T19:32:08.084314+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:01.856961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093299+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:08.093299+0000","last_peered":"2024-03-16T19:32:08.093299+0000","last_clean":"2024-03-16T19:32:08.093299+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:08.093299+0000","last_undegraded":"2024-03-16T19:32:08.093299+0000","last_fullsized":"2024-03-16T19:32:08.093299+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084193+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:08.084193+0000","last_peered":"2024-03-16T19:32:08.084193+0000","last_clean":"2024-03-16T19:32:08.084193+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:08.084193+0000","last_undegraded":"2024-03-16T19:32:08.084193+0000","last_fullsized":"2024-03-16T19:32:08.084193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937906+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:08.937906+0000","last_peered":"2024-03-16T19:32:08.937906+0000","last_clean":"2024-03-16T19:32:08.937906+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:08.937906+0000","last_undegraded":"2024-03-16T19:32:08.937906+0000","last_fullsized":"2024-03-16T19:32:08.937906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093171+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:08.093171+0000","last_peered":"2024-03-16T19:32:08.093171+0000","last_clean":"2024-03-16T19:32:08.093171+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:08.093171+0000","last_undegraded":"2024-03-16T19:32:08.093171+0000","last_fullsized":"2024-03-16T19:32:08.093171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'11","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094654+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:08.094654+0000","last_peered":"2024-03-16T19:32:08.094654+0000","last_clean":"2024-03-16T19:32:08.094654+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:08.094654+0000","last_undegraded":"2024-03-16T19:32:08.094654+0000","last_fullsized":"2024-03-16T19:32:08.094654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088741+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:08.088741+0000","last_peered":"2024-03-16T19:32:08.088741+0000","last_clean":"2024-03-16T19:32:08.088741+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:08.088741+0000","last_undegraded":"2024-03-16T19:32:08.088741+0000","last_fullsized":"2024-03-16T19:32:08.088741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:18.996236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287797+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:09.287797+0000","last_peered":"2024-03-16T19:32:09.287797+0000","last_clean":"2024-03-16T19:32:09.287797+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:09.287797+0000","last_undegraded":"2024-03-16T19:32:09.287797+0000","last_fullsized":"2024-03-16T19:32:09.287797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350908+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:08.350908+0000","last_peered":"2024-03-16T19:32:08.350908+0000","last_clean":"2024-03-16T19:32:08.350908+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:08.350908+0000","last_undegraded":"2024-03-16T19:32:08.350908+0000","last_fullsized":"2024-03-16T19:32:08.350908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:02.024100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937470+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:08.937470+0000","last_peered":"2024-03-16T19:32:08.937470+0000","last_clean":"2024-03-16T19:32:08.937470+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:08.937470+0000","last_undegraded":"2024-03-16T19:32:08.937470+0000","last_fullsized":"2024-03-16T19:32:08.937470+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083795+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:08.083795+0000","last_peered":"2024-03-16T19:32:08.083795+0000","last_clean":"2024-03-16T19:32:08.083795+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:08.083795+0000","last_undegraded":"2024-03-16T19:32:08.083795+0000","last_fullsized":"2024-03-16T19:32:08.083795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:46:49.628764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094246+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:08.094246+0000","last_peered":"2024-03-16T19:32:08.094246+0000","last_clean":"2024-03-16T19:32:08.094246+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:08.094246+0000","last_undegraded":"2024-03-16T19:32:08.094246+0000","last_fullsized":"2024-03-16T19:32:08.094246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.102711+0000","last_change":"2024-03-16T19:32:08.101455+0000","last_active":"2024-03-16T19:32:08.102711+0000","last_peered":"2024-03-16T19:32:08.102711+0000","last_clean":"2024-03-16T19:32:08.102711+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:32:08.102711+0000","last_undegraded":"2024-03-16T19:32:08.102711+0000","last_fullsized":"2024-03-16T19:32:08.102711+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:20:10.984083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.095090+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:08.095090+0000","last_peered":"2024-03-16T19:32:08.095090+0000","last_clean":"2024-03-16T19:32:08.095090+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:08.095090+0000","last_undegraded":"2024-03-16T19:32:08.095090+0000","last_fullsized":"2024-03-16T19:32:08.095090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089208+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:08.089208+0000","last_peered":"2024-03-16T19:32:08.089208+0000","last_clean":"2024-03-16T19:32:08.089208+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:08.089208+0000","last_undegraded":"2024-03-16T19:32:08.089208+0000","last_fullsized":"2024-03-16T19:32:08.089208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:03.618618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937781+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:08.937781+0000","last_peered":"2024-03-16T19:32:08.937781+0000","last_clean":"2024-03-16T19:32:08.937781+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:08.937781+0000","last_undegraded":"2024-03-16T19:32:08.937781+0000","last_fullsized":"2024-03-16T19:32:08.937781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287721+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:09.287721+0000","last_peered":"2024-03-16T19:32:09.287721+0000","last_clean":"2024-03-16T19:32:09.287721+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:09.287721+0000","last_undegraded":"2024-03-16T19:32:09.287721+0000","last_fullsized":"2024-03-16T19:32:09.287721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084106+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:08.084106+0000","last_peered":"2024-03-16T19:32:08.084106+0000","last_clean":"2024-03-16T19:32:08.084106+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:08.084106+0000","last_undegraded":"2024-03-16T19:32:08.084106+0000","last_fullsized":"2024-03-16T19:32:08.084106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937766+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:08.937766+0000","last_peered":"2024-03-16T19:32:08.937766+0000","last_clean":"2024-03-16T19:32:08.937766+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:08.937766+0000","last_undegraded":"2024-03-16T19:32:08.937766+0000","last_fullsized":"2024-03-16T19:32:08.937766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083913+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:08.083913+0000","last_peered":"2024-03-16T19:32:08.083913+0000","last_clean":"2024-03-16T19:32:08.083913+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:08.083913+0000","last_undegraded":"2024-03-16T19:32:08.083913+0000","last_fullsized":"2024-03-16T19:32:08.083913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937841+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:08.937841+0000","last_peered":"2024-03-16T19:32:08.937841+0000","last_clean":"2024-03-16T19:32:08.937841+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:08.937841+0000","last_undegraded":"2024-03-16T19:32:08.937841+0000","last_fullsized":"2024-03-16T19:32:08.937841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":164,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351010+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:08.351010+0000","last_peered":"2024-03-16T19:32:08.351010+0000","last_clean":"2024-03-16T19:32:08.351010+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:08.351010+0000","last_undegraded":"2024-03-16T19:32:08.351010+0000","last_fullsized":"2024-03-16T19:32:08.351010+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:19.317523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088482+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088482+0000","last_peered":"2024-03-16T19:32:08.088482+0000","last_clean":"2024-03-16T19:32:08.088482+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088482+0000","last_undegraded":"2024-03-16T19:32:08.088482+0000","last_fullsized":"2024-03-16T19:32:08.088482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:58:01.695714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086003+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:08.086003+0000","last_peered":"2024-03-16T19:32:08.086003+0000","last_clean":"2024-03-16T19:32:08.086003+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:08.086003+0000","last_undegraded":"2024-03-16T19:32:08.086003+0000","last_fullsized":"2024-03-16T19:32:08.086003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:17.040490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351503+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:08.351503+0000","last_peered":"2024-03-16T19:32:08.351503+0000","last_clean":"2024-03-16T19:32:08.351503+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:08.351503+0000","last_undegraded":"2024-03-16T19:32:08.351503+0000","last_fullsized":"2024-03-16T19:32:08.351503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:28:10.116570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40494,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":80988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28840,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343000,"statfs":{"total":96636764160,"available":96607232000,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1310720,"data_stored":634440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343536,"statfs":{"total":96636764160,"available":96607780864,"internally_reserved":0,"allocated":1314816,"data_stored":634844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122056,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29408,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342432,"statfs":{"total":96636764160,"available":96606650368,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28848,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342992,"statfs":{"total":96636764160,"available":96607223808,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:22.977 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-16T19:32:22.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:23.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.171+0000 7f292190d640 1 Processor -- start 2024-03-16T19:32:23.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.171+0000 7f292190d640 1 -- start start 2024-03-16T19:32:23.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f292190d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f291c1517e0 0x7f291c0ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f291affd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f291c1517e0 0x7f291c0ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.156 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f291affd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f291c1517e0 0x7f291c0ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:54238/0 (socket says 172.21.15.172:54238) 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f291affd640 1 -- 172.21.15.172:0/9093062 learned_addr learned my addr 172.21.15.172:0/9093062 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f292190d640 1 -- 172.21.15.172:0/9093062 --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291c15ae30 con 0x7f291c073030 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f292190d640 1 -- 172.21.15.172:0/9093062 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291c15eb80 con 0x7f291c06be10 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f292190d640 1 -- 172.21.15.172:0/9093062 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291c06b560 con 0x7f291c1517e0 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451188644 0 0) 0x7f291c15eb80 con 0x7f291c06be10 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2904003520 con 0x7f291c06be10 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206479553 0 0) 0x7f291c15ae30 con 0x7f291c073030 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.173+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f291c15eb80 con 0x7f291c073030 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2364773491 0 0) 0x7f2904003520 con 0x7f291c06be10 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 >> v2:172.21.15.174:3300/0 conn(0x7f291c1517e0 msgr2=0x7f291c0ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 --2- 172.21.15.172:0/9093062 >> v2:172.21.15.174:3300/0 conn(0x7f291c1517e0 0x7f291c0ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 >> v1:172.21.15.172:6789/0 conn(0x7f291c073030 legacy=0x7f291c15e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c072cf0 con 0x7f291c06be10 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f2910001670 con 0x7f291c06be10 2024-03-16T19:32:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 >> v1:172.21.15.174:6789/0 conn(0x7f291c06be10 legacy=0x7f291c15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2904004b60 0x7f2904004f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2904003760 0x7f2904003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904004120 0x7f290400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c06b560 con 0x7f2904003760 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2904014780 con 0x7f2904004120 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2904014940 con 0x7f2904004b60 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f291a7fc640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2904003760 0x7f2904003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f291b7fe640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904004120 0x7f290400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f291affd640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2904004b60 0x7f2904004f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f291a7fc640 1 -- 172.21.15.172:0/9093062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2904004b60 msgr2=0x7f2904004f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f291a7fc640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2904004b60 0x7f2904004f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f291a7fc640 1 -- 172.21.15.172:0/9093062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904004120 msgr2=0x7f290400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f291a7fc640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904004120 0x7f290400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f291a7fc640 1 -- 172.21.15.172:0/9093062 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2904003520 con 0x7f2904003760 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.174+0000 7f291b7fe640 1 -- 172.21.15.172:0/9093062 reap_dead start 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.175+0000 7f291a7fc640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2904003760 0x7f2904003be0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2910007f40 tx=0x7f29040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a6d4e0232973b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.175+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291002c920 con 0x7f2904003760 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.175+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291002caa0 con 0x7f2904003760 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.175+0000 7f2919ffb640 1 -- 172.21.15.172:0/9093062 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291002ce50 con 0x7f2904003760 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.176+0000 7f292190d640 1 -- 172.21.15.172:0/9093062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2904003760 msgr2=0x7f2904003be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.176+0000 7f292190d640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2904003760 0x7f2904003be0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2910007f40 tx=0x7f29040057f0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.176+0000 7f292190d640 1 -- 172.21.15.172:0/9093062 shutdown_connections 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.176+0000 7f292190d640 1 --2- 172.21.15.172:0/9093062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2904003760 0x7f2904003be0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.176+0000 7f292190d640 1 -- 172.21.15.172:0/9093062 >> 172.21.15.172:0/9093062 conn(0x7f291c06f6f0 msgr2=0x7f291c06fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.176+0000 7f292190d640 1 -- 172.21.15.172:0/9093062 shutdown_connections 2024-03-16T19:32:23.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.176+0000 7f292190d640 1 -- 172.21.15.172:0/9093062 wait complete. 2024-03-16T19:32:23.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.178+0000 7f292190d640 1 Processor -- start 2024-03-16T19:32:23.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.178+0000 7f292190d640 1 -- start start 2024-03-16T19:32:23.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f292190d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904003760 0x7f291c1537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f292190d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f291c0b2fd0 0x7f291c14d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f292190d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c1443f0 0x7f291c142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f292190d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c14d0c0 con 0x7f291c0b2fd0 2024-03-16T19:32:23.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f292190d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c152ce0 con 0x7f2904003760 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f292190d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c14ce00 con 0x7f291c1443f0 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c1443f0 0x7f291c142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904003760 0x7f291c1537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904003760 0x7f291c1537d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:54264/0 (socket says 172.21.15.172:54264) 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c1443f0 0x7f291c142840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44800/0 (socket says 172.21.15.172:44800) 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291affd640 1 -- 172.21.15.172:0/3191582029 learned_addr learned my addr 172.21.15.172:0/3191582029 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291a7fc640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f291c0b2fd0 0x7f291c14d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291affd640 1 -- 172.21.15.172:0/3191582029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c1443f0 msgr2=0x7f291c142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291affd640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c1443f0 0x7f291c142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291affd640 1 -- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f291c0b2fd0 msgr2=0x7f291c14d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.179+0000 7f291affd640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f291c0b2fd0 0x7f291c14d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.180+0000 7f291affd640 1 -- 172.21.15.172:0/3191582029 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c152720 con 0x7f2904003760 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.180+0000 7f291b7fe640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c1443f0 0x7f291c142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.180+0000 7f291affd640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904003760 0x7f291c1537d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f290c004650 tx=0x7f290c0119a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:23.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.180+0000 7f2903fff640 1 -- 172.21.15.172:0/3191582029 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c015870 con 0x7f2904003760 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.180+0000 7f2903fff640 1 -- 172.21.15.172:0/3191582029 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c0159d0 con 0x7f2904003760 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.180+0000 7f2903fff640 1 -- 172.21.15.172:0/3191582029 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c015da0 con 0x7f2904003760 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.180+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c152a00 con 0x7f2904003760 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.180+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c154900 con 0x7f2904003760 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.181+0000 7f2903fff640 1 -- 172.21.15.172:0/3191582029 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f290c01d400 con 0x7f2904003760 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.182+0000 7f2903fff640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f290806b3b0 0x7f290806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.182+0000 7f2903fff640 1 -- 172.21.15.172:0/3191582029 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f290c08a300 con 0x7f2904003760 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.182+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c154620 con 0x7f2904003760 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.182+0000 7f291a7fc640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f290806b3b0 0x7f290806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.183+0000 7f291a7fc640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f290806b3b0 0x7f290806d870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2910007f40 tx=0x7f2910005040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:23.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.187+0000 7f2903fff640 1 -- 172.21.15.172:0/3191582029 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f291c154620 con 0x7f2904003760 2024-03-16T19:32:23.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.487+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f291c06c8c0 con 0x7f290806b3b0 2024-03-16T19:32:23.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.499+0000 7f2903fff640 1 -- 172.21.15.172:0/3191582029 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252328 (secure 0 0 0) 0x7f291002c720 con 0x7f290806b3b0 2024-03-16T19:32:23.481 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:23.482 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f290806b3b0 msgr2=0x7f290806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f290806b3b0 0x7f290806d870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2910007f40 tx=0x7f2910005040 comp rx=0 tx=0).stop 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904003760 msgr2=0x7f291c1537d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904003760 0x7f291c1537d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f290c004650 tx=0x7f290c0119a0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 shutdown_connections 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f290806b3b0 0x7f290806d870 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c1443f0 0x7f291c142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f291c0b2fd0 0x7f291c14d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 --2- 172.21.15.172:0/3191582029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904003760 0x7f291c1537d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.502+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 >> 172.21.15.172:0/3191582029 conn(0x7f291c06f6f0 msgr2=0x7f291c14b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:23.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.503+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 shutdown_connections 2024-03-16T19:32:23.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.503+0000 7f292190d640 1 -- 172.21.15.172:0/3191582029 wait complete. 2024-03-16T19:32:23.498 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-03-16T19:32:22.233797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630862,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":245,"num_write_kb":2489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228480,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":217199,"kb_avail":754746240,"statfs":{"total":773094113280,"available":772860149760,"internally_reserved":0,"allocated":11427840,"data_stored":6022908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":222412203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002377"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937971+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:08.937971+0000","last_peered":"2024-03-16T19:32:08.937971+0000","last_clean":"2024-03-16T19:32:08.937971+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:08.937971+0000","last_undegraded":"2024-03-16T19:32:08.937971+0000","last_fullsized":"2024-03-16T19:32:08.937971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094518+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:08.094518+0000","last_peered":"2024-03-16T19:32:08.094518+0000","last_clean":"2024-03-16T19:32:08.094518+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:08.094518+0000","last_undegraded":"2024-03-16T19:32:08.094518+0000","last_fullsized":"2024-03-16T19:32:08.094518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351343+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:08.351343+0000","last_peered":"2024-03-16T19:32:08.351343+0000","last_clean":"2024-03-16T19:32:08.351343+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:08.351343+0000","last_undegraded":"2024-03-16T19:32:08.351343+0000","last_fullsized":"2024-03-16T19:32:08.351343+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:47.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083818+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:08.083818+0000","last_peered":"2024-03-16T19:32:08.083818+0000","last_clean":"2024-03-16T19:32:08.083818+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:08.083818+0000","last_undegraded":"2024-03-16T19:32:08.083818+0000","last_fullsized":"2024-03-16T19:32:08.083818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937685+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:08.937685+0000","last_peered":"2024-03-16T19:32:08.937685+0000","last_clean":"2024-03-16T19:32:08.937685+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:08.937685+0000","last_undegraded":"2024-03-16T19:32:08.937685+0000","last_fullsized":"2024-03-16T19:32:08.937685+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083640+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:08.083640+0000","last_peered":"2024-03-16T19:32:08.083640+0000","last_clean":"2024-03-16T19:32:08.083640+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:08.083640+0000","last_undegraded":"2024-03-16T19:32:08.083640+0000","last_fullsized":"2024-03-16T19:32:08.083640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937556+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:08.937556+0000","last_peered":"2024-03-16T19:32:08.937556+0000","last_clean":"2024-03-16T19:32:08.937556+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:08.937556+0000","last_undegraded":"2024-03-16T19:32:08.937556+0000","last_fullsized":"2024-03-16T19:32:08.937556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086880+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:08.086880+0000","last_peered":"2024-03-16T19:32:08.086880+0000","last_clean":"2024-03-16T19:32:08.086880+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:08.086880+0000","last_undegraded":"2024-03-16T19:32:08.086880+0000","last_fullsized":"2024-03-16T19:32:08.086880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:58:31.209988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937978+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:08.937978+0000","last_peered":"2024-03-16T19:32:08.937978+0000","last_clean":"2024-03-16T19:32:08.937978+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:08.937978+0000","last_undegraded":"2024-03-16T19:32:08.937978+0000","last_fullsized":"2024-03-16T19:32:08.937978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287814+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:09.287814+0000","last_peered":"2024-03-16T19:32:09.287814+0000","last_clean":"2024-03-16T19:32:09.287814+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:09.287814+0000","last_undegraded":"2024-03-16T19:32:09.287814+0000","last_fullsized":"2024-03-16T19:32:09.287814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287704+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:09.287704+0000","last_peered":"2024-03-16T19:32:09.287704+0000","last_clean":"2024-03-16T19:32:09.287704+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:09.287704+0000","last_undegraded":"2024-03-16T19:32:09.287704+0000","last_fullsized":"2024-03-16T19:32:09.287704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084143+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:08.084143+0000","last_peered":"2024-03-16T19:32:08.084143+0000","last_clean":"2024-03-16T19:32:08.084143+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:08.084143+0000","last_undegraded":"2024-03-16T19:32:08.084143+0000","last_fullsized":"2024-03-16T19:32:08.084143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:17.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088886+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:08.088886+0000","last_peered":"2024-03-16T19:32:08.088886+0000","last_clean":"2024-03-16T19:32:08.088886+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:08.088886+0000","last_undegraded":"2024-03-16T19:32:08.088886+0000","last_fullsized":"2024-03-16T19:32:08.088886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:52.102388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"84'3","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350888+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:08.350888+0000","last_peered":"2024-03-16T19:32:08.350888+0000","last_clean":"2024-03-16T19:32:08.350888+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:08.350888+0000","last_undegraded":"2024-03-16T19:32:08.350888+0000","last_fullsized":"2024-03-16T19:32:08.350888+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:39.289238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094126+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:08.094126+0000","last_peered":"2024-03-16T19:32:08.094126+0000","last_clean":"2024-03-16T19:32:08.094126+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:08.094126+0000","last_undegraded":"2024-03-16T19:32:08.094126+0000","last_fullsized":"2024-03-16T19:32:08.094126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083403+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:08.083403+0000","last_peered":"2024-03-16T19:32:08.083403+0000","last_clean":"2024-03-16T19:32:08.083403+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:08.083403+0000","last_undegraded":"2024-03-16T19:32:08.083403+0000","last_fullsized":"2024-03-16T19:32:08.083403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084254+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:08.084254+0000","last_peered":"2024-03-16T19:32:08.084254+0000","last_clean":"2024-03-16T19:32:08.084254+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:08.084254+0000","last_undegraded":"2024-03-16T19:32:08.084254+0000","last_fullsized":"2024-03-16T19:32:08.084254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087017+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:08.087017+0000","last_peered":"2024-03-16T19:32:08.087017+0000","last_clean":"2024-03-16T19:32:08.087017+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:08.087017+0000","last_undegraded":"2024-03-16T19:32:08.087017+0000","last_fullsized":"2024-03-16T19:32:08.087017+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:36:06.175327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937878+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:08.937878+0000","last_peered":"2024-03-16T19:32:08.937878+0000","last_clean":"2024-03-16T19:32:08.937878+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:08.937878+0000","last_undegraded":"2024-03-16T19:32:08.937878+0000","last_fullsized":"2024-03-16T19:32:08.937878+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287690+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:09.287690+0000","last_peered":"2024-03-16T19:32:09.287690+0000","last_clean":"2024-03-16T19:32:09.287690+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:09.287690+0000","last_undegraded":"2024-03-16T19:32:09.287690+0000","last_fullsized":"2024-03-16T19:32:09.287690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351462+0000","last_change":"2024-03-16T19:31:51.807295+0000","last_active":"2024-03-16T19:32:08.351462+0000","last_peered":"2024-03-16T19:32:08.351462+0000","last_clean":"2024-03-16T19:32:08.351462+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:08.351462+0000","last_undegraded":"2024-03-16T19:32:08.351462+0000","last_fullsized":"2024-03-16T19:32:08.351462+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:51:04.578640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.085957+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:08.085957+0000","last_peered":"2024-03-16T19:32:08.085957+0000","last_clean":"2024-03-16T19:32:08.085957+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:08.085957+0000","last_undegraded":"2024-03-16T19:32:08.085957+0000","last_fullsized":"2024-03-16T19:32:08.085957+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:41:06.377004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088245+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:08.088245+0000","last_peered":"2024-03-16T19:32:08.088245+0000","last_clean":"2024-03-16T19:32:08.088245+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:08.088245+0000","last_undegraded":"2024-03-16T19:32:08.088245+0000","last_fullsized":"2024-03-16T19:32:08.088245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:47:08.251341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083454+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:08.083454+0000","last_peered":"2024-03-16T19:32:08.083454+0000","last_clean":"2024-03-16T19:32:08.083454+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:08.083454+0000","last_undegraded":"2024-03-16T19:32:08.083454+0000","last_fullsized":"2024-03-16T19:32:08.083454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086134+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:08.086134+0000","last_peered":"2024-03-16T19:32:08.086134+0000","last_clean":"2024-03-16T19:32:08.086134+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:08.086134+0000","last_undegraded":"2024-03-16T19:32:08.086134+0000","last_fullsized":"2024-03-16T19:32:08.086134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:10.818175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287911+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:09.287911+0000","last_peered":"2024-03-16T19:32:09.287911+0000","last_clean":"2024-03-16T19:32:09.287911+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:09.287911+0000","last_undegraded":"2024-03-16T19:32:09.287911+0000","last_fullsized":"2024-03-16T19:32:09.287911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":187,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093433+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:08.093433+0000","last_peered":"2024-03-16T19:32:08.093433+0000","last_clean":"2024-03-16T19:32:08.093433+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:08.093433+0000","last_undegraded":"2024-03-16T19:32:08.093433+0000","last_fullsized":"2024-03-16T19:32:08.093433+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087892+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:08.087892+0000","last_peered":"2024-03-16T19:32:08.087892+0000","last_clean":"2024-03-16T19:32:08.087892+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:08.087892+0000","last_undegraded":"2024-03-16T19:32:08.087892+0000","last_fullsized":"2024-03-16T19:32:08.087892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:12:08.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350890+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:08.350890+0000","last_peered":"2024-03-16T19:32:08.350890+0000","last_clean":"2024-03-16T19:32:08.350890+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:08.350890+0000","last_undegraded":"2024-03-16T19:32:08.350890+0000","last_fullsized":"2024-03-16T19:32:08.350890+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:03:12.646493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083749+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:08.083749+0000","last_peered":"2024-03-16T19:32:08.083749+0000","last_clean":"2024-03-16T19:32:08.083749+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:08.083749+0000","last_undegraded":"2024-03-16T19:32:08.083749+0000","last_fullsized":"2024-03-16T19:32:08.083749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087224+0000","last_change":"2024-03-16T19:31:49.974297+0000","last_active":"2024-03-16T19:32:08.087224+0000","last_peered":"2024-03-16T19:32:08.087224+0000","last_clean":"2024-03-16T19:32:08.087224+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:08.087224+0000","last_undegraded":"2024-03-16T19:32:08.087224+0000","last_fullsized":"2024-03-16T19:32:08.087224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:51:57.535372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287591+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:09.287591+0000","last_peered":"2024-03-16T19:32:09.287591+0000","last_clean":"2024-03-16T19:32:09.287591+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:09.287591+0000","last_undegraded":"2024-03-16T19:32:09.287591+0000","last_fullsized":"2024-03-16T19:32:09.287591+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084174+0000","last_change":"2024-03-16T19:29:26.356478+0000","last_active":"2024-03-16T19:32:08.084174+0000","last_peered":"2024-03-16T19:32:08.084174+0000","last_clean":"2024-03-16T19:32:08.084174+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:08.084174+0000","last_undegraded":"2024-03-16T19:32:08.084174+0000","last_fullsized":"2024-03-16T19:32:08.084174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:42:10.971854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089011+0000","last_change":"2024-03-16T19:31:26.808032+0000","last_active":"2024-03-16T19:32:08.089011+0000","last_peered":"2024-03-16T19:32:08.089011+0000","last_clean":"2024-03-16T19:32:08.089011+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:08.089011+0000","last_undegraded":"2024-03-16T19:32:08.089011+0000","last_fullsized":"2024-03-16T19:32:08.089011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:15.218778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351569+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:08.351569+0000","last_peered":"2024-03-16T19:32:08.351569+0000","last_clean":"2024-03-16T19:32:08.351569+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:08.351569+0000","last_undegraded":"2024-03-16T19:32:08.351569+0000","last_fullsized":"2024-03-16T19:32:08.351569+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:16:31.606469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287611+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:09.287611+0000","last_peered":"2024-03-16T19:32:09.287611+0000","last_clean":"2024-03-16T19:32:09.287611+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:09.287611+0000","last_undegraded":"2024-03-16T19:32:09.287611+0000","last_fullsized":"2024-03-16T19:32:09.287611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086078+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086078+0000","last_peered":"2024-03-16T19:32:08.086078+0000","last_clean":"2024-03-16T19:32:08.086078+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086078+0000","last_undegraded":"2024-03-16T19:32:08.086078+0000","last_fullsized":"2024-03-16T19:32:08.086078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:23:19.824904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086998+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:08.086998+0000","last_peered":"2024-03-16T19:32:08.086998+0000","last_clean":"2024-03-16T19:32:08.086998+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:08.086998+0000","last_undegraded":"2024-03-16T19:32:08.086998+0000","last_fullsized":"2024-03-16T19:32:08.086998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:45.782085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094004+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:08.094004+0000","last_peered":"2024-03-16T19:32:08.094004+0000","last_clean":"2024-03-16T19:32:08.094004+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:08.094004+0000","last_undegraded":"2024-03-16T19:32:08.094004+0000","last_fullsized":"2024-03-16T19:32:08.094004+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287793+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:09.287793+0000","last_peered":"2024-03-16T19:32:09.287793+0000","last_clean":"2024-03-16T19:32:09.287793+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:09.287793+0000","last_undegraded":"2024-03-16T19:32:09.287793+0000","last_fullsized":"2024-03-16T19:32:09.287793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083754+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:08.083754+0000","last_peered":"2024-03-16T19:32:08.083754+0000","last_clean":"2024-03-16T19:32:08.083754+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:08.083754+0000","last_undegraded":"2024-03-16T19:32:08.083754+0000","last_fullsized":"2024-03-16T19:32:08.083754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351532+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:08.351532+0000","last_peered":"2024-03-16T19:32:08.351532+0000","last_clean":"2024-03-16T19:32:08.351532+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:08.351532+0000","last_undegraded":"2024-03-16T19:32:08.351532+0000","last_fullsized":"2024-03-16T19:32:08.351532+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:16.418962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093565+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:08.093565+0000","last_peered":"2024-03-16T19:32:08.093565+0000","last_clean":"2024-03-16T19:32:08.093565+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:08.093565+0000","last_undegraded":"2024-03-16T19:32:08.093565+0000","last_fullsized":"2024-03-16T19:32:08.093565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938817+0000","last_change":"2024-03-16T19:32:08.938817+0000","last_active":"2024-03-16T19:32:08.938817+0000","last_peered":"2024-03-16T19:32:08.938817+0000","last_clean":"2024-03-16T19:32:08.938817+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:08.938817+0000","last_undegraded":"2024-03-16T19:32:08.938817+0000","last_fullsized":"2024-03-16T19:32:08.938817+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:39:32.174855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"86'23","reported_seq":233,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287734+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:09.287734+0000","last_peered":"2024-03-16T19:32:09.287734+0000","last_clean":"2024-03-16T19:32:09.287734+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:09.287734+0000","last_undegraded":"2024-03-16T19:32:09.287734+0000","last_fullsized":"2024-03-16T19:32:09.287734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39292,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937730+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:08.937730+0000","last_peered":"2024-03-16T19:32:08.937730+0000","last_clean":"2024-03-16T19:32:08.937730+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:08.937730+0000","last_undegraded":"2024-03-16T19:32:08.937730+0000","last_fullsized":"2024-03-16T19:32:08.937730+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351564+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:08.351564+0000","last_peered":"2024-03-16T19:32:08.351564+0000","last_clean":"2024-03-16T19:32:08.351564+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:08.351564+0000","last_undegraded":"2024-03-16T19:32:08.351564+0000","last_fullsized":"2024-03-16T19:32:08.351564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:59.818393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351224+0000","last_change":"2024-03-16T19:31:51.807895+0000","last_active":"2024-03-16T19:32:08.351224+0000","last_peered":"2024-03-16T19:32:08.351224+0000","last_clean":"2024-03-16T19:32:08.351224+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:08.351224+0000","last_undegraded":"2024-03-16T19:32:08.351224+0000","last_fullsized":"2024-03-16T19:32:08.351224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:25:25.988237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937695+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:08.937695+0000","last_peered":"2024-03-16T19:32:08.937695+0000","last_clean":"2024-03-16T19:32:08.937695+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:08.937695+0000","last_undegraded":"2024-03-16T19:32:08.937695+0000","last_fullsized":"2024-03-16T19:32:08.937695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287610+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:09.287610+0000","last_peered":"2024-03-16T19:32:09.287610+0000","last_clean":"2024-03-16T19:32:09.287610+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:09.287610+0000","last_undegraded":"2024-03-16T19:32:09.287610+0000","last_fullsized":"2024-03-16T19:32:09.287610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287595+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:09.287595+0000","last_peered":"2024-03-16T19:32:09.287595+0000","last_clean":"2024-03-16T19:32:09.287595+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:09.287595+0000","last_undegraded":"2024-03-16T19:32:09.287595+0000","last_fullsized":"2024-03-16T19:32:09.287595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087374+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:08.087374+0000","last_peered":"2024-03-16T19:32:08.087374+0000","last_clean":"2024-03-16T19:32:08.087374+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:08.087374+0000","last_undegraded":"2024-03-16T19:32:08.087374+0000","last_fullsized":"2024-03-16T19:32:08.087374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:33.634146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937562+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:08.937562+0000","last_peered":"2024-03-16T19:32:08.937562+0000","last_clean":"2024-03-16T19:32:08.937562+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:08.937562+0000","last_undegraded":"2024-03-16T19:32:08.937562+0000","last_fullsized":"2024-03-16T19:32:08.937562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084319+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084319+0000","last_peered":"2024-03-16T19:32:08.084319+0000","last_clean":"2024-03-16T19:32:08.084319+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084319+0000","last_undegraded":"2024-03-16T19:32:08.084319+0000","last_fullsized":"2024-03-16T19:32:08.084319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:53.021797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937869+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:08.937869+0000","last_peered":"2024-03-16T19:32:08.937869+0000","last_clean":"2024-03-16T19:32:08.937869+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:08.937869+0000","last_undegraded":"2024-03-16T19:32:08.937869+0000","last_fullsized":"2024-03-16T19:32:08.937869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":55,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086917+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:08.086917+0000","last_peered":"2024-03-16T19:32:08.086917+0000","last_clean":"2024-03-16T19:32:08.086917+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:08.086917+0000","last_undegraded":"2024-03-16T19:32:08.086917+0000","last_fullsized":"2024-03-16T19:32:08.086917+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:07:39.853290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":179,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.938036+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:08.938036+0000","last_peered":"2024-03-16T19:32:08.938036+0000","last_clean":"2024-03-16T19:32:08.938036+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:08.938036+0000","last_undegraded":"2024-03-16T19:32:08.938036+0000","last_fullsized":"2024-03-16T19:32:08.938036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093703+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:08.093703+0000","last_peered":"2024-03-16T19:32:08.093703+0000","last_clean":"2024-03-16T19:32:08.093703+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:08.093703+0000","last_undegraded":"2024-03-16T19:32:08.093703+0000","last_fullsized":"2024-03-16T19:32:08.093703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084002+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:08.084002+0000","last_peered":"2024-03-16T19:32:08.084002+0000","last_clean":"2024-03-16T19:32:08.084002+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:08.084002+0000","last_undegraded":"2024-03-16T19:32:08.084002+0000","last_fullsized":"2024-03-16T19:32:08.084002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086673+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:08.086673+0000","last_peered":"2024-03-16T19:32:08.086673+0000","last_clean":"2024-03-16T19:32:08.086673+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:08.086673+0000","last_undegraded":"2024-03-16T19:32:08.086673+0000","last_fullsized":"2024-03-16T19:32:08.086673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:59:55.019036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":148,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937729+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:08.937729+0000","last_peered":"2024-03-16T19:32:08.937729+0000","last_clean":"2024-03-16T19:32:08.937729+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:08.937729+0000","last_undegraded":"2024-03-16T19:32:08.937729+0000","last_fullsized":"2024-03-16T19:32:08.937729+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":363,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937413+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:08.937413+0000","last_peered":"2024-03-16T19:32:08.937413+0000","last_clean":"2024-03-16T19:32:08.937413+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:08.937413+0000","last_undegraded":"2024-03-16T19:32:08.937413+0000","last_fullsized":"2024-03-16T19:32:08.937413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087541+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:08.087541+0000","last_peered":"2024-03-16T19:32:08.087541+0000","last_clean":"2024-03-16T19:32:08.087541+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:08.087541+0000","last_undegraded":"2024-03-16T19:32:08.087541+0000","last_fullsized":"2024-03-16T19:32:08.087541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:16:54.768710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937520+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:08.937520+0000","last_peered":"2024-03-16T19:32:08.937520+0000","last_clean":"2024-03-16T19:32:08.937520+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:08.937520+0000","last_undegraded":"2024-03-16T19:32:08.937520+0000","last_fullsized":"2024-03-16T19:32:08.937520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351264+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:08.351264+0000","last_peered":"2024-03-16T19:32:08.351264+0000","last_clean":"2024-03-16T19:32:08.351264+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:08.351264+0000","last_undegraded":"2024-03-16T19:32:08.351264+0000","last_fullsized":"2024-03-16T19:32:08.351264+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:44.308565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087130+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:08.087130+0000","last_peered":"2024-03-16T19:32:08.087130+0000","last_clean":"2024-03-16T19:32:08.087130+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:08.087130+0000","last_undegraded":"2024-03-16T19:32:08.087130+0000","last_fullsized":"2024-03-16T19:32:08.087130+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:20:21.014268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.096035+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:08.096035+0000","last_peered":"2024-03-16T19:32:08.096035+0000","last_clean":"2024-03-16T19:32:08.096035+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:08.096035+0000","last_undegraded":"2024-03-16T19:32:08.096035+0000","last_fullsized":"2024-03-16T19:32:08.096035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084314+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:08.084314+0000","last_peered":"2024-03-16T19:32:08.084314+0000","last_clean":"2024-03-16T19:32:08.084314+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:08.084314+0000","last_undegraded":"2024-03-16T19:32:08.084314+0000","last_fullsized":"2024-03-16T19:32:08.084314+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:01.856961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093299+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:08.093299+0000","last_peered":"2024-03-16T19:32:08.093299+0000","last_clean":"2024-03-16T19:32:08.093299+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:08.093299+0000","last_undegraded":"2024-03-16T19:32:08.093299+0000","last_fullsized":"2024-03-16T19:32:08.093299+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084193+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:08.084193+0000","last_peered":"2024-03-16T19:32:08.084193+0000","last_clean":"2024-03-16T19:32:08.084193+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:08.084193+0000","last_undegraded":"2024-03-16T19:32:08.084193+0000","last_fullsized":"2024-03-16T19:32:08.084193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937906+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:08.937906+0000","last_peered":"2024-03-16T19:32:08.937906+0000","last_clean":"2024-03-16T19:32:08.937906+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:08.937906+0000","last_undegraded":"2024-03-16T19:32:08.937906+0000","last_fullsized":"2024-03-16T19:32:08.937906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093171+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:08.093171+0000","last_peered":"2024-03-16T19:32:08.093171+0000","last_clean":"2024-03-16T19:32:08.093171+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:08.093171+0000","last_undegraded":"2024-03-16T19:32:08.093171+0000","last_fullsized":"2024-03-16T19:32:08.093171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'11","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094654+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:08.094654+0000","last_peered":"2024-03-16T19:32:08.094654+0000","last_clean":"2024-03-16T19:32:08.094654+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:08.094654+0000","last_undegraded":"2024-03-16T19:32:08.094654+0000","last_fullsized":"2024-03-16T19:32:08.094654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088741+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:08.088741+0000","last_peered":"2024-03-16T19:32:08.088741+0000","last_clean":"2024-03-16T19:32:08.088741+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:08.088741+0000","last_undegraded":"2024-03-16T19:32:08.088741+0000","last_fullsized":"2024-03-16T19:32:08.088741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:18.996236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287797+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:09.287797+0000","last_peered":"2024-03-16T19:32:09.287797+0000","last_clean":"2024-03-16T19:32:09.287797+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:09.287797+0000","last_undegraded":"2024-03-16T19:32:09.287797+0000","last_fullsized":"2024-03-16T19:32:09.287797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.350908+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:08.350908+0000","last_peered":"2024-03-16T19:32:08.350908+0000","last_clean":"2024-03-16T19:32:08.350908+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:08.350908+0000","last_undegraded":"2024-03-16T19:32:08.350908+0000","last_fullsized":"2024-03-16T19:32:08.350908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:27:02.024100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937470+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:08.937470+0000","last_peered":"2024-03-16T19:32:08.937470+0000","last_clean":"2024-03-16T19:32:08.937470+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:08.937470+0000","last_undegraded":"2024-03-16T19:32:08.937470+0000","last_fullsized":"2024-03-16T19:32:08.937470+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083795+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:08.083795+0000","last_peered":"2024-03-16T19:32:08.083795+0000","last_clean":"2024-03-16T19:32:08.083795+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:08.083795+0000","last_undegraded":"2024-03-16T19:32:08.083795+0000","last_fullsized":"2024-03-16T19:32:08.083795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:46:49.628764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094246+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:08.094246+0000","last_peered":"2024-03-16T19:32:08.094246+0000","last_clean":"2024-03-16T19:32:08.094246+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:08.094246+0000","last_undegraded":"2024-03-16T19:32:08.094246+0000","last_fullsized":"2024-03-16T19:32:08.094246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.102711+0000","last_change":"2024-03-16T19:32:08.101455+0000","last_active":"2024-03-16T19:32:08.102711+0000","last_peered":"2024-03-16T19:32:08.102711+0000","last_clean":"2024-03-16T19:32:08.102711+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:32:08.102711+0000","last_undegraded":"2024-03-16T19:32:08.102711+0000","last_fullsized":"2024-03-16T19:32:08.102711+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:20:10.984083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.095090+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:08.095090+0000","last_peered":"2024-03-16T19:32:08.095090+0000","last_clean":"2024-03-16T19:32:08.095090+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:08.095090+0000","last_undegraded":"2024-03-16T19:32:08.095090+0000","last_fullsized":"2024-03-16T19:32:08.095090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.089208+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:08.089208+0000","last_peered":"2024-03-16T19:32:08.089208+0000","last_clean":"2024-03-16T19:32:08.089208+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:08.089208+0000","last_undegraded":"2024-03-16T19:32:08.089208+0000","last_fullsized":"2024-03-16T19:32:08.089208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:03.618618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937781+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:08.937781+0000","last_peered":"2024-03-16T19:32:08.937781+0000","last_clean":"2024-03-16T19:32:08.937781+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:08.937781+0000","last_undegraded":"2024-03-16T19:32:08.937781+0000","last_fullsized":"2024-03-16T19:32:08.937781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":178,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287721+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:09.287721+0000","last_peered":"2024-03-16T19:32:09.287721+0000","last_clean":"2024-03-16T19:32:09.287721+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:09.287721+0000","last_undegraded":"2024-03-16T19:32:09.287721+0000","last_fullsized":"2024-03-16T19:32:09.287721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084106+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:08.084106+0000","last_peered":"2024-03-16T19:32:08.084106+0000","last_clean":"2024-03-16T19:32:08.084106+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:08.084106+0000","last_undegraded":"2024-03-16T19:32:08.084106+0000","last_fullsized":"2024-03-16T19:32:08.084106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937766+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:08.937766+0000","last_peered":"2024-03-16T19:32:08.937766+0000","last_clean":"2024-03-16T19:32:08.937766+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:08.937766+0000","last_undegraded":"2024-03-16T19:32:08.937766+0000","last_fullsized":"2024-03-16T19:32:08.937766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083913+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:08.083913+0000","last_peered":"2024-03-16T19:32:08.083913+0000","last_clean":"2024-03-16T19:32:08.083913+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:08.083913+0000","last_undegraded":"2024-03-16T19:32:08.083913+0000","last_fullsized":"2024-03-16T19:32:08.083913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937841+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:08.937841+0000","last_peered":"2024-03-16T19:32:08.937841+0000","last_clean":"2024-03-16T19:32:08.937841+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:08.937841+0000","last_undegraded":"2024-03-16T19:32:08.937841+0000","last_fullsized":"2024-03-16T19:32:08.937841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":164,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351010+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:08.351010+0000","last_peered":"2024-03-16T19:32:08.351010+0000","last_clean":"2024-03-16T19:32:08.351010+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:08.351010+0000","last_undegraded":"2024-03-16T19:32:08.351010+0000","last_fullsized":"2024-03-16T19:32:08.351010+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:19.317523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088482+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088482+0000","last_peered":"2024-03-16T19:32:08.088482+0000","last_clean":"2024-03-16T19:32:08.088482+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088482+0000","last_undegraded":"2024-03-16T19:32:08.088482+0000","last_fullsized":"2024-03-16T19:32:08.088482+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:58:01.695714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":160,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086003+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:08.086003+0000","last_peered":"2024-03-16T19:32:08.086003+0000","last_clean":"2024-03-16T19:32:08.086003+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:08.086003+0000","last_undegraded":"2024-03-16T19:32:08.086003+0000","last_fullsized":"2024-03-16T19:32:08.086003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:00:17.040490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351503+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:08.351503+0000","last_peered":"2024-03-16T19:32:08.351503+0000","last_clean":"2024-03-16T19:32:08.351503+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:08.351503+0000","last_undegraded":"2024-03-16T19:32:08.351503+0000","last_fullsized":"2024-03-16T19:32:08.351503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:28:10.116570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40494,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":80988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28840,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343000,"statfs":{"total":96636764160,"available":96607232000,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1310720,"data_stored":634440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343536,"statfs":{"total":96636764160,"available":96607780864,"internally_reserved":0,"allocated":1314816,"data_stored":634844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122056,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29408,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342432,"statfs":{"total":96636764160,"available":96606650368,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28848,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342992,"statfs":{"total":96636764160,"available":96607223808,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:23.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T19:32:23.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.661+0000 7f145ee84640 1 Processor -- start 2024-03-16T19:32:23.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.661+0000 7f145ee84640 1 -- start start 2024-03-16T19:32:23.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f145ee84640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f145806bef0 0x7f1458154660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f145ee84640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14580cc720 con 0x7f14580cb560 2024-03-16T19:32:23.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f145ee84640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14580cc9c0 con 0x7f145806bb10 2024-03-16T19:32:23.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f145ee84640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f145815de30 con 0x7f145806bef0 2024-03-16T19:32:23.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f145d421640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f14580cb560 0x7f14581651a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40034/0 (socket says 172.21.15.172:40034) 2024-03-16T19:32:23.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f145d421640 1 -- 172.21.15.172:0/537501242 learned_addr learned my addr 172.21.15.172:0/537501242 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:23.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f1457fff640 1 --2- 172.21.15.172:0/537501242 >> v2:172.21.15.174:3300/0 conn(0x7f145806bef0 0x7f1458154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f14577fe640 1 -- 172.21.15.172:0/537501242 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024848028 0 0) 0x7f14580cc9c0 con 0x7f145806bb10 2024-03-16T19:32:23.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f14577fe640 1 -- 172.21.15.172:0/537501242 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1448003520 con 0x7f145806bb10 2024-03-16T19:32:23.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f14577fe640 1 -- 172.21.15.172:0/537501242 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114360339 0 0) 0x7f14580cc720 con 0x7f14580cb560 2024-03-16T19:32:23.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f14577fe640 1 -- 172.21.15.172:0/537501242 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14580cc9c0 con 0x7f14580cb560 2024-03-16T19:32:23.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f1457fff640 1 -- 172.21.15.172:0/537501242 >> v1:172.21.15.174:6789/0 conn(0x7f145806bb10 legacy=0x7f14580cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f1457fff640 1 -- 172.21.15.172:0/537501242 >> v1:172.21.15.172:6789/0 conn(0x7f14580cb560 legacy=0x7f14581651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.662+0000 7f1457fff640 1 -- 172.21.15.172:0/537501242 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145815e110 con 0x7f145806bef0 2024-03-16T19:32:23.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.663+0000 7f1457fff640 1 --2- 172.21.15.172:0/537501242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f145806bef0 0x7f1458154660 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f14440097a0 tx=0x7f144402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cee26df5abd5316 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:23.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.663+0000 7f14577fe640 1 -- 172.21.15.172:0/537501242 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144402f810 con 0x7f145806bef0 2024-03-16T19:32:23.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.664+0000 7f14577fe640 1 -- 172.21.15.172:0/537501242 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144402f970 con 0x7f145806bef0 2024-03-16T19:32:23.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.664+0000 7f14577fe640 1 -- 172.21.15.172:0/537501242 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144402fd00 con 0x7f145806bef0 2024-03-16T19:32:23.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.664+0000 7f145ee84640 1 -- 172.21.15.172:0/537501242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f145806bef0 msgr2=0x7f1458154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.664+0000 7f145ee84640 1 --2- 172.21.15.172:0/537501242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f145806bef0 0x7f1458154660 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f14440097a0 tx=0x7f144402b9f0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.664+0000 7f145ee84640 1 -- 172.21.15.172:0/537501242 shutdown_connections 2024-03-16T19:32:23.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.664+0000 7f145ee84640 1 --2- 172.21.15.172:0/537501242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f145806bef0 0x7f1458154660 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.664+0000 7f145ee84640 1 -- 172.21.15.172:0/537501242 >> 172.21.15.172:0/537501242 conn(0x7f1458067250 msgr2=0x7f14580d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:23.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.664+0000 7f145ee84640 1 -- 172.21.15.172:0/537501242 shutdown_connections 2024-03-16T19:32:23.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.664+0000 7f145ee84640 1 -- 172.21.15.172:0/537501242 wait complete. 2024-03-16T19:32:23.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.665+0000 7f145ee84640 1 Processor -- start 2024-03-16T19:32:23.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.665+0000 7f145ee84640 1 -- start start 2024-03-16T19:32:23.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145ee84640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145806bb10 0x7f1458147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145ee84640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f145806bef0 0x7f14580b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145ee84640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14580cb560 0x7f145815cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145ee84640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14580b42c0 con 0x7f145806bb10 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145ee84640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1458148e50 con 0x7f14580cb560 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145ee84640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14580b4510 con 0x7f145806bef0 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145cc20640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145806bb10 0x7f1458147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145cc20640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145806bb10 0x7f1458147140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46714/0 (socket says 172.21.15.172:46714) 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145cc20640 1 -- 172.21.15.172:0/1305688513 learned_addr learned my addr 172.21.15.172:0/1305688513 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f145d421640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14580cb560 0x7f145815cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.666+0000 7f1457fff640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f145806bef0 0x7f14580b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f145d421640 1 -- 172.21.15.172:0/1305688513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f145806bef0 msgr2=0x7f14580b52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f145d421640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f145806bef0 0x7f14580b52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f145d421640 1 -- 172.21.15.172:0/1305688513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145806bb10 msgr2=0x7f1458147140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f145d421640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145806bb10 0x7f1458147140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f145d421640 1 -- 172.21.15.172:0/1305688513 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1444009400 con 0x7f14580cb560 2024-03-16T19:32:23.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f145d421640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14580cb560 0x7f145815cdb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f145000caa0 tx=0x7f145000cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:23.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f14557fa640 1 -- 172.21.15.172:0/1305688513 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145000ab40 con 0x7f14580cb560 2024-03-16T19:32:23.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f14557fa640 1 -- 172.21.15.172:0/1305688513 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145000aca0 con 0x7f14580cb560 2024-03-16T19:32:23.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14580b47f0 con 0x7f14580cb560 2024-03-16T19:32:23.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f14557fa640 1 -- 172.21.15.172:0/1305688513 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450012600 con 0x7f14580cb560 2024-03-16T19:32:23.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.667+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14580b4db0 con 0x7f14580cb560 2024-03-16T19:32:23.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.668+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14580681c0 con 0x7f14580cb560 2024-03-16T19:32:23.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.668+0000 7f14557fa640 1 -- 172.21.15.172:0/1305688513 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1450007070 con 0x7f14580cb560 2024-03-16T19:32:23.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.669+0000 7f14557fa640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f143c06b3b0 0x7f143c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:23.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.669+0000 7f14557fa640 1 -- 172.21.15.172:0/1305688513 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f1450089b50 con 0x7f14580cb560 2024-03-16T19:32:23.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.669+0000 7f145cc20640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f143c06b3b0 0x7f143c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:23.650 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.669+0000 7f145cc20640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f143c06b3b0 0x7f143c06d870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f144c007fd0 tx=0x7f144c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:23.652 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.673+0000 7f14557fa640 1 -- 172.21.15.172:0/1305688513 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1450054730 con 0x7f14580cb560 2024-03-16T19:32:23.878 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.898+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14580c6cf0 con 0x7f14580cb560 2024-03-16T19:32:23.879 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.899+0000 7f14557fa640 1 -- 172.21.15.172:0/1305688513 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16884 (secure 0 0 0) 0x7f1450017070 con 0x7f14580cb560 2024-03-16T19:32:23.879 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:23.879 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":106,"fsid":"6c00e600-a879-4eb6-acee-0fdf0f76922a","created":"2024-03-16T19:29:02.010581+0000","modified":"2024-03-16T19:32:08.075358+0000","last_up_change":"2024-03-16T19:31:48.956759+0000","last_in_change":"2024-03-16T19:31:48.956759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T19:29:14.242742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T19:29:24.121132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T19:29:24.651099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"460ed9e4-eacf-4544-9199-23ca7208b663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6801","nonce":491754535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6803","nonce":491754535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6807","nonce":491754535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":491754535},{"type":"v1","addr":"172.21.15.172:6805","nonce":491754535}]},"public_addr":"172.21.15.172:6801/491754535","cluster_addr":"172.21.15.172:6803/491754535","heartbeat_back_addr":"172.21.15.172:6807/491754535","heartbeat_front_addr":"172.21.15.172:6805/491754535","state":["exists","up"]},{"osd":1,"uuid":"b84ebb50-079d-460f-bf05-71040f17500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6809","nonce":3098410089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6811","nonce":3098410089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6815","nonce":3098410089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":3098410089},{"type":"v1","addr":"172.21.15.172:6813","nonce":3098410089}]},"public_addr":"172.21.15.172:6809/3098410089","cluster_addr":"172.21.15.172:6811/3098410089","heartbeat_back_addr":"172.21.15.172:6815/3098410089","heartbeat_front_addr":"172.21.15.172:6813/3098410089","state":["exists","up"]},{"osd":2,"uuid":"6774c42c-5373-4543-8f8c-ed7b82c12057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":58,"up_from":88,"up_thru":105,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6823","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6825","nonce":3572014952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6827","nonce":3572014952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6831","nonce":3572014952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3572014952},{"type":"v1","addr":"172.21.15.172:6829","nonce":3572014952}]},"public_addr":"172.21.15.172:6825/3572014952","cluster_addr":"172.21.15.172:6827/3572014952","heartbeat_back_addr":"172.21.15.172:6831/3572014952","heartbeat_front_addr":"172.21.15.172:6829/3572014952","state":["exists","up"]},{"osd":3,"uuid":"b96838ef-d991-43ab-b2d7-16deab58ca78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6817","nonce":1613555619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6819","nonce":1613555619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6824","nonce":1613555619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1613555619},{"type":"v1","addr":"172.21.15.172:6821","nonce":1613555619}]},"public_addr":"172.21.15.172:6817/1613555619","cluster_addr":"172.21.15.172:6819/1613555619","heartbeat_back_addr":"172.21.15.172:6824/1613555619","heartbeat_front_addr":"172.21.15.172:6821/1613555619","state":["exists","up"]},{"osd":4,"uuid":"f5c78559-d22e-4847-be3f-2f597d7f6789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6809","nonce":967866068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6811","nonce":967866068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6815","nonce":967866068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":967866068},{"type":"v1","addr":"172.21.15.174:6813","nonce":967866068}]},"public_addr":"172.21.15.174:6809/967866068","cluster_addr":"172.21.15.174:6811/967866068","heartbeat_back_addr":"172.21.15.174:6815/967866068","heartbeat_front_addr":"172.21.15.174:6813/967866068","state":["exists","up"]},{"osd":5,"uuid":"e1f4797f-31c3-4292-a4a1-88fa0ab64f9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6817","nonce":3985312519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6820","nonce":3985312519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6827","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6829","nonce":3985312519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":3985312519},{"type":"v1","addr":"172.21.15.174:6825","nonce":3985312519}]},"public_addr":"172.21.15.174:6817/3985312519","cluster_addr":"172.21.15.174:6820/3985312519","heartbeat_back_addr":"172.21.15.174:6829/3985312519","heartbeat_front_addr":"172.21.15.174:6825/3985312519","state":["exists","up"]},{"osd":6,"uuid":"ea3e888e-c4c6-444a-8ebe-e26cddb2084b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6819","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6821","nonce":3676139844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6823","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6824","nonce":3676139844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6831","nonce":3676139844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3676139844},{"type":"v1","addr":"172.21.15.174:6828","nonce":3676139844}]},"public_addr":"172.21.15.174:6821/3676139844","cluster_addr":"172.21.15.174:6824/3676139844","heartbeat_back_addr":"172.21.15.174:6831/3676139844","heartbeat_front_addr":"172.21.15.174:6828/3676139844","state":["exists","up"]},{"osd":7,"uuid":"bb0252a5-3aa1-43f0-be38-37413819720b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6801","nonce":339941177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6803","nonce":339941177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6807","nonce":339941177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":339941177},{"type":"v1","addr":"172.21.15.174:6805","nonce":339941177}]},"public_addr":"172.21.15.174:6801/339941177","cluster_addr":"172.21.15.174:6803/339941177","heartbeat_back_addr":"172.21.15.174:6807/339941177","heartbeat_front_addr":"172.21.15.174:6805/339941177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.304258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.336924+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.359766+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.384687+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.291288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.318479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.313564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T19:29:09.232618+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T19:32:23.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f143c06b3b0 msgr2=0x7f143c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f143c06b3b0 0x7f143c06d870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f144c007fd0 tx=0x7f144c009040 comp rx=0 tx=0).stop 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14580cb560 msgr2=0x7f145815cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14580cb560 0x7f145815cdb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f145000caa0 tx=0x7f145000cea0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 shutdown_connections 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14580cb560 0x7f145815cdb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f143c06b3b0 0x7f143c06d870 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f145806bef0 0x7f14580b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 --2- 172.21.15.172:0/1305688513 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f145806bb10 0x7f1458147140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 >> 172.21.15.172:0/1305688513 conn(0x7f1458067250 msgr2=0x7f14581535a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 shutdown_connections 2024-03-16T19:32:23.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:23.902+0000 7f145ee84640 1 -- 172.21.15.172:0/1305688513 wait complete. 2024-03-16T19:32:24.897 INFO:tasks.ceph:Scrubbing osd.0 2024-03-16T19:32:24.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-16T19:32:25.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.057+0000 7fb7307e9640 1 Processor -- start 2024-03-16T19:32:25.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.057+0000 7fb7307e9640 1 -- start start 2024-03-16T19:32:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.058+0000 7fb7307e9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb728154cb0 0x7fb72806e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.058+0000 7fb7307e9640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb72806f590 0x7fb7280ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.058+0000 7fb7307e9640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72806edd0 con 0x7fb72806ef90 2024-03-16T19:32:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.058+0000 7fb7307e9640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7280c8ba0 con 0x7fb72806f590 2024-03-16T19:32:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.058+0000 7fb7307e9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb728155090 con 0x7fb728154cb0 2024-03-16T19:32:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.058+0000 7fb72ed86640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb72806f590 0x7fb7280ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.058+0000 7fb72ed86640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fb72806f590 0x7fb7280ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46720/0 (socket says 172.21.15.172:46720) 2024-03-16T19:32:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.058+0000 7fb72e585640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb728154cb0 0x7fb72806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.058+0000 7fb72ed86640 1 -- 172.21.15.172:0/3991798319 learned_addr learned my addr 172.21.15.172:0/3991798319 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.059+0000 7fb72ed86640 1 -- 172.21.15.172:0/3991798319 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb728154cb0 msgr2=0x7fb72806e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.059+0000 7fb72ed86640 1 --2- 172.21.15.172:0/3991798319 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb728154cb0 0x7fb72806e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.059+0000 7fb72ed86640 1 -- 172.21.15.172:0/3991798319 >> v1:172.21.15.172:6789/0 conn(0x7fb72806ef90 legacy=0x7fb7280ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.059+0000 7fb72ed86640 1 -- 172.21.15.172:0/3991798319 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72806c350 con 0x7fb72806f590 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.060+0000 7fb72ed86640 1 --2- 172.21.15.172:0/3991798319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb72806f590 0x7fb7280ce0e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb720009e50 tx=0x7fb72002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b9b8b59b36ab1d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.060+0000 7fb72d583640 1 -- 172.21.15.172:0/3991798319 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72002cb00 con 0x7fb72806f590 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.060+0000 7fb72d583640 1 -- 172.21.15.172:0/3991798319 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72002cc60 con 0x7fb72806f590 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.060+0000 7fb72d583640 1 -- 172.21.15.172:0/3991798319 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7200343f0 con 0x7fb72806f590 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.061+0000 7fb7307e9640 1 -- 172.21.15.172:0/3991798319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb72806f590 msgr2=0x7fb7280ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.061+0000 7fb7307e9640 1 --2- 172.21.15.172:0/3991798319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb72806f590 0x7fb7280ce0e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb720009e50 tx=0x7fb72002bf60 comp rx=0 tx=0).stop 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.061+0000 7fb7307e9640 1 -- 172.21.15.172:0/3991798319 shutdown_connections 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.061+0000 7fb7307e9640 1 --2- 172.21.15.172:0/3991798319 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb72806f590 0x7fb7280ce0e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.061+0000 7fb7307e9640 1 --2- 172.21.15.172:0/3991798319 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb728154cb0 0x7fb72806e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.061+0000 7fb7307e9640 1 -- 172.21.15.172:0/3991798319 >> 172.21.15.172:0/3991798319 conn(0x7fb728066c20 msgr2=0x7fb728067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.061+0000 7fb7307e9640 1 -- 172.21.15.172:0/3991798319 shutdown_connections 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.061+0000 7fb7307e9640 1 -- 172.21.15.172:0/3991798319 wait complete. 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.063+0000 7fb7307e9640 1 Processor -- start 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.063+0000 7fb7307e9640 1 -- start start 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb7307e9640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb72806ef90 0x7fb728152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb7307e9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb72806f590 0x7fb7280cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb7307e9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb728154cb0 0x7fb7281439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb7307e9640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7280cc8c0 con 0x7fb72806ef90 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb7307e9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7281f41e0 con 0x7fb728154cb0 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb7307e9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7281f44c0 con 0x7fb72806f590 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72e585640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb72806ef90 0x7fb728152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72dd84640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb72806f590 0x7fb7280cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72dd84640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb72806f590 0x7fb7280cbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44834/0 (socket says 172.21.15.172:44834) 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72dd84640 1 -- 172.21.15.172:0/2849633840 learned_addr learned my addr 172.21.15.172:0/2849633840 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72ed86640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb728154cb0 0x7fb7281439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72dd84640 1 -- 172.21.15.172:0/2849633840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb728154cb0 msgr2=0x7fb7281439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72dd84640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb728154cb0 0x7fb7281439f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72dd84640 1 -- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb72806ef90 msgr2=0x7fb728152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72dd84640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb72806ef90 0x7fb728152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72dd84640 1 -- 172.21.15.172:0/2849633840 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720009b30 con 0x7fb72806f590 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72ed86640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb728154cb0 0x7fb7281439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb72dd84640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb72806f590 0x7fb7280cbfa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb718001b80 tx=0x7fb71800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.065+0000 7fb71effd640 1 -- 172.21.15.172:0/2849633840 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb718001d30 con 0x7fb72806f590 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.066+0000 7fb71effd640 1 -- 172.21.15.172:0/2849633840 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7180050d0 con 0x7fb72806f590 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.066+0000 7fb71effd640 1 -- 172.21.15.172:0/2849633840 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7180054c0 con 0x7fb72806f590 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.066+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7281f47a0 con 0x7fb72806f590 2024-03-16T19:32:25.111 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.066+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7281f4d60 con 0x7fb72806f590 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.067+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb7280ca2c0 con 0x7fb72806f590 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.067+0000 7fb71effd640 1 -- 172.21.15.172:0/2849633840 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb7180138e0 con 0x7fb72806f590 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.067+0000 7fb71effd640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb70c06b430 0x7fb70c06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.068+0000 7fb71effd640 1 -- 172.21.15.172:0/2849633840 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fb718091a70 con 0x7fb72806f590 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.068+0000 7fb72e585640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb70c06b430 0x7fb70c06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.068+0000 7fb71effd640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb70c073890 0x7fb70c075d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.068+0000 7fb71effd640 1 -- 172.21.15.172:0/2849633840 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb70c076400 con 0x7fb70c073890 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.068+0000 7fb71effd640 1 -- 172.21.15.172:0/2849633840 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7180962c0 con 0x7fb72806f590 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.068+0000 7fb72ed86640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb70c073890 0x7fb70c075d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.068+0000 7fb72e585640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb70c06b430 0x7fb70c06d8f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb724009aa0 tx=0x7fb724009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.068+0000 7fb72ed86640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb70c073890 0x7fb70c075d30 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.069+0000 7fb71effd640 1 -- 172.21.15.172:0/2849633840 <== osd.0 v2:172.21.15.172:6800/491754535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb70c076400 con 0x7fb70c073890 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.090+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 --> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb72806c7e0 con 0x7fb70c073890 2024-03-16T19:32:25.112 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.090+0000 7fb71effd640 1 -- 172.21.15.172:0/2849633840 <== osd.0 v2:172.21.15.172:6800/491754535 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb72002c980 con 0x7fb70c073890 2024-03-16T19:32:25.113 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:32:25.113 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T19:32:25.113 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:32:25.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.090+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb70c073890 msgr2=0x7fb70c075d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.090+0000 7fb7307e9640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6800/491754535,v1:172.21.15.172:6801/491754535] conn(0x7fb70c073890 0x7fb70c075d30 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.091+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb70c06b430 msgr2=0x7fb70c06d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.091+0000 7fb7307e9640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fb70c06b430 0x7fb70c06d8f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb724009aa0 tx=0x7fb724009190 comp rx=0 tx=0).stop 2024-03-16T19:32:25.114 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.091+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb72806f590 msgr2=0x7fb7280cbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.091+0000 7fb7307e9640 1 --2- 172.21.15.172:0/2849633840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb72806f590 0x7fb7280cbfa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb718001b80 tx=0x7fb71800a430 comp rx=0 tx=0).stop 2024-03-16T19:32:25.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.091+0000 7fb72ed86640 1 -- 172.21.15.172:0/2849633840 reap_dead start 2024-03-16T19:32:25.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.091+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 shutdown_connections 2024-03-16T19:32:25.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.091+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 >> 172.21.15.172:0/2849633840 conn(0x7fb728066c20 msgr2=0x7fb7280cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:25.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.091+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 shutdown_connections 2024-03-16T19:32:25.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.091+0000 7fb7307e9640 1 -- 172.21.15.172:0/2849633840 wait complete. 2024-03-16T19:32:25.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-16T19:32:25.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.279+0000 7fafbde64640 1 Processor -- start 2024-03-16T19:32:25.259 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.279+0000 7fafbde64640 1 -- start start 2024-03-16T19:32:25.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafbde64640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fafb8151910 0x7fafb8072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafbde64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb80c3ea0 0x7fafb80c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafbde64640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb8072b10 con 0x7fafb80c3930 2024-03-16T19:32:25.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafbde64640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb80c2330 con 0x7fafb8151910 2024-03-16T19:32:25.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafbde64640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb80c7da0 con 0x7fafb80c3ea0 2024-03-16T19:32:25.260 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb77fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fafb8151910 0x7fafb8072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb77fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fafb8151910 0x7fafb8072fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46744/0 (socket says 172.21.15.172:46744) 2024-03-16T19:32:25.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb77fe640 1 -- 172.21.15.172:0/2633228119 learned_addr learned my addr 172.21.15.172:0/2633228119 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:25.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb6ffd640 1 --2- 172.21.15.172:0/2633228119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb80c3ea0 0x7fafb80c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb7fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fafb80c3930 0x7fafb80ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:37442/0 (socket says 172.21.15.172:37442) 2024-03-16T19:32:25.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb77fe640 1 -- 172.21.15.172:0/2633228119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb80c3ea0 msgr2=0x7fafb80c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:25.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb77fe640 1 --2- 172.21.15.172:0/2633228119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb80c3ea0 0x7fafb80c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb77fe640 1 -- 172.21.15.172:0/2633228119 >> v1:172.21.15.174:6789/0 conn(0x7fafb80c3930 legacy=0x7fafb80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb77fe640 1 -- 172.21.15.172:0/2633228119 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8071b10 con 0x7fafb8151910 2024-03-16T19:32:25.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.280+0000 7fafb6ffd640 1 --2- 172.21.15.172:0/2633228119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb80c3ea0 0x7fafb80c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:25.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.281+0000 7fafb77fe640 1 --2- 172.21.15.172:0/2633228119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb8151910 0x7fafb8072fa0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009870 tx=0x7fafa802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a043d07d074cecb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafb67fc640 1 -- 172.21.15.172:0/2633228119 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8030860 con 0x7fafb8151910 2024-03-16T19:32:25.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafb67fc640 1 -- 172.21.15.172:0/2633228119 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa80309c0 con 0x7fafb8151910 2024-03-16T19:32:25.262 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafb67fc640 1 -- 172.21.15.172:0/2633228119 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8030d90 con 0x7fafb8151910 2024-03-16T19:32:25.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafbde64640 1 -- 172.21.15.172:0/2633228119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb8151910 msgr2=0x7fafb8072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafbde64640 1 --2- 172.21.15.172:0/2633228119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb8151910 0x7fafb8072fa0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009870 tx=0x7fafa802b9a0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafbde64640 1 -- 172.21.15.172:0/2633228119 shutdown_connections 2024-03-16T19:32:25.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafbde64640 1 --2- 172.21.15.172:0/2633228119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb80c3ea0 0x7fafb80c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafbde64640 1 --2- 172.21.15.172:0/2633228119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb8151910 0x7fafb8072fa0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafbde64640 1 -- 172.21.15.172:0/2633228119 >> 172.21.15.172:0/2633228119 conn(0x7fafb806db70 msgr2=0x7fafb806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:25.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.282+0000 7fafbde64640 1 -- 172.21.15.172:0/2633228119 shutdown_connections 2024-03-16T19:32:25.263 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.283+0000 7fafbde64640 1 -- 172.21.15.172:0/2633228119 wait complete. 2024-03-16T19:32:25.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.284+0000 7fafbde64640 1 Processor -- start 2024-03-16T19:32:25.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.284+0000 7fafbde64640 1 -- start start 2024-03-16T19:32:25.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafbde64640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb80c3930 0x7fafb8162f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafbde64640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafb80c3ea0 0x7fafb8142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafbde64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb8151910 0x7fafb813d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafbde64640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8152380 con 0x7fafb80c3930 2024-03-16T19:32:25.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafbde64640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8152050 con 0x7fafb80c3ea0 2024-03-16T19:32:25.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafbde64640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb813f260 con 0x7fafb8151910 2024-03-16T19:32:25.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb80c3930 0x7fafb8162f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb7fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb8151910 0x7fafb813d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb80c3930 0x7fafb8162f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46748/0 (socket says 172.21.15.172:46748) 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb77fe640 1 -- 172.21.15.172:0/483556893 learned_addr learned my addr 172.21.15.172:0/483556893 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb6ffd640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafb80c3ea0 0x7fafb8142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb7fff640 1 -- 172.21.15.172:0/483556893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafb80c3ea0 msgr2=0x7fafb8142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb7fff640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafb80c3ea0 0x7fafb8142150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb7fff640 1 -- 172.21.15.172:0/483556893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb80c3930 msgr2=0x7fafb8162f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb7fff640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb80c3930 0x7fafb8162f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb7fff640 1 -- 172.21.15.172:0/483556893 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac009400 con 0x7fafb8151910 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.286+0000 7fafb7fff640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb8151910 0x7fafb813d910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fafb000c730 tx=0x7fafb000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.287+0000 7faf9ffff640 1 -- 172.21.15.172:0/483556893 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0004d60 con 0x7fafb8151910 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.287+0000 7faf9ffff640 1 -- 172.21.15.172:0/483556893 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb000a640 con 0x7fafb8151910 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.287+0000 7faf9ffff640 1 -- 172.21.15.172:0/483556893 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb000aa10 con 0x7fafb8151910 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.287+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa80094d0 con 0x7fafb8151910 2024-03-16T19:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.287+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb8164c00 con 0x7fafb8151910 2024-03-16T19:32:25.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.289+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb8152530 con 0x7fafb8151910 2024-03-16T19:32:25.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.289+0000 7faf9ffff640 1 -- 172.21.15.172:0/483556893 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fafb0007070 con 0x7fafb8151910 2024-03-16T19:32:25.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.289+0000 7faf9ffff640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafa406b3b0 0x7fafa406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.289+0000 7faf9ffff640 1 -- 172.21.15.172:0/483556893 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fafb00894f0 con 0x7fafb8151910 2024-03-16T19:32:25.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.290+0000 7fafb77fe640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafa406b3b0 0x7fafa406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.290+0000 7fafb77fe640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafa406b3b0 0x7fafa406d870 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fafa802b9e0 tx=0x7fafa8045040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.294+0000 7faf9ffff640 1 -- 172.21.15.172:0/483556893 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafb8152530 con 0x7fafb8151910 2024-03-16T19:32:25.542 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.563+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fafb80727a0 con 0x7fafa406b3b0 2024-03-16T19:32:25.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.563+0000 7faf9ffff640 1 -- 172.21.15.172:0/483556893 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fafb80727a0 con 0x7fafa406b3b0 2024-03-16T19:32:25.543 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 0 to deep-scrub 2024-03-16T19:32:25.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafa406b3b0 msgr2=0x7fafa406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafa406b3b0 0x7fafa406d870 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fafa802b9e0 tx=0x7fafa8045040 comp rx=0 tx=0).stop 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb8151910 msgr2=0x7fafb813d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb8151910 0x7fafb813d910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fafb000c730 tx=0x7fafb000cbf0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 shutdown_connections 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafa406b3b0 0x7fafa406d870 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafb8151910 0x7fafb813d910 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafb80c3ea0 0x7fafb8142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 --2- 172.21.15.172:0/483556893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafb80c3930 0x7fafb8162f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.566+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 >> 172.21.15.172:0/483556893 conn(0x7fafb806db70 msgr2=0x7fafb806aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.567+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 shutdown_connections 2024-03-16T19:32:25.547 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.567+0000 7fafbde64640 1 -- 172.21.15.172:0/483556893 wait complete. 2024-03-16T19:32:25.558 INFO:tasks.ceph:Scrubbing osd.1 2024-03-16T19:32:25.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-16T19:32:25.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.686+0000 7f3a52cfb640 1 Processor -- start 2024-03-16T19:32:25.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.686+0000 7f3a52cfb640 1 -- start start 2024-03-16T19:32:25.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a52cfb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3a4c0d0250 0x7f3a4c0d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a52cfb640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c0cb880 con 0x7f3a4c0ca480 2024-03-16T19:32:25.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a52cfb640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c1554b0 con 0x7f3a4c0b33f0 2024-03-16T19:32:25.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a52cfb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a4c0b5c20 con 0x7f3a4c0d0250 2024-03-16T19:32:25.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a50a97640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3a4c0ca480 0x7f3a4c15b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40048/0 (socket says 172.21.15.172:40048) 2024-03-16T19:32:25.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a50a97640 1 -- 172.21.15.172:0/3216204850 learned_addr learned my addr 172.21.15.172:0/3216204850 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:25.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a51298640 1 --2- 172.21.15.172:0/3216204850 >> v2:172.21.15.174:3300/0 conn(0x7f3a4c0d0250 0x7f3a4c0d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a4b7fe640 1 -- 172.21.15.172:0/3216204850 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347006145 0 0) 0x7f3a4c0cb880 con 0x7f3a4c0ca480 2024-03-16T19:32:25.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a4b7fe640 1 -- 172.21.15.172:0/3216204850 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a3c003520 con 0x7f3a4c0ca480 2024-03-16T19:32:25.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a4b7fe640 1 -- 172.21.15.172:0/3216204850 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596242898 0 0) 0x7f3a4c1554b0 con 0x7f3a4c0b33f0 2024-03-16T19:32:25.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a4b7fe640 1 -- 172.21.15.172:0/3216204850 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c0cb880 con 0x7f3a4c0b33f0 2024-03-16T19:32:25.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a51298640 1 -- 172.21.15.172:0/3216204850 >> v1:172.21.15.174:6789/0 conn(0x7f3a4c0b33f0 legacy=0x7f3a4c0b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a51298640 1 -- 172.21.15.172:0/3216204850 >> v1:172.21.15.172:6789/0 conn(0x7f3a4c0ca480 legacy=0x7f3a4c15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.687+0000 7f3a51298640 1 -- 172.21.15.172:0/3216204850 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c0cb300 con 0x7f3a4c0d0250 2024-03-16T19:32:25.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.688+0000 7f3a51298640 1 --2- 172.21.15.172:0/3216204850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a4c0d0250 0x7f3a4c0d0660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009d90 tx=0x7f3a4402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcefe3959ed0fe7a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.688+0000 7f3a4b7fe640 1 -- 172.21.15.172:0/3216204850 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4402ccf0 con 0x7f3a4c0d0250 2024-03-16T19:32:25.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.688+0000 7f3a4b7fe640 1 -- 172.21.15.172:0/3216204850 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4402ce50 con 0x7f3a4c0d0250 2024-03-16T19:32:25.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.688+0000 7f3a4b7fe640 1 -- 172.21.15.172:0/3216204850 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44034660 con 0x7f3a4c0d0250 2024-03-16T19:32:25.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.689+0000 7f3a52cfb640 1 -- 172.21.15.172:0/3216204850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a4c0d0250 msgr2=0x7f3a4c0d0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.689+0000 7f3a52cfb640 1 --2- 172.21.15.172:0/3216204850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a4c0d0250 0x7f3a4c0d0660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009d90 tx=0x7f3a4402bf40 comp rx=0 tx=0).stop 2024-03-16T19:32:25.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.689+0000 7f3a52cfb640 1 -- 172.21.15.172:0/3216204850 shutdown_connections 2024-03-16T19:32:25.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.689+0000 7f3a52cfb640 1 --2- 172.21.15.172:0/3216204850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a4c0d0250 0x7f3a4c0d0660 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.689+0000 7f3a52cfb640 1 -- 172.21.15.172:0/3216204850 >> 172.21.15.172:0/3216204850 conn(0x7f3a4c06db50 msgr2=0x7f3a4c06b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:25.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.689+0000 7f3a52cfb640 1 -- 172.21.15.172:0/3216204850 shutdown_connections 2024-03-16T19:32:25.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.689+0000 7f3a52cfb640 1 -- 172.21.15.172:0/3216204850 wait complete. 2024-03-16T19:32:25.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.690+0000 7f3a52cfb640 1 Processor -- start 2024-03-16T19:32:25.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.690+0000 7f3a52cfb640 1 -- start start 2024-03-16T19:32:25.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a52cfb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c0b33f0 0x7f3a4c1f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a52cfb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a4c0ca480 0x7f3a4c157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a52cfb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a4c0d0250 0x7f3a4c1f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a52cfb640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c0d0ba0 con 0x7f3a4c0b33f0 2024-03-16T19:32:25.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a52cfb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c1f0f40 con 0x7f3a4c0ca480 2024-03-16T19:32:25.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a52cfb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c158560 con 0x7f3a4c0d0250 2024-03-16T19:32:25.672 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a50a97640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c0b33f0 0x7f3a4c1f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a50a97640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c0b33f0 0x7f3a4c1f1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46752/0 (socket says 172.21.15.172:46752) 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a50a97640 1 -- 172.21.15.172:0/1326758279 learned_addr learned my addr 172.21.15.172:0/1326758279 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a51298640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a4c0d0250 0x7f3a4c1f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.691+0000 7f3a4bfff640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a4c0ca480 0x7f3a4c157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a51298640 1 -- 172.21.15.172:0/1326758279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a4c0ca480 msgr2=0x7f3a4c157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a51298640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a4c0ca480 0x7f3a4c157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a51298640 1 -- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c0b33f0 msgr2=0x7f3a4c1f1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a51298640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c0b33f0 0x7f3a4c1f1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a51298640 1 -- 172.21.15.172:0/1326758279 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a44009a70 con 0x7f3a4c0d0250 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a50a97640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c0b33f0 0x7f3a4c1f1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a51298640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a4c0d0250 0x7f3a4c1f0a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a44033000 tx=0x7f3a44004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a497fa640 1 -- 172.21.15.172:0/1326758279 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44003b90 con 0x7f3a4c0d0250 2024-03-16T19:32:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a497fa640 1 -- 172.21.15.172:0/1326758279 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a44003cf0 con 0x7f3a4c0d0250 2024-03-16T19:32:25.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c0af740 con 0x7f3a4c0d0250 2024-03-16T19:32:25.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a497fa640 1 -- 172.21.15.172:0/1326758279 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44003ea0 con 0x7f3a4c0d0250 2024-03-16T19:32:25.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.692+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c158280 con 0x7f3a4c0d0250 2024-03-16T19:32:25.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.693+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a4c0b4380 con 0x7f3a4c0d0250 2024-03-16T19:32:25.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.694+0000 7f3a497fa640 1 -- 172.21.15.172:0/1326758279 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3a4403b040 con 0x7f3a4c0d0250 2024-03-16T19:32:25.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.694+0000 7f3a497fa640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a3006b3b0 0x7f3a3006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.694+0000 7f3a50a97640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a3006b3b0 0x7f3a3006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.694+0000 7f3a497fa640 1 -- 172.21.15.172:0/1326758279 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f3a440abc50 con 0x7f3a4c0d0250 2024-03-16T19:32:25.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.694+0000 7f3a497fa640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3a30073730 0x7f3a30075bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.675 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.694+0000 7f3a497fa640 1 -- 172.21.15.172:0/1326758279 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a300762a0 con 0x7f3a30073730 2024-03-16T19:32:25.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.694+0000 7f3a497fa640 1 -- 172.21.15.172:0/1326758279 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a440ae310 con 0x7f3a4c0d0250 2024-03-16T19:32:25.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.695+0000 7f3a4bfff640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3a30073730 0x7f3a30075bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.695+0000 7f3a50a97640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a3006b3b0 0x7f3a3006d870 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3a40000f60 tx=0x7f3a40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.695+0000 7f3a4bfff640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3a30073730 0x7f3a30075bd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.676 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.696+0000 7f3a497fa640 1 -- 172.21.15.172:0/1326758279 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a300762a0 con 0x7f3a30073730 2024-03-16T19:32:25.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.713+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3a4c0b40a0 con 0x7f3a30073730 2024-03-16T19:32:25.693 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.713+0000 7f3a497fa640 1 -- 172.21.15.172:0/1326758279 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3a380014a0 con 0x7f3a30073730 2024-03-16T19:32:25.693 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:32:25.693 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T19:32:25.693 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:32:25.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3a30073730 msgr2=0x7f3a30075bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f3a30073730 0x7f3a30075bd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a3006b3b0 msgr2=0x7f3a3006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3a3006b3b0 0x7f3a3006d870 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3a40000f60 tx=0x7f3a40009040 comp rx=0 tx=0).stop 2024-03-16T19:32:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a4c0d0250 msgr2=0x7f3a4c1f0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 --2- 172.21.15.172:0/1326758279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a4c0d0250 0x7f3a4c1f0a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a44033000 tx=0x7f3a44004a40 comp rx=0 tx=0).stop 2024-03-16T19:32:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a51298640 1 -- 172.21.15.172:0/1326758279 reap_dead start 2024-03-16T19:32:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 shutdown_connections 2024-03-16T19:32:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 >> 172.21.15.172:0/1326758279 conn(0x7f3a4c06db50 msgr2=0x7f3a4c0b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 shutdown_connections 2024-03-16T19:32:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.714+0000 7f3a52cfb640 1 -- 172.21.15.172:0/1326758279 wait complete. 2024-03-16T19:32:25.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-16T19:32:25.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.865+0000 7f51a7156640 1 Processor -- start 2024-03-16T19:32:25.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.865+0000 7f51a7156640 1 -- start start 2024-03-16T19:32:25.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a7156640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f51a0151910 0x7f51a0072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.847 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a7156640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f51a00c3ea0 0x7f51a00c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a7156640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a00c3930 0x7f51a00ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a7156640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a00c2330 con 0x7f51a00c3ea0 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a7156640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a0072b10 con 0x7f51a0151910 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a7156640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a00c7da0 con 0x7f51a00c3930 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f519ffff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f51a00c3ea0 0x7f51a00c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a4ef2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f51a0151910 0x7f51a0072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a56f3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a00c3930 0x7f51a00ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a56f3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a00c3930 0x7f51a00ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44862/0 (socket says 172.21.15.172:44862) 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a4ef2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f51a0151910 0x7f51a0072fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:54346/0 (socket says 172.21.15.172:54346) 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.866+0000 7f51a56f3640 1 -- 172.21.15.172:0/4074671652 learned_addr learned my addr 172.21.15.172:0/4074671652 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.867+0000 7f51a56f3640 1 -- 172.21.15.172:0/4074671652 >> v2:172.21.15.174:3300/0 conn(0x7f51a0151910 msgr2=0x7f51a0072fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.867+0000 7f51a56f3640 1 --2- 172.21.15.172:0/4074671652 >> v2:172.21.15.174:3300/0 conn(0x7f51a0151910 0x7f51a0072fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.867+0000 7f51a56f3640 1 -- 172.21.15.172:0/4074671652 >> v2:172.21.15.172:3300/0 conn(0x7f51a00c3ea0 msgr2=0x7f51a00c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:25.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.867+0000 7f51a56f3640 1 --2- 172.21.15.172:0/4074671652 >> v2:172.21.15.172:3300/0 conn(0x7f51a00c3ea0 0x7f51a00c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.867+0000 7f51a56f3640 1 -- 172.21.15.172:0/4074671652 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a0071b10 con 0x7f51a00c3930 2024-03-16T19:32:25.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.868+0000 7f51a56f3640 1 --2- 172.21.15.172:0/4074671652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a00c3930 0x7f51a00ccf60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f51a00c9a30 tx=0x7f519802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6692ab0eb42a8a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.868+0000 7f519f7fe640 1 -- 172.21.15.172:0/4074671652 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519802ca80 con 0x7f51a00c3930 2024-03-16T19:32:25.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.868+0000 7f519f7fe640 1 -- 172.21.15.172:0/4074671652 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519802cbe0 con 0x7f51a00c3930 2024-03-16T19:32:25.849 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.868+0000 7f519f7fe640 1 -- 172.21.15.172:0/4074671652 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51980343f0 con 0x7f51a00c3930 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.869+0000 7f51a7156640 1 -- 172.21.15.172:0/4074671652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a00c3930 msgr2=0x7f51a00ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.869+0000 7f51a7156640 1 --2- 172.21.15.172:0/4074671652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a00c3930 0x7f51a00ccf60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f51a00c9a30 tx=0x7f519802bee0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.869+0000 7f51a7156640 1 -- 172.21.15.172:0/4074671652 shutdown_connections 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.869+0000 7f51a7156640 1 --2- 172.21.15.172:0/4074671652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a00c3930 0x7f51a00ccf60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.869+0000 7f51a7156640 1 --2- 172.21.15.172:0/4074671652 >> v2:172.21.15.172:3300/0 conn(0x7f51a00c3ea0 0x7f51a00c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.869+0000 7f51a7156640 1 --2- 172.21.15.172:0/4074671652 >> v2:172.21.15.174:3300/0 conn(0x7f51a0151910 0x7f51a0072fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.869+0000 7f51a7156640 1 -- 172.21.15.172:0/4074671652 >> 172.21.15.172:0/4074671652 conn(0x7f51a006db70 msgr2=0x7f51a006bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.869+0000 7f51a7156640 1 -- 172.21.15.172:0/4074671652 shutdown_connections 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.869+0000 7f51a7156640 1 -- 172.21.15.172:0/4074671652 wait complete. 2024-03-16T19:32:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.871+0000 7f51a7156640 1 Processor -- start 2024-03-16T19:32:25.852 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.871+0000 7f51a7156640 1 -- start start 2024-03-16T19:32:25.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a7156640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51a00c3930 0x7f51a01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.853 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a7156640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51a00c3ea0 0x7f51a01f0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a7156640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a0151910 0x7f51a00cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a7156640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a014b800 con 0x7f51a00c3930 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a7156640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a0158570 con 0x7f51a00c3ea0 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a7156640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a01f3fa0 con 0x7f51a0151910 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a56f3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a0151910 0x7f51a00cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a4ef2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51a00c3930 0x7f51a01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a4ef2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51a00c3930 0x7f51a01529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46776/0 (socket says 172.21.15.172:46776) 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f51a4ef2640 1 -- 172.21.15.172:0/3019815562 learned_addr learned my addr 172.21.15.172:0/3019815562 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.872+0000 7f519ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51a00c3ea0 0x7f51a01f0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.854 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f51a56f3640 1 -- 172.21.15.172:0/3019815562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51a00c3ea0 msgr2=0x7f51a01f0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f51a56f3640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51a00c3ea0 0x7f51a01f0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f51a56f3640 1 -- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51a00c3930 msgr2=0x7f51a01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f51a56f3640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51a00c3930 0x7f51a01529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f51a56f3640 1 -- 172.21.15.172:0/3019815562 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5198009b30 con 0x7f51a0151910 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f519ffff640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51a00c3ea0 0x7f51a01f0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f51a4ef2640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51a00c3930 0x7f51a01529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f51a56f3640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a0151910 0x7f51a00cc1c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5198032000 tx=0x7f5198004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f519d7fa640 1 -- 172.21.15.172:0/3019815562 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198009510 con 0x7f51a0151910 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f519d7fa640 1 -- 172.21.15.172:0/3019815562 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519803e450 con 0x7f51a0151910 2024-03-16T19:32:25.857 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f519d7fa640 1 -- 172.21.15.172:0/3019815562 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519803e800 con 0x7f51a0151910 2024-03-16T19:32:25.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.873+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a01f3990 con 0x7f51a0151910 2024-03-16T19:32:25.858 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.874+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a0157fb0 con 0x7f51a0151910 2024-03-16T19:32:25.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.875+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a01f37f0 con 0x7f51a0151910 2024-03-16T19:32:25.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.875+0000 7f519d7fa640 1 -- 172.21.15.172:0/3019815562 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f519802cd50 con 0x7f51a0151910 2024-03-16T19:32:25.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.875+0000 7f519d7fa640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f518406b400 0x7f518406d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:25.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.876+0000 7f519d7fa640 1 -- 172.21.15.172:0/3019815562 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f51980ac2c0 con 0x7f51a0151910 2024-03-16T19:32:25.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.876+0000 7f51a4ef2640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f518406b400 0x7f518406d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:25.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.877+0000 7f51a4ef2640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f518406b400 0x7f518406d8c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5194009f70 tx=0x7f5194009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:25.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:25.880+0000 7f519d7fa640 1 -- 172.21.15.172:0/3019815562 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51a01f37f0 con 0x7f51a0151910 2024-03-16T19:32:26.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.146+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f51a006cc00 con 0x7f518406b400 2024-03-16T19:32:26.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.146+0000 7f519d7fa640 1 -- 172.21.15.172:0/3019815562 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5194009ca0 con 0x7f518406b400 2024-03-16T19:32:26.126 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 1 to deep-scrub 2024-03-16T19:32:26.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f518406b400 msgr2=0x7f518406d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f518406b400 0x7f518406d8c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5194009f70 tx=0x7f5194009190 comp rx=0 tx=0).stop 2024-03-16T19:32:26.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a0151910 msgr2=0x7f51a00cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a0151910 0x7f51a00cc1c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5198032000 tx=0x7f5198004a40 comp rx=0 tx=0).stop 2024-03-16T19:32:26.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 shutdown_connections 2024-03-16T19:32:26.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f518406b400 0x7f518406d8c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51a0151910 0x7f51a00cc1c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51a00c3ea0 0x7f51a01f0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 --2- 172.21.15.172:0/3019815562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51a00c3930 0x7f51a01529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 >> 172.21.15.172:0/3019815562 conn(0x7f51a006db70 msgr2=0x7f51a00c9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:26.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 shutdown_connections 2024-03-16T19:32:26.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.149+0000 7f51a7156640 1 -- 172.21.15.172:0/3019815562 wait complete. 2024-03-16T19:32:26.142 INFO:tasks.ceph:Scrubbing osd.2 2024-03-16T19:32:26.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-16T19:32:26.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.281+0000 7efcf26c3640 1 Processor -- start 2024-03-16T19:32:26.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.281+0000 7efcf26c3640 1 -- start start 2024-03-16T19:32:26.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.282+0000 7efcf26c3640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7efcec154ee0 0x7efcec15b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.282+0000 7efcf26c3640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec15df30 con 0x7efcec0caed0 2024-03-16T19:32:26.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.282+0000 7efcf26c3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec06bfe0 con 0x7efcec154b00 2024-03-16T19:32:26.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.282+0000 7efcf26c3640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec15e180 con 0x7efcec154ee0 2024-03-16T19:32:26.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.282+0000 7efcf0c60640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7efcec0caed0 0x7efcec06c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40058/0 (socket says 172.21.15.172:40058) 2024-03-16T19:32:26.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.282+0000 7efcf0c60640 1 -- 172.21.15.172:0/609724633 learned_addr learned my addr 172.21.15.172:0/609724633 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:26.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.282+0000 7efcebfff640 1 --1- 172.21.15.172:0/609724633 >> v1:172.21.15.174:6789/0 conn(0x7efcec154b00 0x7efcec0710a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:37458/0 (socket says 172.21.15.172:37458) 2024-03-16T19:32:26.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.282+0000 7efceb7fe640 1 --2- 172.21.15.172:0/609724633 >> v2:172.21.15.172:3300/0 conn(0x7efcec154ee0 0x7efcec15b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581595424 0 0) 0x7efcec06bfe0 con 0x7efcec154b00 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd4003520 con 0x7efcec154b00 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105908959 0 0) 0x7efcec15df30 con 0x7efcec0caed0 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcec06bfe0 con 0x7efcec0caed0 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1150308205 0 0) 0x7efcd4003520 con 0x7efcec154b00 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 >> v2:172.21.15.172:3300/0 conn(0x7efcec154ee0 msgr2=0x7efcec15b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 --2- 172.21.15.172:0/609724633 >> v2:172.21.15.172:3300/0 conn(0x7efcec154ee0 0x7efcec15b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 >> v1:172.21.15.172:6789/0 conn(0x7efcec0caed0 legacy=0x7efcec06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec15e460 con 0x7efcec154b00 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7efcdc001670 con 0x7efcec154b00 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 >> v1:172.21.15.174:6789/0 conn(0x7efcec154b00 legacy=0x7efcec0710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcd4003b00 0x7efcd4003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcd40047c0 0x7efcd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd4005180 0x7efcd400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.266 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec15e180 con 0x7efcd4005180 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd400bfd0 con 0x7efcd40047c0 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd400c190 con 0x7efcd4003b00 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efcf0c60640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd4005180 0x7efcd400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efcebfff640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcd4003b00 0x7efcd4003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.283+0000 7efceb7fe640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcd40047c0 0x7efcd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efcf0c60640 1 -- 172.21.15.172:0/609724633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcd4003b00 msgr2=0x7efcd4003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efcf0c60640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcd4003b00 0x7efcd4003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efcf0c60640 1 -- 172.21.15.172:0/609724633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcd40047c0 msgr2=0x7efcd4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efcf0c60640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcd40047c0 0x7efcd4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efcf0c60640 1 -- 172.21.15.172:0/609724633 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd4003520 con 0x7efcd4005180 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efcebfff640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcd4003b00 0x7efcd4003ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efcf0c60640 1 -- 172.21.15.172:0/609724633 reap_dead start 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efcf0c60640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd4005180 0x7efcd400b980 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7efce400c320 tx=0x7efcd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5306ca69798bdac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.267 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce402cbf0 con 0x7efcd4005180 2024-03-16T19:32:26.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce402cd50 con 0x7efcd4005180 2024-03-16T19:32:26.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.284+0000 7efceaffd640 1 -- 172.21.15.172:0/609724633 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4035690 con 0x7efcd4005180 2024-03-16T19:32:26.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.285+0000 7efcf26c3640 1 -- 172.21.15.172:0/609724633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd4005180 msgr2=0x7efcd400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.285+0000 7efcf26c3640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd4005180 0x7efcd400b980 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7efce400c320 tx=0x7efcd40057f0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.285+0000 7efcf26c3640 1 -- 172.21.15.172:0/609724633 shutdown_connections 2024-03-16T19:32:26.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.285+0000 7efcf26c3640 1 --2- 172.21.15.172:0/609724633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcd4005180 0x7efcd400b980 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.285+0000 7efcf26c3640 1 -- 172.21.15.172:0/609724633 >> 172.21.15.172:0/609724633 conn(0x7efcec066c20 msgr2=0x7efcec067000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:26.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.285+0000 7efcf26c3640 1 -- 172.21.15.172:0/609724633 shutdown_connections 2024-03-16T19:32:26.268 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.285+0000 7efcf26c3640 1 -- 172.21.15.172:0/609724633 wait complete. 2024-03-16T19:32:26.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.286+0000 7efcf26c3640 1 Processor -- start 2024-03-16T19:32:26.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efcf26c3640 1 -- start start 2024-03-16T19:32:26.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efcf26c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcd4005180 0x7efcec15d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efcf26c3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcec158010 0x7efcec14fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efcf26c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcec0c67b0 0x7efcec150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efcf26c3640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0c8460 con 0x7efcec158010 2024-03-16T19:32:26.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efcf26c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0cf2a0 con 0x7efcec0c67b0 2024-03-16T19:32:26.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efcf26c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0cece0 con 0x7efcd4005180 2024-03-16T19:32:26.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efceb7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcec158010 0x7efcec14fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efceb7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcec158010 0x7efcec14fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46800/0 (socket says 172.21.15.172:46800) 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efceb7fe640 1 -- 172.21.15.172:0/4177594870 learned_addr learned my addr 172.21.15.172:0/4177594870 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efcf0c60640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcec0c67b0 0x7efcec150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.287+0000 7efcebfff640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcd4005180 0x7efcec15d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efceb7fe640 1 -- 172.21.15.172:0/4177594870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcd4005180 msgr2=0x7efcec15d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efceb7fe640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcd4005180 0x7efcec15d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efceb7fe640 1 -- 172.21.15.172:0/4177594870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcec0c67b0 msgr2=0x7efcec150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efceb7fe640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcec0c67b0 0x7efcec150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efceb7fe640 1 -- 172.21.15.172:0/4177594870 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0cefc0 con 0x7efcec158010 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efceb7fe640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcec158010 0x7efcec14fec0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7efce00118e0 tx=0x7efce0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efce8ff9640 1 -- 172.21.15.172:0/4177594870 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce000faf0 con 0x7efcec158010 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efce8ff9640 1 -- 172.21.15.172:0/4177594870 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce000fc50 con 0x7efcec158010 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec0c7e10 con 0x7efcec158010 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efce8ff9640 1 -- 172.21.15.172:0/4177594870 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0014440 con 0x7efcec158010 2024-03-16T19:32:26.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.288+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec15d640 con 0x7efcec158010 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.289+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efcec15ed00 con 0x7efcec158010 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.290+0000 7efce8ff9640 1 -- 172.21.15.172:0/4177594870 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7efce00145a0 con 0x7efcec158010 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.290+0000 7efce8ff9640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcd806b3b0 0x7efcd806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.290+0000 7efce8ff9640 1 -- 172.21.15.172:0/4177594870 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7efce008a470 con 0x7efcec158010 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.290+0000 7efcebfff640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcd806b3b0 0x7efcd806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.291+0000 7efce8ff9640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7efcd8073790 0x7efcd8075c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.291+0000 7efce8ff9640 1 -- 172.21.15.172:0/4177594870 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efcd8076300 con 0x7efcd8073790 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.291+0000 7efce8ff9640 1 -- 172.21.15.172:0/4177594870 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7efce008cb30 con 0x7efcec158010 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.291+0000 7efcf0c60640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7efcd8073790 0x7efcd8075c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.291+0000 7efcebfff640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcd806b3b0 0x7efcd806d870 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0017a0 tx=0x7efcdc001490 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.291+0000 7efcf0c60640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7efcd8073790 0x7efcd8075c30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.292+0000 7efce8ff9640 1 -- 172.21.15.172:0/4177594870 <== osd.2 v2:172.21.15.172:6823/3572014952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efcd8076300 con 0x7efcd8073790 2024-03-16T19:32:26.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.305+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efcec15ea20 con 0x7efcd8073790 2024-03-16T19:32:26.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efce8ff9640 1 -- 172.21.15.172:0/4177594870 <== osd.2 v2:172.21.15.172:6823/3572014952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+632 (crc 0 0 0) 0x7efce403d040 con 0x7efcd8073790 2024-03-16T19:32:26.285 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:32:26.286 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-16T19:32:26.286 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:32:26.286 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7efcd8073790 msgr2=0x7efcd8075c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.286 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7efcd8073790 0x7efcd8075c30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcd806b3b0 msgr2=0x7efcd806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7efcd806b3b0 0x7efcd806d870 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0017a0 tx=0x7efcdc001490 comp rx=0 tx=0).stop 2024-03-16T19:32:26.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcec158010 msgr2=0x7efcec14fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 --2- 172.21.15.172:0/4177594870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efcec158010 0x7efcec14fec0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7efce00118e0 tx=0x7efce0011da0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf0c60640 1 -- 172.21.15.172:0/4177594870 reap_dead start 2024-03-16T19:32:26.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 shutdown_connections 2024-03-16T19:32:26.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 >> 172.21.15.172:0/4177594870 conn(0x7efcec066c20 msgr2=0x7efcec0ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:26.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 shutdown_connections 2024-03-16T19:32:26.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.306+0000 7efcf26c3640 1 -- 172.21.15.172:0/4177594870 wait complete. 2024-03-16T19:32:26.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-16T19:32:26.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:32:26.378 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:32:26.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.438+0000 7fafc9437640 1 Processor -- start 2024-03-16T19:32:26.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.438+0000 7fafc9437640 1 -- start start 2024-03-16T19:32:26.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.439+0000 7fafc9437640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc4151b20 0x7fafc40c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc9437640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc40c6430 con 0x7fafc415acd0 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc9437640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc4151f00 con 0x7fafc40c3c20 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc9437640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40c4d10 con 0x7fafc4151b20 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc27fc640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fafc415acd0 0x7fafc4072c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40068/0 (socket says 172.21.15.172:40068) 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc27fc640 1 -- 172.21.15.172:0/285900736 learned_addr learned my addr 172.21.15.172:0/285900736 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc2ffd640 1 --2- 172.21.15.172:0/285900736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc4151b20 0x7fafc40c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc1ffb640 1 -- 172.21.15.172:0/285900736 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691925111 0 0) 0x7fafc4151f00 con 0x7fafc40c3c20 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc1ffb640 1 -- 172.21.15.172:0/285900736 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafac003520 con 0x7fafc40c3c20 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc1ffb640 1 -- 172.21.15.172:0/285900736 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112116607 0 0) 0x7fafc40c6430 con 0x7fafc415acd0 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc1ffb640 1 -- 172.21.15.172:0/285900736 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc4151f00 con 0x7fafc415acd0 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc2ffd640 1 -- 172.21.15.172:0/285900736 >> v1:172.21.15.174:6789/0 conn(0x7fafc40c3c20 legacy=0x7fafc40c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc2ffd640 1 -- 172.21.15.172:0/285900736 >> v1:172.21.15.172:6789/0 conn(0x7fafc415acd0 legacy=0x7fafc4072c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.440+0000 7fafc2ffd640 1 -- 172.21.15.172:0/285900736 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40c6c40 con 0x7fafc4151b20 2024-03-16T19:32:26.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.441+0000 7fafc2ffd640 1 --2- 172.21.15.172:0/285900736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc4151b20 0x7fafc40c91b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fafb40097d0 tx=0x7fafb402dbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c0fd40b5649281 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc1ffb640 1 -- 172.21.15.172:0/285900736 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb402e980 con 0x7fafc4151b20 2024-03-16T19:32:26.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc1ffb640 1 -- 172.21.15.172:0/285900736 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb402eae0 con 0x7fafc4151b20 2024-03-16T19:32:26.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc1ffb640 1 -- 172.21.15.172:0/285900736 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb402ee70 con 0x7fafc4151b20 2024-03-16T19:32:26.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc9437640 1 -- 172.21.15.172:0/285900736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc4151b20 msgr2=0x7fafc40c91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc9437640 1 --2- 172.21.15.172:0/285900736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc4151b20 0x7fafc40c91b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fafb40097d0 tx=0x7fafb402dbd0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.422 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc9437640 1 -- 172.21.15.172:0/285900736 shutdown_connections 2024-03-16T19:32:26.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc9437640 1 --2- 172.21.15.172:0/285900736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc4151b20 0x7fafc40c91b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc9437640 1 -- 172.21.15.172:0/285900736 >> 172.21.15.172:0/285900736 conn(0x7fafc406d510 msgr2=0x7fafc406aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:26.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc9437640 1 -- 172.21.15.172:0/285900736 shutdown_connections 2024-03-16T19:32:26.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.442+0000 7fafc9437640 1 -- 172.21.15.172:0/285900736 wait complete. 2024-03-16T19:32:26.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.444+0000 7fafc9437640 1 Processor -- start 2024-03-16T19:32:26.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.444+0000 7fafc9437640 1 -- start start 2024-03-16T19:32:26.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc9437640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc40c3c20 0x7fafc4150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc9437640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafc4151b20 0x7fafc41429b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc9437640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc415acd0 0x7fafc4142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc9437640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc4141630 con 0x7fafc415acd0 2024-03-16T19:32:26.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc9437640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc4141910 con 0x7fafc4151b20 2024-03-16T19:32:26.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc9437640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc4141bf0 con 0x7fafc40c3c20 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc415acd0 0x7fafc4142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc2ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc40c3c20 0x7fafc4150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc37fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc415acd0 0x7fafc4142ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46808/0 (socket says 172.21.15.172:46808) 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc2ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc40c3c20 0x7fafc4150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44914/0 (socket says 172.21.15.172:44914) 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc37fe640 1 -- 172.21.15.172:0/4167469552 learned_addr learned my addr 172.21.15.172:0/4167469552 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.445+0000 7fafc27fc640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafc4151b20 0x7fafc41429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafc37fe640 1 -- 172.21.15.172:0/4167469552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc40c3c20 msgr2=0x7fafc4150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafc37fe640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc40c3c20 0x7fafc4150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafc37fe640 1 -- 172.21.15.172:0/4167469552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafc4151b20 msgr2=0x7fafc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafc37fe640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafc4151b20 0x7fafc41429b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafc37fe640 1 -- 172.21.15.172:0/4167469552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb4009400 con 0x7fafc415acd0 2024-03-16T19:32:26.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafc37fe640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc415acd0 0x7fafc4142ef0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fafbc001460 tx=0x7fafbc00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafab7fe640 1 -- 172.21.15.172:0/4167469552 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc00cea0 con 0x7fafc415acd0 2024-03-16T19:32:26.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafab7fe640 1 -- 172.21.15.172:0/4167469552 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafbc003b70 con 0x7fafc415acd0 2024-03-16T19:32:26.427 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafab7fe640 1 -- 172.21.15.172:0/4167469552 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc00a850 con 0x7fafc415acd0 2024-03-16T19:32:26.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc4141ed0 con 0x7fafc415acd0 2024-03-16T19:32:26.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.446+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc4142490 con 0x7fafc415acd0 2024-03-16T19:32:26.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.447+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc41426b0 con 0x7fafc415acd0 2024-03-16T19:32:26.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.448+0000 7fafab7fe640 1 -- 172.21.15.172:0/4167469552 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fafbc00a9b0 con 0x7fafc415acd0 2024-03-16T19:32:26.428 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.449+0000 7fafab7fe640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafb006b3e0 0x7fafb006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.449+0000 7fafc2ffd640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafb006b3e0 0x7fafb006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.449+0000 7fafab7fe640 1 -- 172.21.15.172:0/4167469552 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fafbc089970 con 0x7fafc415acd0 2024-03-16T19:32:26.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.450+0000 7fafc2ffd640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafb006b3e0 0x7fafb006d8a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fafb4038000 tx=0x7fafb4032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.454+0000 7fafab7fe640 1 -- 172.21.15.172:0/4167469552 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafbc054670 con 0x7fafc415acd0 2024-03-16T19:32:26.694 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.715+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fafc40c6680 con 0x7fafb006b3e0 2024-03-16T19:32:26.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.715+0000 7fafab7fe640 1 -- 172.21.15.172:0/4167469552 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fafb4045090 con 0x7fafb006b3e0 2024-03-16T19:32:26.695 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 2 to deep-scrub 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafb006b3e0 msgr2=0x7fafb006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafb006b3e0 0x7fafb006d8a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fafb4038000 tx=0x7fafb4032000 comp rx=0 tx=0).stop 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc415acd0 msgr2=0x7fafc4142ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc415acd0 0x7fafc4142ef0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fafbc001460 tx=0x7fafbc00cc50 comp rx=0 tx=0).stop 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 shutdown_connections 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fafc415acd0 0x7fafc4142ef0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafc4151b20 0x7fafc41429b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fafb006b3e0 0x7fafb006d8a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 --2- 172.21.15.172:0/4167469552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fafc40c3c20 0x7fafc4150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 >> 172.21.15.172:0/4167469552 conn(0x7fafc406d510 msgr2=0x7fafc40c7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 shutdown_connections 2024-03-16T19:32:26.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.718+0000 7fafc9437640 1 -- 172.21.15.172:0/4167469552 wait complete. 2024-03-16T19:32:26.712 INFO:tasks.ceph:Scrubbing osd.3 2024-03-16T19:32:26.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-16T19:32:26.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.879+0000 7f3f8d7e4640 1 Processor -- start 2024-03-16T19:32:26.859 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.879+0000 7f3f8d7e4640 1 -- start start 2024-03-16T19:32:26.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.880+0000 7f3f8d7e4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 0x7f3f8806b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.880+0000 7f3f8d7e4640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3f880c6f40 0x7f3f8806bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.880+0000 7f3f8d7e4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8806b2e0 con 0x7f3f88154c10 2024-03-16T19:32:26.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.880+0000 7f3f8d7e4640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f880c8120 con 0x7f3f880c6f40 2024-03-16T19:32:26.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.880+0000 7f3f8d7e4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8806e960 con 0x7f3f8815b1e0 2024-03-16T19:32:26.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.880+0000 7f3f86ffd640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f3f88154c10 0x7f3f8815aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40076/0 (socket says 172.21.15.172:40076) 2024-03-16T19:32:26.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.880+0000 7f3f877fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f3f880c6f40 0x7f3f8806bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.880+0000 7f3f86ffd640 1 -- 172.21.15.172:0/3150668659 learned_addr learned my addr 172.21.15.172:0/3150668659 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:26.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.880+0000 7f3f867fc640 1 --2- 172.21.15.172:0/3150668659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 0x7f3f8806b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.881+0000 7f3f877fe640 1 -- 172.21.15.172:0/3150668659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 msgr2=0x7f3f8806b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:26.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.881+0000 7f3f877fe640 1 --2- 172.21.15.172:0/3150668659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 0x7f3f8806b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.881+0000 7f3f877fe640 1 -- 172.21.15.172:0/3150668659 >> v1:172.21.15.172:6789/0 conn(0x7f3f88154c10 legacy=0x7f3f8815aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.861 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.881+0000 7f3f877fe640 1 -- 172.21.15.172:0/3150668659 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f880c7b60 con 0x7f3f880c6f40 2024-03-16T19:32:26.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.882+0000 7f3f877fe640 1 --2- 172.21.15.172:0/3150668659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f880c6f40 0x7f3f8806bd20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3f80009d90 tx=0x7f3f8002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc70bcfa494df075 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.882+0000 7f3f85ffb640 1 -- 172.21.15.172:0/3150668659 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8002cce0 con 0x7f3f880c6f40 2024-03-16T19:32:26.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.882+0000 7f3f85ffb640 1 -- 172.21.15.172:0/3150668659 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8002ce40 con 0x7f3f880c6f40 2024-03-16T19:32:26.862 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.882+0000 7f3f85ffb640 1 -- 172.21.15.172:0/3150668659 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80034660 con 0x7f3f880c6f40 2024-03-16T19:32:26.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.883+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/3150668659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f880c6f40 msgr2=0x7f3f8806bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.883+0000 7f3f8d7e4640 1 --2- 172.21.15.172:0/3150668659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f880c6f40 0x7f3f8806bd20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3f80009d90 tx=0x7f3f8002bf30 comp rx=0 tx=0).stop 2024-03-16T19:32:26.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.883+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/3150668659 shutdown_connections 2024-03-16T19:32:26.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.883+0000 7f3f8d7e4640 1 --2- 172.21.15.172:0/3150668659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 0x7f3f8806b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.883+0000 7f3f8d7e4640 1 --2- 172.21.15.172:0/3150668659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f880c6f40 0x7f3f8806bd20 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.883+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/3150668659 >> 172.21.15.172:0/3150668659 conn(0x7f3f8806f7f0 msgr2=0x7f3f8806fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:26.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.883+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/3150668659 shutdown_connections 2024-03-16T19:32:26.863 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.883+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/3150668659 wait complete. 2024-03-16T19:32:26.864 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.885+0000 7f3f8d7e4640 1 Processor -- start 2024-03-16T19:32:26.865 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.885+0000 7f3f8d7e4640 1 -- start start 2024-03-16T19:32:26.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f8d7e4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f880c6f40 0x7f3f88148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f8d7e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f88154c10 0x7f3f881f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f8d7e4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 0x7f3f881f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f86ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f880c6f40 0x7f3f88148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f86ffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f880c6f40 0x7f3f88148f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46816/0 (socket says 172.21.15.172:46816) 2024-03-16T19:32:26.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f86ffd640 1 -- 172.21.15.172:0/4035998898 learned_addr learned my addr 172.21.15.172:0/4035998898 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:26.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f881483f0 con 0x7f3f880c6f40 2024-03-16T19:32:26.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f88155890 con 0x7f3f88154c10 2024-03-16T19:32:26.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f88155b70 con 0x7f3f8815b1e0 2024-03-16T19:32:26.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f877fe640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 0x7f3f881f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.867 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.886+0000 7f3f867fc640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f88154c10 0x7f3f881f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f877fe640 1 -- 172.21.15.172:0/4035998898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f88154c10 msgr2=0x7f3f881f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f877fe640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f88154c10 0x7f3f881f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f877fe640 1 -- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f880c6f40 msgr2=0x7f3f88148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f877fe640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f880c6f40 0x7f3f88148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f877fe640 1 -- 172.21.15.172:0/4035998898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f80009a70 con 0x7f3f8815b1e0 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f877fe640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 0x7f3f881f4390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3f880706b0 tx=0x7f3f80004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f6f7fe640 1 -- 172.21.15.172:0/4035998898 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8003d600 con 0x7f3f8815b1e0 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f6f7fe640 1 -- 172.21.15.172:0/4035998898 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8003d760 con 0x7f3f8815b1e0 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f88155e50 con 0x7f3f8815b1e0 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f6f7fe640 1 -- 172.21.15.172:0/4035998898 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8003db10 con 0x7f3f8815b1e0 2024-03-16T19:32:26.868 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.887+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f88156410 con 0x7f3f8815b1e0 2024-03-16T19:32:26.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.888+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f8815aeb0 con 0x7f3f8815b1e0 2024-03-16T19:32:26.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.889+0000 7f3f6f7fe640 1 -- 172.21.15.172:0/4035998898 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3f8003b040 con 0x7f3f8815b1e0 2024-03-16T19:32:26.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.890+0000 7f3f6f7fe640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3f7406b3b0 0x7f3f7406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.890+0000 7f3f6f7fe640 1 -- 172.21.15.172:0/4035998898 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f3f800abdd0 con 0x7f3f8815b1e0 2024-03-16T19:32:26.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.890+0000 7f3f86ffd640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3f7406b3b0 0x7f3f7406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.890+0000 7f3f6f7fe640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3f74073790 0x7f3f74075c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:26.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.890+0000 7f3f6f7fe640 1 -- 172.21.15.172:0/4035998898 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f74076300 con 0x7f3f74073790 2024-03-16T19:32:26.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.890+0000 7f3f6f7fe640 1 -- 172.21.15.172:0/4035998898 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f800ae490 con 0x7f3f8815b1e0 2024-03-16T19:32:26.870 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.890+0000 7f3f867fc640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3f74073790 0x7f3f74075c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:26.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.891+0000 7f3f86ffd640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3f7406b3b0 0x7f3f7406d870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3f78004e40 tx=0x7f3f78007f10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.891+0000 7f3f867fc640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3f74073790 0x7f3f74075c30 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:26.871 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.891+0000 7f3f6f7fe640 1 -- 172.21.15.172:0/4035998898 <== osd.3 v2:172.21.15.172:6816/1613555619 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3f74076300 con 0x7f3f74073790 2024-03-16T19:32:26.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.917+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 --> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3f880c7590 con 0x7f3f74073790 2024-03-16T19:32:26.897 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.917+0000 7f3f6f7fe640 1 -- 172.21.15.172:0/4035998898 <== osd.3 v2:172.21.15.172:6816/1613555619 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3f7c009c20 con 0x7f3f74073790 2024-03-16T19:32:26.897 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:32:26.897 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T19:32:26.897 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:32:26.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3f74073790 msgr2=0x7f3f74075c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6816/1613555619,v1:172.21.15.172:6817/1613555619] conn(0x7f3f74073790 0x7f3f74075c30 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:26.898 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3f7406b3b0 msgr2=0x7f3f7406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f3f7406b3b0 0x7f3f7406d870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3f78004e40 tx=0x7f3f78007f10 comp rx=0 tx=0).stop 2024-03-16T19:32:26.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 msgr2=0x7f3f881f4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:26.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 --2- 172.21.15.172:0/4035998898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f8815b1e0 0x7f3f881f4390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3f880706b0 tx=0x7f3f80004a40 comp rx=0 tx=0).stop 2024-03-16T19:32:26.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f877fe640 1 -- 172.21.15.172:0/4035998898 reap_dead start 2024-03-16T19:32:26.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 shutdown_connections 2024-03-16T19:32:26.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 >> 172.21.15.172:0/4035998898 conn(0x7f3f8806f7f0 msgr2=0x7f3f8814fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:26.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 shutdown_connections 2024-03-16T19:32:26.899 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:26.918+0000 7f3f8d7e4640 1 -- 172.21.15.172:0/4035998898 wait complete. 2024-03-16T19:32:26.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-16T19:32:27.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.034+0000 7fdcb3bd0640 1 Processor -- start 2024-03-16T19:32:27.014 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.034+0000 7fdcb3bd0640 1 -- start start 2024-03-16T19:32:27.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.035+0000 7fdcb3bd0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 0x7fdcac06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.035+0000 7fdcb3bd0640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdcac0c79d0 0x7fdcac06b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.035+0000 7fdcb3bd0640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac072610 con 0x7fdcac0c93a0 2024-03-16T19:32:27.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.035+0000 7fdcb3bd0640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcac0c9890 con 0x7fdcac0c79d0 2024-03-16T19:32:27.015 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.035+0000 7fdcb3bd0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac0729c0 con 0x7fdcac152a20 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.035+0000 7fdcb216d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdcac0c79d0 0x7fdcac06b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.035+0000 7fdcb216d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fdcac0c79d0 0x7fdcac06b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46824/0 (socket says 172.21.15.172:46824) 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.035+0000 7fdcb216d640 1 -- 172.21.15.172:0/1589387690 learned_addr learned my addr 172.21.15.172:0/1589387690 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.035+0000 7fdcb196c640 1 --2- 172.21.15.172:0/1589387690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 0x7fdcac06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.036+0000 7fdcb216d640 1 -- 172.21.15.172:0/1589387690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 msgr2=0x7fdcac06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.036+0000 7fdcb216d640 1 --2- 172.21.15.172:0/1589387690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 0x7fdcac06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.036+0000 7fdcb216d640 1 -- 172.21.15.172:0/1589387690 >> v1:172.21.15.172:6789/0 conn(0x7fdcac0c93a0 legacy=0x7fdcac0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.036+0000 7fdcb216d640 1 -- 172.21.15.172:0/1589387690 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac072ca0 con 0x7fdcac0c79d0 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.036+0000 7fdcb096a640 1 -- 172.21.15.172:0/1589387690 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567066938 0 0) 0x7fdcac072610 con 0x7fdcac0c93a0 2024-03-16T19:32:27.016 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.036+0000 7fdcb196c640 1 --2- 172.21.15.172:0/1589387690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 0x7fdcac06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:27.017 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.036+0000 7fdcb216d640 1 --2- 172.21.15.172:0/1589387690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c79d0 0x7fdcac06b2b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009e50 tx=0x7fdca402bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03c4babac2a5a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.017 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb096a640 1 -- 172.21.15.172:0/1589387690 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca402cb80 con 0x7fdcac0c79d0 2024-03-16T19:32:27.017 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb096a640 1 -- 172.21.15.172:0/1589387690 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca402cce0 con 0x7fdcac0c79d0 2024-03-16T19:32:27.017 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb096a640 1 -- 172.21.15.172:0/1589387690 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca4034610 con 0x7fdcac0c79d0 2024-03-16T19:32:27.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1589387690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c79d0 msgr2=0x7fdcac06b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb3bd0640 1 --2- 172.21.15.172:0/1589387690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c79d0 0x7fdcac06b2b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009e50 tx=0x7fdca402bfe0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1589387690 shutdown_connections 2024-03-16T19:32:27.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb3bd0640 1 --2- 172.21.15.172:0/1589387690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c79d0 0x7fdcac06b2b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb3bd0640 1 --2- 172.21.15.172:0/1589387690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 0x7fdcac06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1589387690 >> 172.21.15.172:0/1589387690 conn(0x7fdcac06db70 msgr2=0x7fdcac06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:27.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1589387690 shutdown_connections 2024-03-16T19:32:27.018 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.037+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1589387690 wait complete. 2024-03-16T19:32:27.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.039+0000 7fdcb3bd0640 1 Processor -- start 2024-03-16T19:32:27.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.039+0000 7fdcb3bd0640 1 -- start start 2024-03-16T19:32:27.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.039+0000 7fdcb3bd0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcac0c79d0 0x7fdcac0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.039+0000 7fdcb3bd0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c93a0 0x7fdcac0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.039+0000 7fdcb3bd0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 0x7fdcac14f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.039+0000 7fdcb3bd0640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac1eece0 con 0x7fdcac0c93a0 2024-03-16T19:32:27.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.039+0000 7fdcb3bd0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac14c870 con 0x7fdcac0c79d0 2024-03-16T19:32:27.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.039+0000 7fdcb3bd0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac14ee80 con 0x7fdcac152a20 2024-03-16T19:32:27.020 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.039+0000 7fdcb116b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c93a0 0x7fdcac0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb116b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c93a0 0x7fdcac0cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46838/0 (socket says 172.21.15.172:46838) 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb116b640 1 -- 172.21.15.172:0/1689426801 learned_addr learned my addr 172.21.15.172:0/1689426801 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb216d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 0x7fdcac14f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb196c640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcac0c79d0 0x7fdcac0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb116b640 1 -- 172.21.15.172:0/1689426801 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 msgr2=0x7fdcac14f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb116b640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 0x7fdcac14f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb116b640 1 -- 172.21.15.172:0/1689426801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcac0c79d0 msgr2=0x7fdcac0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb116b640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcac0c79d0 0x7fdcac0cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb116b640 1 -- 172.21.15.172:0/1689426801 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca8009400 con 0x7fdcac0c93a0 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb116b640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c93a0 0x7fdcac0cc730 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00c460 tx=0x7fdc9c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdca27fc640 1 -- 172.21.15.172:0/1689426801 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c005480 con 0x7fdcac0c93a0 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdca27fc640 1 -- 172.21.15.172:0/1689426801 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c0035f0 con 0x7fdcac0c93a0 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca4009b30 con 0x7fdcac0c93a0 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdca27fc640 1 -- 172.21.15.172:0/1689426801 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c0039a0 con 0x7fdcac0c93a0 2024-03-16T19:32:27.021 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.040+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcac152f80 con 0x7fdcac0c93a0 2024-03-16T19:32:27.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.041+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcac1ee200 con 0x7fdcac0c93a0 2024-03-16T19:32:27.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.042+0000 7fdca27fc640 1 -- 172.21.15.172:0/1689426801 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fdc9c003b00 con 0x7fdcac0c93a0 2024-03-16T19:32:27.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.042+0000 7fdca27fc640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdc9006b3b0 0x7fdc9006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.022 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.042+0000 7fdca27fc640 1 -- 172.21.15.172:0/1689426801 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fdc9c089710 con 0x7fdcac0c93a0 2024-03-16T19:32:27.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.042+0000 7fdcb196c640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdc9006b3b0 0x7fdc9006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.023 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.043+0000 7fdcb196c640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdc9006b3b0 0x7fdc9006d870 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009c30 tx=0x7fdca8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.049+0000 7fdca27fc640 1 -- 172.21.15.172:0/1689426801 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc9c0542f0 con 0x7fdcac0c93a0 2024-03-16T19:32:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.352+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcac152e00 con 0x7fdc9006b3b0 2024-03-16T19:32:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.352+0000 7fdca27fc640 1 -- 172.21.15.172:0/1689426801 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdca8009880 con 0x7fdc9006b3b0 2024-03-16T19:32:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 3 to deep-scrub 2024-03-16T19:32:27.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdc9006b3b0 msgr2=0x7fdc9006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdc9006b3b0 0x7fdc9006d870 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009c30 tx=0x7fdca8009190 comp rx=0 tx=0).stop 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c93a0 msgr2=0x7fdcac0cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c93a0 0x7fdcac0cc730 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00c460 tx=0x7fdc9c00c920 comp rx=0 tx=0).stop 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 shutdown_connections 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fdc9006b3b0 0x7fdc9006d870 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcac152a20 0x7fdcac14f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdcac0c93a0 0x7fdcac0cc730 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 --2- 172.21.15.172:0/1689426801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcac0c79d0 0x7fdcac0cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 >> 172.21.15.172:0/1689426801 conn(0x7fdcac06db70 msgr2=0x7fdcac1511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.355+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 shutdown_connections 2024-03-16T19:32:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.356+0000 7fdcb3bd0640 1 -- 172.21.15.172:0/1689426801 wait complete. 2024-03-16T19:32:27.347 INFO:tasks.ceph:Scrubbing osd.4 2024-03-16T19:32:27.348 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-16T19:32:27.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.503+0000 7fe0b4c72640 1 Processor -- start 2024-03-16T19:32:27.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.503+0000 7fe0b4c72640 1 -- start start 2024-03-16T19:32:27.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0b4c72640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe0b006e1a0 0x7fe0b015b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0b4c72640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006ec90 0x7fe0b0151590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0b4c72640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b00c4660 con 0x7fe0b015b980 2024-03-16T19:32:27.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0b4c72640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b015acd0 con 0x7fe0b006e1a0 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0b4c72640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b015afb0 con 0x7fe0b006ec90 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0add9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006ec90 0x7fe0b0151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0ae59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe0b006e1a0 0x7fe0b015b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0add9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006ec90 0x7fe0b0151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:44966/0 (socket says 172.21.15.172:44966) 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0ae59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe0b006e1a0 0x7fe0b015b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:54404/0 (socket says 172.21.15.172:54404) 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0add9b640 1 -- 172.21.15.172:0/1840600790 learned_addr learned my addr 172.21.15.172:0/1840600790 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0ae59c640 1 -- 172.21.15.172:0/1840600790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006ec90 msgr2=0x7fe0b0151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0ae59c640 1 --2- 172.21.15.172:0/1840600790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006ec90 0x7fe0b0151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0ae59c640 1 -- 172.21.15.172:0/1840600790 >> v1:172.21.15.174:6789/0 conn(0x7fe0b015b980 legacy=0x7fe0b006bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0ae59c640 1 -- 172.21.15.172:0/1840600790 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00c4b30 con 0x7fe0b006e1a0 2024-03-16T19:32:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.504+0000 7fe0add9b640 1 --2- 172.21.15.172:0/1840600790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006ec90 0x7fe0b0151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:27.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.505+0000 7fe0ae59c640 1 --2- 172.21.15.172:0/1840600790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006e1a0 0x7fe0b015b440 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe0a00097a0 tx=0x7fe0a002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1622192dd8be29a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.505+0000 7fe0ad59a640 1 -- 172.21.15.172:0/1840600790 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a002f050 con 0x7fe0b006e1a0 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.506+0000 7fe0ad59a640 1 -- 172.21.15.172:0/1840600790 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a0031700 con 0x7fe0b006e1a0 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.506+0000 7fe0ad59a640 1 -- 172.21.15.172:0/1840600790 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a0031a90 con 0x7fe0b006e1a0 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.506+0000 7fe0b4c72640 1 -- 172.21.15.172:0/1840600790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006e1a0 msgr2=0x7fe0b015b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.506+0000 7fe0b4c72640 1 --2- 172.21.15.172:0/1840600790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006e1a0 0x7fe0b015b440 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe0a00097a0 tx=0x7fe0a002b7a0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.506+0000 7fe0b4c72640 1 -- 172.21.15.172:0/1840600790 shutdown_connections 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.506+0000 7fe0b4c72640 1 --2- 172.21.15.172:0/1840600790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006ec90 0x7fe0b0151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.506+0000 7fe0b4c72640 1 --2- 172.21.15.172:0/1840600790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006e1a0 0x7fe0b015b440 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.506+0000 7fe0b4c72640 1 -- 172.21.15.172:0/1840600790 >> 172.21.15.172:0/1840600790 conn(0x7fe0b006f7f0 msgr2=0x7fe0b006fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.506+0000 7fe0b4c72640 1 -- 172.21.15.172:0/1840600790 shutdown_connections 2024-03-16T19:32:27.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.507+0000 7fe0b4c72640 1 -- 172.21.15.172:0/1840600790 wait complete. 2024-03-16T19:32:27.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.508+0000 7fe0b4c72640 1 Processor -- start 2024-03-16T19:32:27.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.509+0000 7fe0b4c72640 1 -- start start 2024-03-16T19:32:27.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0b4c72640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006e1a0 0x7fe0b014e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0b4c72640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006ec90 0x7fe0b015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.491 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0b4c72640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0b015b980 0x7fe0b006bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0ae59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006e1a0 0x7fe0b014e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0b4c72640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b006b980 con 0x7fe0b015b980 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0b4c72640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b014dbe0 con 0x7fe0b006ec90 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0b4c72640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b014de80 con 0x7fe0b006e1a0 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0add9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006ec90 0x7fe0b015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0add9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006ec90 0x7fe0b015a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:54410/0 (socket says 172.21.15.172:54410) 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0add9b640 1 -- 172.21.15.172:0/3053183 learned_addr learned my addr 172.21.15.172:0/3053183 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0aed9d640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0b015b980 0x7fe0b006bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.510+0000 7fe0add9b640 1 -- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006e1a0 msgr2=0x7fe0b014e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe0add9b640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006e1a0 0x7fe0b014e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe0add9b640 1 -- 172.21.15.172:0/3053183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0b015b980 msgr2=0x7fe0b006bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe0add9b640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0b015b980 0x7fe0b006bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe0add9b640 1 -- 172.21.15.172:0/3053183 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a400b770 con 0x7fe0b006ec90 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe0ae59c640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0b006e1a0 0x7fe0b014e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T19:32:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe0add9b640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006ec90 0x7fe0b015a260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4010df0 tx=0x7fe0a400e730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe09effd640 1 -- 172.21.15.172:0/3053183 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4011860 con 0x7fe0b006ec90 2024-03-16T19:32:27.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe09effd640 1 -- 172.21.15.172:0/3053183 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a40119c0 con 0x7fe0b006ec90 2024-03-16T19:32:27.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe09effd640 1 -- 172.21.15.172:0/3053183 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4011d30 con 0x7fe0b006ec90 2024-03-16T19:32:27.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a0009400 con 0x7fe0b006ec90 2024-03-16T19:32:27.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.511+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b014ee40 con 0x7fe0b006ec90 2024-03-16T19:32:27.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.512+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe0b00c7fe0 con 0x7fe0b006ec90 2024-03-16T19:32:27.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.513+0000 7fe09effd640 1 -- 172.21.15.172:0/3053183 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fe0a4019360 con 0x7fe0b006ec90 2024-03-16T19:32:27.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.514+0000 7fe09effd640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe09006b3b0 0x7fe09006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.514+0000 7fe09effd640 1 -- 172.21.15.172:0/3053183 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fe0a408a260 con 0x7fe0b006ec90 2024-03-16T19:32:27.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.514+0000 7fe09effd640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fe090073730 0x7fe090075bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.514+0000 7fe09effd640 1 -- 172.21.15.172:0/3053183 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0900762a0 con 0x7fe090073730 2024-03-16T19:32:27.494 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.514+0000 7fe09effd640 1 -- 172.21.15.172:0/3053183 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0a4054cf0 con 0x7fe0b006ec90 2024-03-16T19:32:27.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.514+0000 7fe0ae59c640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe09006b3b0 0x7fe09006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.514+0000 7fe0aed9d640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fe090073730 0x7fe090075bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.515+0000 7fe0aed9d640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fe090073730 0x7fe090075bd0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.515+0000 7fe0ae59c640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe09006b3b0 0x7fe09006d870 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0035000 tx=0x7fe0a0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.515+0000 7fe09effd640 1 -- 172.21.15.172:0/3053183 <== osd.4 v2:172.21.15.174:6808/967866068 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe0900762a0 con 0x7fe090073730 2024-03-16T19:32:27.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 --> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe0b00c4890 con 0x7fe090073730 2024-03-16T19:32:27.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe09effd640 1 -- 172.21.15.172:0/3053183 <== osd.4 v2:172.21.15.174:6808/967866068 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe0a8014040 con 0x7fe090073730 2024-03-16T19:32:27.508 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:32:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T19:32:27.509 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:32:27.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fe090073730 msgr2=0x7fe090075bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0b4c72640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:6808/967866068,v1:172.21.15.174:6809/967866068] conn(0x7fe090073730 0x7fe090075bd0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe09006b3b0 msgr2=0x7fe09006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0b4c72640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe09006b3b0 0x7fe09006d870 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0035000 tx=0x7fe0a0008030 comp rx=0 tx=0).stop 2024-03-16T19:32:27.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006ec90 msgr2=0x7fe0b015a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0b4c72640 1 --2- 172.21.15.172:0/3053183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b006ec90 0x7fe0b015a260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4010df0 tx=0x7fe0a400e730 comp rx=0 tx=0).stop 2024-03-16T19:32:27.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0aed9d640 1 -- 172.21.15.172:0/3053183 reap_dead start 2024-03-16T19:32:27.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 shutdown_connections 2024-03-16T19:32:27.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.529+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 >> 172.21.15.172:0/3053183 conn(0x7fe0b006f7f0 msgr2=0x7fe0b014f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:27.511 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.530+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 shutdown_connections 2024-03-16T19:32:27.511 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.530+0000 7fe0b4c72640 1 -- 172.21.15.172:0/3053183 wait complete. 2024-03-16T19:32:27.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-16T19:32:27.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.655+0000 7f5411cc3640 1 Processor -- start 2024-03-16T19:32:27.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.655+0000 7f5411cc3640 1 -- start start 2024-03-16T19:32:27.636 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.656+0000 7f5411cc3640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f540c152a20 0x7f540c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.636 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.656+0000 7f5411cc3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0ce4e0 0x7f540c0ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.636 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.656+0000 7f5411cc3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c0c9cf0 con 0x7f540c0b0de0 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.656+0000 7f5411cc3640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540c0c9f90 con 0x7f540c152a20 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.656+0000 7f5411cc3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540c0ca250 con 0x7f540c0ce4e0 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.656+0000 7f540b7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f540c152a20 0x7f540c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.656+0000 7f540affd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f540c0b0de0 0x7f540c15c560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:37490/0 (socket says 172.21.15.172:37490) 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.656+0000 7f540affd640 1 -- 172.21.15.172:0/396168639 learned_addr learned my addr 172.21.15.172:0/396168639 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.656+0000 7f540bfff640 1 --2- 172.21.15.172:0/396168639 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0ce4e0 0x7f540c0ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.657+0000 7f540b7fe640 1 -- 172.21.15.172:0/396168639 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0ce4e0 msgr2=0x7f540c0ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.657+0000 7f540a7fc640 1 -- 172.21.15.172:0/396168639 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959624576 0 0) 0x7f540c0c9cf0 con 0x7f540c0b0de0 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.657+0000 7f540b7fe640 1 --2- 172.21.15.172:0/396168639 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0ce4e0 0x7f540c0ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.657+0000 7f540b7fe640 1 -- 172.21.15.172:0/396168639 >> v1:172.21.15.174:6789/0 conn(0x7f540c0b0de0 legacy=0x7f540c15c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.657+0000 7f540b7fe640 1 -- 172.21.15.172:0/396168639 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c0ca530 con 0x7f540c152a20 2024-03-16T19:32:27.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.657+0000 7f540b7fe640 1 --2- 172.21.15.172:0/396168639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 0x7f540c06f070 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0097a0 tx=0x7f53fc02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13554adbbbf68d14 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f540a7fc640 1 -- 172.21.15.172:0/396168639 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc030800 con 0x7f540c152a20 2024-03-16T19:32:27.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f540a7fc640 1 -- 172.21.15.172:0/396168639 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc030960 con 0x7f540c152a20 2024-03-16T19:32:27.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f540a7fc640 1 -- 172.21.15.172:0/396168639 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc030cf0 con 0x7f540c152a20 2024-03-16T19:32:27.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f5411cc3640 1 -- 172.21.15.172:0/396168639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 msgr2=0x7f540c06f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f5411cc3640 1 --2- 172.21.15.172:0/396168639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 0x7f540c06f070 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0097a0 tx=0x7f53fc02b940 comp rx=0 tx=0).stop 2024-03-16T19:32:27.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f5411cc3640 1 -- 172.21.15.172:0/396168639 shutdown_connections 2024-03-16T19:32:27.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f5411cc3640 1 --2- 172.21.15.172:0/396168639 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0ce4e0 0x7f540c0ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f5411cc3640 1 --2- 172.21.15.172:0/396168639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 0x7f540c06f070 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f5411cc3640 1 -- 172.21.15.172:0/396168639 >> 172.21.15.172:0/396168639 conn(0x7f540c066e00 msgr2=0x7f540c0671e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:27.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f5411cc3640 1 -- 172.21.15.172:0/396168639 shutdown_connections 2024-03-16T19:32:27.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.658+0000 7f5411cc3640 1 -- 172.21.15.172:0/396168639 wait complete. 2024-03-16T19:32:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.659+0000 7f5411cc3640 1 Processor -- start 2024-03-16T19:32:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.660+0000 7f5411cc3640 1 -- start start 2024-03-16T19:32:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.660+0000 7f5411cc3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0b0de0 0x7f540c157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.660+0000 7f5411cc3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f540c0ce4e0 0x7f540c14ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.660+0000 7f5411cc3640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 0x7f540c14f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.660+0000 7f5411cc3640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c1f31c0 con 0x7f540c152a20 2024-03-16T19:32:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.660+0000 7f5411cc3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c158690 con 0x7f540c0ce4e0 2024-03-16T19:32:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.660+0000 7f5411cc3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540c1f3430 con 0x7f540c0b0de0 2024-03-16T19:32:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0b0de0 0x7f540c157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 0x7f540c14f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f540c0ce4e0 0x7f540c14ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0b0de0 0x7f540c157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:45004/0 (socket says 172.21.15.172:45004) 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540b7fe640 1 -- 172.21.15.172:0/772143826 learned_addr learned my addr 172.21.15.172:0/772143826 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540bfff640 1 -- 172.21.15.172:0/772143826 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0b0de0 msgr2=0x7f540c157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540bfff640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0b0de0 0x7f540c157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540bfff640 1 -- 172.21.15.172:0/772143826 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f540c0ce4e0 msgr2=0x7f540c14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540bfff640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f540c0ce4e0 0x7f540c14ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540bfff640 1 -- 172.21.15.172:0/772143826 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc009400 con 0x7f540c152a20 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540affd640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f540c0ce4e0 0x7f540c14ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f540bfff640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 0x7f540c14f490 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f540400efc0 tx=0x7f540400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f53f3fff640 1 -- 172.21.15.172:0/772143826 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404009c00 con 0x7f540c152a20 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f53f3fff640 1 -- 172.21.15.172:0/772143826 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540400f040 con 0x7f540c152a20 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540c1f3710 con 0x7f540c152a20 2024-03-16T19:32:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f53f3fff640 1 -- 172.21.15.172:0/772143826 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54040146c0 con 0x7f540c152a20 2024-03-16T19:32:27.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.661+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540c1f3cd0 con 0x7f540c152a20 2024-03-16T19:32:27.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.662+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f540c1f3ef0 con 0x7f540c152a20 2024-03-16T19:32:27.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.663+0000 7f53f3fff640 1 -- 172.21.15.172:0/772143826 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5404014820 con 0x7f540c152a20 2024-03-16T19:32:27.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.663+0000 7f53f3fff640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f53f806b3b0 0x7f53f806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:27.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.663+0000 7f53f3fff640 1 -- 172.21.15.172:0/772143826 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f540408ab60 con 0x7f540c152a20 2024-03-16T19:32:27.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.664+0000 7f540b7fe640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f53f806b3b0 0x7f53f806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:27.644 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.664+0000 7f540b7fe640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f53f806b3b0 0x7f53f806d870 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f53fc02b980 tx=0x7f53fc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:27.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.668+0000 7f53f3fff640 1 -- 172.21.15.172:0/772143826 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54040556a0 con 0x7f540c152a20 2024-03-16T19:32:27.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.920+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f540c0c8be0 con 0x7f53f806b3b0 2024-03-16T19:32:27.902 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.921+0000 7f53f3fff640 1 -- 172.21.15.172:0/772143826 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f53fc03f040 con 0x7f53f806b3b0 2024-03-16T19:32:27.902 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 4 to deep-scrub 2024-03-16T19:32:27.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.923+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f53f806b3b0 msgr2=0x7f53f806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.923+0000 7f5411cc3640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f53f806b3b0 0x7f53f806d870 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f53fc02b980 tx=0x7f53fc03b040 comp rx=0 tx=0).stop 2024-03-16T19:32:27.903 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 msgr2=0x7f540c14f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:27.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 0x7f540c14f490 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f540400efc0 tx=0x7f540400c490 comp rx=0 tx=0).stop 2024-03-16T19:32:27.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 shutdown_connections 2024-03-16T19:32:27.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f540c152a20 0x7f540c14f490 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f540c0ce4e0 0x7f540c14ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f53f806b3b0 0x7f53f806d870 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 --2- 172.21.15.172:0/772143826 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f540c0b0de0 0x7f540c157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:27.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 >> 172.21.15.172:0/772143826 conn(0x7f540c066e00 msgr2=0x7f540c14ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:27.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 shutdown_connections 2024-03-16T19:32:27.905 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:27.924+0000 7f5411cc3640 1 -- 172.21.15.172:0/772143826 wait complete. 2024-03-16T19:32:27.918 INFO:tasks.ceph:Scrubbing osd.5 2024-03-16T19:32:27.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-16T19:32:28.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.081+0000 7f9338e4d640 1 Processor -- start 2024-03-16T19:32:28.061 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.081+0000 7f9338e4d640 1 -- start start 2024-03-16T19:32:28.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.082+0000 7f9338e4d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9334151a90 0x7f93340721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.082+0000 7f9338e4d640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93340c8850 con 0x7f93340c3b20 2024-03-16T19:32:28.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.082+0000 7f9338e4d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93340c64c0 con 0x7f93340c90b0 2024-03-16T19:32:28.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.082+0000 7f9338e4d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9334151e70 con 0x7f9334151a90 2024-03-16T19:32:28.062 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.083+0000 7f932bfff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f93340c3b20 0x7f93340712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40094/0 (socket says 172.21.15.172:40094) 2024-03-16T19:32:28.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.083+0000 7f932bfff640 1 -- 172.21.15.172:0/1147078043 learned_addr learned my addr 172.21.15.172:0/1147078043 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:28.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.083+0000 7f933259c640 1 --2- 172.21.15.172:0/1147078043 >> v2:172.21.15.174:3300/0 conn(0x7f9334151a90 0x7f93340721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.083+0000 7f9331d9b640 1 -- 172.21.15.172:0/1147078043 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605532666 0 0) 0x7f93340c8850 con 0x7f93340c3b20 2024-03-16T19:32:28.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.083+0000 7f933259c640 1 -- 172.21.15.172:0/1147078043 >> v1:172.21.15.174:6789/0 conn(0x7f93340c90b0 legacy=0x7f9334161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.083+0000 7f933259c640 1 -- 172.21.15.172:0/1147078043 >> v1:172.21.15.172:6789/0 conn(0x7f93340c3b20 legacy=0x7f93340712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.063 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.083+0000 7f933259c640 1 -- 172.21.15.172:0/1147078043 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340c6760 con 0x7f9334151a90 2024-03-16T19:32:28.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.084+0000 7f933259c640 1 --2- 172.21.15.172:0/1147078043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9334151a90 0x7f93340721d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9324009720 tx=0x7f932402b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adba020ae6d2f792 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.084+0000 7f9331d9b640 1 -- 172.21.15.172:0/1147078043 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324031520 con 0x7f9334151a90 2024-03-16T19:32:28.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.084+0000 7f9331d9b640 1 -- 172.21.15.172:0/1147078043 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9324031680 con 0x7f9334151a90 2024-03-16T19:32:28.064 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.085+0000 7f9331d9b640 1 -- 172.21.15.172:0/1147078043 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324031a10 con 0x7f9334151a90 2024-03-16T19:32:28.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.085+0000 7f9338e4d640 1 -- 172.21.15.172:0/1147078043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9334151a90 msgr2=0x7f93340721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.085+0000 7f9338e4d640 1 --2- 172.21.15.172:0/1147078043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9334151a90 0x7f93340721d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9324009720 tx=0x7f932402b7c0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.085+0000 7f9338e4d640 1 -- 172.21.15.172:0/1147078043 shutdown_connections 2024-03-16T19:32:28.065 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.085+0000 7f9338e4d640 1 --2- 172.21.15.172:0/1147078043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9334151a90 0x7f93340721d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.085+0000 7f9338e4d640 1 -- 172.21.15.172:0/1147078043 >> 172.21.15.172:0/1147078043 conn(0x7f933406aed0 msgr2=0x7f933406b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:28.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.086+0000 7f9338e4d640 1 -- 172.21.15.172:0/1147078043 shutdown_connections 2024-03-16T19:32:28.066 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.086+0000 7f9338e4d640 1 -- 172.21.15.172:0/1147078043 wait complete. 2024-03-16T19:32:28.067 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.087+0000 7f9338e4d640 1 Processor -- start 2024-03-16T19:32:28.067 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.088+0000 7f9338e4d640 1 -- start start 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f9338e4d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93340c3b20 0x7f93341591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f9338e4d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93340c90b0 0x7f93340cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f9338e4d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9334151a90 0x7f93341471c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f933259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93340c3b20 0x7f93341591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f932bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93340c90b0 0x7f93340cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f933259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93340c3b20 0x7f93341591d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:54442/0 (socket says 172.21.15.172:54442) 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f932bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93340c90b0 0x7f93340cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46862/0 (socket says 172.21.15.172:46862) 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f9338e4d640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9334158db0 con 0x7f93340c90b0 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f9338e4d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9334145f50 con 0x7f93340c3b20 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93341587c0 con 0x7f9334151a90 2024-03-16T19:32:28.070 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f933259c640 1 -- 172.21.15.172:0/1865441812 learned_addr learned my addr 172.21.15.172:0/1865441812 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f9332d9d640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9334151a90 0x7f93341471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f933259c640 1 -- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9334151a90 msgr2=0x7f93341471c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f933259c640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9334151a90 0x7f93341471c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f933259c640 1 -- 172.21.15.172:0/1865441812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93340c90b0 msgr2=0x7f93340cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f933259c640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93340c90b0 0x7f93340cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f933259c640 1 -- 172.21.15.172:0/1865441812 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9324009400 con 0x7f93340c3b20 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.089+0000 7f9332d9d640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9334151a90 0x7f93341471c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.090+0000 7f933259c640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93340c3b20 0x7f93341591d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9324009840 tx=0x7f93240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.090+0000 7f932affd640 1 -- 172.21.15.172:0/1865441812 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324033c50 con 0x7f93340c3b20 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.090+0000 7f932affd640 1 -- 172.21.15.172:0/1865441812 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9324033db0 con 0x7f93340c3b20 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.090+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933415aec0 con 0x7f93340c3b20 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.090+0000 7f932affd640 1 -- 172.21.15.172:0/1865441812 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932403c640 con 0x7f93340c3b20 2024-03-16T19:32:28.071 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.090+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933415b400 con 0x7f93340c3b20 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.091+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9334075ba0 con 0x7f93340c3b20 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.091+0000 7f932affd640 1 -- 172.21.15.172:0/1865441812 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f932403a040 con 0x7f93340c3b20 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.092+0000 7f932affd640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f931406b3b0 0x7f931406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.092+0000 7f932affd640 1 -- 172.21.15.172:0/1865441812 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f93240ab490 con 0x7f93340c3b20 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.092+0000 7f932bfff640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f931406b3b0 0x7f931406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.092+0000 7f932affd640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9314073790 0x7f9314075c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.092+0000 7f932affd640 1 -- 172.21.15.172:0/1865441812 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9314076300 con 0x7f9314073790 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.092+0000 7f932affd640 1 -- 172.21.15.172:0/1865441812 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93240adb50 con 0x7f93340c3b20 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.092+0000 7f9332d9d640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9314073790 0x7f9314075c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.073 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.092+0000 7f932bfff640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f931406b3b0 0x7f931406d870 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f931c007f70 tx=0x7f931c007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.093+0000 7f9332d9d640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9314073790 0x7f9314075c30 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.074 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.094+0000 7f932affd640 1 -- 172.21.15.172:0/1865441812 <== osd.5 v2:172.21.15.174:6816/3985312519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9314076300 con 0x7f9314073790 2024-03-16T19:32:28.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.119+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 --> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9334072f00 con 0x7f9314073790 2024-03-16T19:32:28.099 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f932affd640 1 -- 172.21.15.172:0/1865441812 <== osd.5 v2:172.21.15.174:6816/3985312519 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f932c014040 con 0x7f9314073790 2024-03-16T19:32:28.099 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:32:28.100 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T19:32:28.100 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9314073790 msgr2=0x7f9314075c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:6816/3985312519,v1:172.21.15.174:6817/3985312519] conn(0x7f9314073790 0x7f9314075c30 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f931406b3b0 msgr2=0x7f931406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f931406b3b0 0x7f931406d870 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f931c007f70 tx=0x7f931c007f00 comp rx=0 tx=0).stop 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93340c3b20 msgr2=0x7f93341591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 --2- 172.21.15.172:0/1865441812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93340c3b20 0x7f93341591d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9324009840 tx=0x7f93240046c0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9332d9d640 1 -- 172.21.15.172:0/1865441812 reap_dead start 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 shutdown_connections 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 >> 172.21.15.172:0/1865441812 conn(0x7f933406aed0 msgr2=0x7f933414fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 shutdown_connections 2024-03-16T19:32:28.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.120+0000 7f9338e4d640 1 -- 172.21.15.172:0/1865441812 wait complete. 2024-03-16T19:32:28.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-16T19:32:28.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.268+0000 7f863bcae640 1 Processor -- start 2024-03-16T19:32:28.248 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.268+0000 7f863bcae640 1 -- start start 2024-03-16T19:32:28.249 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.269+0000 7f863bcae640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8634150fd0 0x7f8634157720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.269+0000 7f863bcae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c3360 0x7f86340c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.269+0000 7f863bcae640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86340c7590 con 0x7f86340c82d0 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.269+0000 7f863bcae640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86340c94c0 con 0x7f8634150fd0 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.269+0000 7f863bcae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86341513b0 con 0x7f86340c3360 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.269+0000 7f8639a4a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8634150fd0 0x7f8634157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.270+0000 7f863a24b640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f86340c82d0 0x7f86340cc670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40110/0 (socket says 172.21.15.172:40110) 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.270+0000 7f863a24b640 1 -- 172.21.15.172:0/1423674285 learned_addr learned my addr 172.21.15.172:0/1423674285 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.270+0000 7f8639249640 1 --2- 172.21.15.172:0/1423674285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c3360 0x7f86340c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.270+0000 7f8639a4a640 1 -- 172.21.15.172:0/1423674285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c3360 msgr2=0x7f86340c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.270+0000 7f8639a4a640 1 --2- 172.21.15.172:0/1423674285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c3360 0x7f86340c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.270+0000 7f8639a4a640 1 -- 172.21.15.172:0/1423674285 >> v1:172.21.15.172:6789/0 conn(0x7f86340c82d0 legacy=0x7f86340cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.250 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.270+0000 7f8639a4a640 1 -- 172.21.15.172:0/1423674285 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86340c8eb0 con 0x7f8634150fd0 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.271+0000 7f8639a4a640 1 --2- 172.21.15.172:0/1423674285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8634150fd0 0x7f8634157720 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f86300097a0 tx=0x7f863002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10134ccd85e1ae0b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.271+0000 7f8638a48640 1 -- 172.21.15.172:0/1423674285 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630031830 con 0x7f8634150fd0 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.271+0000 7f8638a48640 1 -- 172.21.15.172:0/1423674285 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8630031990 con 0x7f8634150fd0 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.271+0000 7f8638a48640 1 -- 172.21.15.172:0/1423674285 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630031d20 con 0x7f8634150fd0 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.272+0000 7f863bcae640 1 -- 172.21.15.172:0/1423674285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8634150fd0 msgr2=0x7f8634157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.272+0000 7f863bcae640 1 --2- 172.21.15.172:0/1423674285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8634150fd0 0x7f8634157720 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f86300097a0 tx=0x7f863002b8d0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.272+0000 7f863bcae640 1 -- 172.21.15.172:0/1423674285 shutdown_connections 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.272+0000 7f863bcae640 1 --2- 172.21.15.172:0/1423674285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c3360 0x7f86340c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.272+0000 7f863bcae640 1 --2- 172.21.15.172:0/1423674285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8634150fd0 0x7f8634157720 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.272+0000 7f863bcae640 1 -- 172.21.15.172:0/1423674285 >> 172.21.15.172:0/1423674285 conn(0x7f863406d510 msgr2=0x7f863406aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:28.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.272+0000 7f863bcae640 1 -- 172.21.15.172:0/1423674285 shutdown_connections 2024-03-16T19:32:28.253 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.272+0000 7f863bcae640 1 -- 172.21.15.172:0/1423674285 wait complete. 2024-03-16T19:32:28.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.273+0000 7f863bcae640 1 Processor -- start 2024-03-16T19:32:28.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.273+0000 7f863bcae640 1 -- start start 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f863bcae640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86340c3360 0x7f86341ebbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f863bcae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c82d0 0x7f8634151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f863bcae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634150fd0 0x7f8634152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f863bcae640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86341480d0 con 0x7f86340c3360 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f863bcae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8634148390 con 0x7f8634150fd0 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f863bcae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8634148670 con 0x7f86340c82d0 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f8639a4a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86340c3360 0x7f86341ebbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f8639a4a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86340c3360 0x7f86341ebbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:46876/0 (socket says 172.21.15.172:46876) 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f8639a4a640 1 -- 172.21.15.172:0/3235291533 learned_addr learned my addr 172.21.15.172:0/3235291533 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f863a24b640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634150fd0 0x7f8634152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f8639249640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c82d0 0x7f8634151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f8639a4a640 1 -- 172.21.15.172:0/3235291533 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c82d0 msgr2=0x7f8634151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f8639a4a640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c82d0 0x7f8634151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f8639a4a640 1 -- 172.21.15.172:0/3235291533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634150fd0 msgr2=0x7f8634152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.255 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f8639a4a640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634150fd0 0x7f8634152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f8639a4a640 1 -- 172.21.15.172:0/3235291533 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630009400 con 0x7f86340c3360 2024-03-16T19:32:28.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.274+0000 7f863a24b640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634150fd0 0x7f8634152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:32:28.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.275+0000 7f8639a4a640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86340c3360 0x7f86341ebbf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8630034040 tx=0x7f86300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.275+0000 7f862a7fc640 1 -- 172.21.15.172:0/3235291533 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863003a9e0 con 0x7f86340c3360 2024-03-16T19:32:28.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.275+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8634148950 con 0x7f86340c3360 2024-03-16T19:32:28.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.275+0000 7f862a7fc640 1 -- 172.21.15.172:0/3235291533 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863003ab40 con 0x7f86340c3360 2024-03-16T19:32:28.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.275+0000 7f862a7fc640 1 -- 172.21.15.172:0/3235291533 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863003a9e0 con 0x7f86340c3360 2024-03-16T19:32:28.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.275+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8634151630 con 0x7f86340c3360 2024-03-16T19:32:28.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.276+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86340c4050 con 0x7f86340c3360 2024-03-16T19:32:28.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.277+0000 7f862a7fc640 1 -- 172.21.15.172:0/3235291533 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f863003acb0 con 0x7f86340c3360 2024-03-16T19:32:28.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.277+0000 7f862a7fc640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f861806b470 0x7f861806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.278+0000 7f862a7fc640 1 -- 172.21.15.172:0/3235291533 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f86300ab570 con 0x7f86340c3360 2024-03-16T19:32:28.257 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.278+0000 7f8639249640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f861806b470 0x7f861806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.258 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.278+0000 7f8639249640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f861806b470 0x7f861806d930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f86341e64b0 tx=0x7f862400f040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.282+0000 7f862a7fc640 1 -- 172.21.15.172:0/3235291533 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86300760d0 con 0x7f86340c3360 2024-03-16T19:32:28.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.500+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f86340c7aa0 con 0x7f861806b470 2024-03-16T19:32:28.480 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.500+0000 7f862a7fc640 1 -- 172.21.15.172:0/3235291533 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8624013040 con 0x7f861806b470 2024-03-16T19:32:28.480 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 5 to deep-scrub 2024-03-16T19:32:28.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f861806b470 msgr2=0x7f861806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f861806b470 0x7f861806d930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f86341e64b0 tx=0x7f862400f040 comp rx=0 tx=0).stop 2024-03-16T19:32:28.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86340c3360 msgr2=0x7f86341ebbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86340c3360 0x7f86341ebbf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8630034040 tx=0x7f86300046c0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 shutdown_connections 2024-03-16T19:32:28.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8634150fd0 0x7f8634152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f861806b470 0x7f861806d930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86340c82d0 0x7f8634151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 --2- 172.21.15.172:0/3235291533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86340c3360 0x7f86341ebbf0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 >> 172.21.15.172:0/3235291533 conn(0x7f863406d510 msgr2=0x7f863414fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:28.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 shutdown_connections 2024-03-16T19:32:28.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.503+0000 7f863bcae640 1 -- 172.21.15.172:0/3235291533 wait complete. 2024-03-16T19:32:28.496 INFO:tasks.ceph:Scrubbing osd.6 2024-03-16T19:32:28.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-16T19:32:28.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.658+0000 7f944693a640 1 Processor -- start 2024-03-16T19:32:28.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.658+0000 7f944693a640 1 -- start start 2024-03-16T19:32:28.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.659+0000 7f944693a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9440152b80 0x7f9440070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.659+0000 7f944693a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f94400b0ee0 0x7f9440071200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.659+0000 7f944693a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 0x7f944015bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f944693a640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94400c3830 con 0x7f94400b0ee0 2024-03-16T19:32:28.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f944693a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94400c5110 con 0x7f9440152b80 2024-03-16T19:32:28.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f944693a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94400c34e0 con 0x7f94400cbad0 2024-03-16T19:32:28.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f943f7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f94400b0ee0 0x7f9440071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f9444ed7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 0x7f944015bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f943ffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9440152b80 0x7f9440070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f943f7fe640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f94400b0ee0 0x7f9440071200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:48488/0 (socket says 172.21.15.172:48488) 2024-03-16T19:32:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f9444ed7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 0x7f944015bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:40786/0 (socket says 172.21.15.172:40786) 2024-03-16T19:32:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f943f7fe640 1 -- 172.21.15.172:0/2305914502 learned_addr learned my addr 172.21.15.172:0/2305914502 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f943f7fe640 1 -- 172.21.15.172:0/2305914502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 msgr2=0x7f944015bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f943f7fe640 1 --2- 172.21.15.172:0/2305914502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 0x7f944015bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f943f7fe640 1 -- 172.21.15.172:0/2305914502 >> v2:172.21.15.174:3300/0 conn(0x7f9440152b80 msgr2=0x7f9440070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f943f7fe640 1 --2- 172.21.15.172:0/2305914502 >> v2:172.21.15.174:3300/0 conn(0x7f9440152b80 0x7f9440070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.660+0000 7f943f7fe640 1 -- 172.21.15.172:0/2305914502 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94400c53d0 con 0x7f94400b0ee0 2024-03-16T19:32:28.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.661+0000 7f943f7fe640 1 --2- 172.21.15.172:0/2305914502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94400b0ee0 0x7f9440071200 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f94400caf60 tx=0x7f943402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142b10955defa20e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.661+0000 7f943effd640 1 -- 172.21.15.172:0/2305914502 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434031870 con 0x7f94400b0ee0 2024-03-16T19:32:28.642 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.661+0000 7f943effd640 1 -- 172.21.15.172:0/2305914502 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94340319d0 con 0x7f94400b0ee0 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.661+0000 7f943effd640 1 -- 172.21.15.172:0/2305914502 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434031d60 con 0x7f94400b0ee0 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.662+0000 7f944693a640 1 -- 172.21.15.172:0/2305914502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94400b0ee0 msgr2=0x7f9440071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.662+0000 7f944693a640 1 --2- 172.21.15.172:0/2305914502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94400b0ee0 0x7f9440071200 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f94400caf60 tx=0x7f943402b910 comp rx=0 tx=0).stop 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.662+0000 7f944693a640 1 -- 172.21.15.172:0/2305914502 shutdown_connections 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.662+0000 7f944693a640 1 --2- 172.21.15.172:0/2305914502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 0x7f944015bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.662+0000 7f944693a640 1 --2- 172.21.15.172:0/2305914502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94400b0ee0 0x7f9440071200 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.662+0000 7f944693a640 1 --2- 172.21.15.172:0/2305914502 >> v2:172.21.15.174:3300/0 conn(0x7f9440152b80 0x7f9440070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.662+0000 7f944693a640 1 -- 172.21.15.172:0/2305914502 >> 172.21.15.172:0/2305914502 conn(0x7f944006f940 msgr2=0x7f944006fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.662+0000 7f944693a640 1 -- 172.21.15.172:0/2305914502 shutdown_connections 2024-03-16T19:32:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.662+0000 7f944693a640 1 -- 172.21.15.172:0/2305914502 wait complete. 2024-03-16T19:32:28.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.663+0000 7f944693a640 1 Processor -- start 2024-03-16T19:32:28.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f944693a640 1 -- start start 2024-03-16T19:32:28.645 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f944693a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94400b0ee0 0x7f944014fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f944693a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 0x7f94400b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f944693a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9440152b80 0x7f944014e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f944693a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944014f060 con 0x7f94400b0ee0 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f944693a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9440150420 con 0x7f9440152b80 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f944693a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94400c8760 con 0x7f94400cbad0 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f943ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94400b0ee0 0x7f944014fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f9444ed7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9440152b80 0x7f944014e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f9444ed7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9440152b80 0x7f944014e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:58352/0 (socket says 172.21.15.172:58352) 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f9444ed7640 1 -- 172.21.15.172:0/1416126061 learned_addr learned my addr 172.21.15.172:0/1416126061 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f943f7fe640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 0x7f94400b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.664+0000 7f943ffff640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94400b0ee0 0x7f944014fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:48498/0 (socket says 172.21.15.172:48498) 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f9444ed7640 1 -- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 msgr2=0x7f94400b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.646 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f9444ed7640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94400cbad0 0x7f94400b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f9444ed7640 1 -- 172.21.15.172:0/1416126061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94400b0ee0 msgr2=0x7f944014fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f9444ed7640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f94400b0ee0 0x7f944014fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f9444ed7640 1 -- 172.21.15.172:0/1416126061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340094d0 con 0x7f9440152b80 2024-03-16T19:32:28.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f9444ed7640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9440152b80 0x7f944014e2a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f943800eed0 tx=0x7f943800b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f943cff9640 1 -- 172.21.15.172:0/1416126061 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438010040 con 0x7f9440152b80 2024-03-16T19:32:28.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f943cff9640 1 -- 172.21.15.172:0/1416126061 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9438014400 con 0x7f9440152b80 2024-03-16T19:32:28.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f943cff9640 1 -- 172.21.15.172:0/1416126061 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94380147b0 con 0x7f9440152b80 2024-03-16T19:32:28.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.665+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944014de10 con 0x7f9440152b80 2024-03-16T19:32:28.647 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.666+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94400c7ec0 con 0x7f9440152b80 2024-03-16T19:32:28.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.666+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94401596a0 con 0x7f9440152b80 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.667+0000 7f943cff9640 1 -- 172.21.15.172:0/1416126061 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9438014910 con 0x7f9440152b80 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.667+0000 7f943cff9640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f942c06b3e0 0x7f942c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.668+0000 7f943cff9640 1 -- 172.21.15.172:0/1416126061 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f943808a5c0 con 0x7f9440152b80 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.668+0000 7f943ffff640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f942c06b3e0 0x7f942c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.668+0000 7f943cff9640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f942c073760 0x7f942c075c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.668+0000 7f943cff9640 1 -- 172.21.15.172:0/1416126061 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f942c0762d0 con 0x7f942c073760 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.668+0000 7f943cff9640 1 -- 172.21.15.172:0/1416126061 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f943808cc80 con 0x7f9440152b80 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.668+0000 7f943f7fe640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f942c073760 0x7f942c075c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.668+0000 7f943ffff640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f942c06b3e0 0x7f942c06d8a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9430009aa0 tx=0x7f9430009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.668+0000 7f943f7fe640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f942c073760 0x7f942c075c00 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.649 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.669+0000 7f943cff9640 1 -- 172.21.15.172:0/1416126061 <== osd.6 v2:172.21.15.174:6819/3676139844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f942c0762d0 con 0x7f942c073760 2024-03-16T19:32:28.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.686+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 --> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f94400681c0 con 0x7f942c073760 2024-03-16T19:32:28.666 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f943cff9640 1 -- 172.21.15.172:0/1416126061 <== osd.6 v2:172.21.15.174:6819/3676139844 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f94400681c0 con 0x7f942c073760 2024-03-16T19:32:28.666 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:32:28.666 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T19:32:28.667 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:32:28.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f942c073760 msgr2=0x7f942c075c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f944693a640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:6819/3676139844,v1:172.21.15.174:6821/3676139844] conn(0x7f942c073760 0x7f942c075c00 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f942c06b3e0 msgr2=0x7f942c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f944693a640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f942c06b3e0 0x7f942c06d8a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9430009aa0 tx=0x7f9430009190 comp rx=0 tx=0).stop 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9440152b80 msgr2=0x7f944014e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f944693a640 1 --2- 172.21.15.172:0/1416126061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9440152b80 0x7f944014e2a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f943800eed0 tx=0x7f943800b480 comp rx=0 tx=0).stop 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f9444ed7640 1 -- 172.21.15.172:0/1416126061 reap_dead start 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 shutdown_connections 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 >> 172.21.15.172:0/1416126061 conn(0x7f944006f940 msgr2=0x7f9440150e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.687+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 shutdown_connections 2024-03-16T19:32:28.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.688+0000 7f944693a640 1 -- 172.21.15.172:0/1416126061 wait complete. 2024-03-16T19:32:28.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-16T19:32:28.829 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.850+0000 7f9d7380f640 1 Processor -- start 2024-03-16T19:32:28.830 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.850+0000 7f9d7380f640 1 -- start start 2024-03-16T19:32:28.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.851+0000 7f9d7380f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9d6c0c7c10 0x7f9d6c0c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.831 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.851+0000 7f9d7380f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c0c7120 con 0x7f9d6c0c84e0 2024-03-16T19:32:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.851+0000 7f9d7380f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c0c73e0 con 0x7f9d6c154a30 2024-03-16T19:32:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.851+0000 7f9d7380f640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6c0c76c0 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.851+0000 7f9d70daa640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f9d6c0c7c10 0x7f9d6c0c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.851+0000 7f9d71dac640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f9d6c0c84e0 0x7f9d6c165140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:33322/0 (socket says 172.21.15.172:33322) 2024-03-16T19:32:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.852+0000 7f9d71dac640 1 -- 172.21.15.172:0/3300470868 learned_addr learned my addr 172.21.15.172:0/3300470868 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.852+0000 7f9d70daa640 1 -- 172.21.15.172:0/3300470868 >> v1:172.21.15.174:6789/0 conn(0x7f9d6c154a30 legacy=0x7f9d6c0cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.852+0000 7f9d70daa640 1 -- 172.21.15.172:0/3300470868 >> v1:172.21.15.172:6789/0 conn(0x7f9d6c0c84e0 legacy=0x7f9d6c165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.852+0000 7f9d70daa640 1 -- 172.21.15.172:0/3300470868 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c15ddb0 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.853+0000 7f9d70daa640 1 --2- 172.21.15.172:0/3300470868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d6c0c7c10 0x7f9d6c0c8a00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0097a0 tx=0x7f9d5c02b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d361ba749cacec7d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.853+0000 7f9d63fff640 1 -- 172.21.15.172:0/3300470868 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c030510 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.853+0000 7f9d63fff640 1 -- 172.21.15.172:0/3300470868 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c02e050 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.853+0000 7f9d63fff640 1 -- 172.21.15.172:0/3300470868 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c030890 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.854+0000 7f9d7380f640 1 -- 172.21.15.172:0/3300470868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d6c0c7c10 msgr2=0x7f9d6c0c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.854+0000 7f9d7380f640 1 --2- 172.21.15.172:0/3300470868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d6c0c7c10 0x7f9d6c0c8a00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0097a0 tx=0x7f9d5c02b600 comp rx=0 tx=0).stop 2024-03-16T19:32:28.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.855+0000 7f9d7380f640 1 -- 172.21.15.172:0/3300470868 shutdown_connections 2024-03-16T19:32:28.834 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.855+0000 7f9d7380f640 1 --2- 172.21.15.172:0/3300470868 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d6c0c7c10 0x7f9d6c0c8a00 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.855+0000 7f9d7380f640 1 -- 172.21.15.172:0/3300470868 >> 172.21.15.172:0/3300470868 conn(0x7f9d6c06d510 msgr2=0x7f9d6c06aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:28.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.855+0000 7f9d7380f640 1 -- 172.21.15.172:0/3300470868 shutdown_connections 2024-03-16T19:32:28.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.855+0000 7f9d7380f640 1 -- 172.21.15.172:0/3300470868 wait complete. 2024-03-16T19:32:28.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.856+0000 7f9d7380f640 1 Processor -- start 2024-03-16T19:32:28.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.856+0000 7f9d7380f640 1 -- start start 2024-03-16T19:32:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d7380f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9d6c0c7c10 0x7f9d6c0c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d7380f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d6c0c84e0 0x7f9d6c14c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d7380f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d6c154a30 0x7f9d6c15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d7380f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c157130 con 0x7f9d6c0c84e0 2024-03-16T19:32:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d7380f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0c63a0 con 0x7f9d6c154a30 2024-03-16T19:32:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d7380f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0c5de0 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d71dac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d6c154a30 0x7f9d6c15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d715ab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9d6c0c7c10 0x7f9d6c0c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d71dac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d6c154a30 0x7f9d6c15c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:58358/0 (socket says 172.21.15.172:58358) 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d71dac640 1 -- 172.21.15.172:0/2741007649 learned_addr learned my addr 172.21.15.172:0/2741007649 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d70daa640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d6c0c84e0 0x7f9d6c14c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d715ab640 1 -- 172.21.15.172:0/2741007649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d6c154a30 msgr2=0x7f9d6c15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d715ab640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d6c154a30 0x7f9d6c15c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d715ab640 1 -- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d6c0c84e0 msgr2=0x7f9d6c14c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d715ab640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d6c0c84e0 0x7f9d6c14c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d715ab640 1 -- 172.21.15.172:0/2741007649 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c009400 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d71dac640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d6c154a30 0x7f9d6c15c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.858+0000 7f9d715ab640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9d6c0c7c10 0x7f9d6c0c49e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800c4a0 tx=0x7f9d6800c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.859+0000 7f9d61ffb640 1 -- 172.21.15.172:0/2741007649 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68005480 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.859+0000 7f9d61ffb640 1 -- 172.21.15.172:0/2741007649 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d680035f0 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.859+0000 7f9d61ffb640 1 -- 172.21.15.172:0/2741007649 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d680039a0 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.859+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6c0c60c0 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.859+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6c15bd90 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.860+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c154290 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.861+0000 7f9d61ffb640 1 -- 172.21.15.172:0/2741007649 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9d68003b00 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.861+0000 7f9d61ffb640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d5006b3b0 0x7f9d5006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:28.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.861+0000 7f9d61ffb640 1 -- 172.21.15.172:0/2741007649 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f9d68089190 con 0x7f9d6c0c7c10 2024-03-16T19:32:28.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.861+0000 7f9d70daa640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d5006b3b0 0x7f9d5006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:28.842 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.862+0000 7f9d70daa640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d5006b3b0 0x7f9d5006d870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c034000 tx=0x7f9d5c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:28.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:28.865+0000 7f9d61ffb640 1 -- 172.21.15.172:0/2741007649 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d68053e90 con 0x7f9d6c0c7c10 2024-03-16T19:32:29.126 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.147+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d6c072430 con 0x7f9d5006b3b0 2024-03-16T19:32:29.127 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.147+0000 7f9d61ffb640 1 -- 172.21.15.172:0/2741007649 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9d5c030810 con 0x7f9d5006b3b0 2024-03-16T19:32:29.127 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 6 to deep-scrub 2024-03-16T19:32:29.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d5006b3b0 msgr2=0x7f9d5006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d5006b3b0 0x7f9d5006d870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c034000 tx=0x7f9d5c008030 comp rx=0 tx=0).stop 2024-03-16T19:32:29.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9d6c0c7c10 msgr2=0x7f9d6c0c49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9d6c0c7c10 0x7f9d6c0c49e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800c4a0 tx=0x7f9d6800c960 comp rx=0 tx=0).stop 2024-03-16T19:32:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 shutdown_connections 2024-03-16T19:32:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d6c154a30 0x7f9d6c15c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d6c0c84e0 0x7f9d6c14c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f9d5006b3b0 0x7f9d5006d870 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 --2- 172.21.15.172:0/2741007649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9d6c0c7c10 0x7f9d6c0c49e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.149+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 >> 172.21.15.172:0/2741007649 conn(0x7f9d6c06d510 msgr2=0x7f9d6c06aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.150+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 shutdown_connections 2024-03-16T19:32:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.150+0000 7f9d7380f640 1 -- 172.21.15.172:0/2741007649 wait complete. 2024-03-16T19:32:29.142 INFO:tasks.ceph:Scrubbing osd.7 2024-03-16T19:32:29.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-16T19:32:29.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.303+0000 7f4c328aa640 1 Processor -- start 2024-03-16T19:32:29.283 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.303+0000 7f4c328aa640 1 -- start start 2024-03-16T19:32:29.284 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c328aa640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4c2c152b80 0x7f4c2c070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c328aa640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4c2c0b0ee0 0x7f4c2c071200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c328aa640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0c3830 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c328aa640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c0c5110 con 0x7f4c2c152b80 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c328aa640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c0c34e0 con 0x7f4c2c0b0ee0 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c2bfff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4c2c152b80 0x7f4c2c070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c2bfff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f4c2c152b80 0x7f4c2c070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:48512/0 (socket says 172.21.15.172:48512) 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c2bfff640 1 -- 172.21.15.172:0/3809016008 learned_addr learned my addr 172.21.15.172:0/3809016008 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c23fff640 1 --2- 172.21.15.172:0/3809016008 >> v2:172.21.15.174:3300/0 conn(0x7f4c2c0b0ee0 0x7f4c2c071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c2bfff640 1 -- 172.21.15.172:0/3809016008 >> v2:172.21.15.174:3300/0 conn(0x7f4c2c0b0ee0 msgr2=0x7f4c2c071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c2bfff640 1 --2- 172.21.15.172:0/3809016008 >> v2:172.21.15.174:3300/0 conn(0x7f4c2c0b0ee0 0x7f4c2c071200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c2bfff640 1 -- 172.21.15.172:0/3809016008 >> v1:172.21.15.174:6789/0 conn(0x7f4c2c0cbad0 legacy=0x7f4c2c15bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.305+0000 7f4c2bfff640 1 -- 172.21.15.172:0/3809016008 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0c53d0 con 0x7f4c2c152b80 2024-03-16T19:32:29.286 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.306+0000 7f4c2bfff640 1 --2- 172.21.15.172:0/3809016008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c2c152b80 0x7f4c2c070cc0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0cb810 tx=0x7f4c1c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0fc4601a105a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.286 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.306+0000 7f4c2b7fe640 1 -- 172.21.15.172:0/3809016008 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c030860 con 0x7f4c2c152b80 2024-03-16T19:32:29.286 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.306+0000 7f4c2b7fe640 1 -- 172.21.15.172:0/3809016008 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1c0309c0 con 0x7f4c2c152b80 2024-03-16T19:32:29.286 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.306+0000 7f4c2b7fe640 1 -- 172.21.15.172:0/3809016008 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c030d50 con 0x7f4c2c152b80 2024-03-16T19:32:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.307+0000 7f4c328aa640 1 -- 172.21.15.172:0/3809016008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c2c152b80 msgr2=0x7f4c2c070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.307+0000 7f4c328aa640 1 --2- 172.21.15.172:0/3809016008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c2c152b80 0x7f4c2c070cc0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0cb810 tx=0x7f4c1c02b9a0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.307+0000 7f4c328aa640 1 -- 172.21.15.172:0/3809016008 shutdown_connections 2024-03-16T19:32:29.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.307+0000 7f4c328aa640 1 --2- 172.21.15.172:0/3809016008 >> v2:172.21.15.174:3300/0 conn(0x7f4c2c0b0ee0 0x7f4c2c071200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.307+0000 7f4c328aa640 1 --2- 172.21.15.172:0/3809016008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c2c152b80 0x7f4c2c070cc0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.307+0000 7f4c328aa640 1 -- 172.21.15.172:0/3809016008 >> 172.21.15.172:0/3809016008 conn(0x7f4c2c06f940 msgr2=0x7f4c2c06fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:29.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.307+0000 7f4c328aa640 1 -- 172.21.15.172:0/3809016008 shutdown_connections 2024-03-16T19:32:29.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.308+0000 7f4c328aa640 1 -- 172.21.15.172:0/3809016008 wait complete. 2024-03-16T19:32:29.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.309+0000 7f4c328aa640 1 Processor -- start 2024-03-16T19:32:29.288 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.309+0000 7f4c328aa640 1 -- start start 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c328aa640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c2c0b0ee0 0x7f4c2c15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c328aa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c2c0cbad0 0x7f4c2c0cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c328aa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2c152b80 0x7f4c2c1f0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c328aa640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0c9430 con 0x7f4c2c0b0ee0 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c328aa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0c9710 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c328aa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0c99f0 con 0x7f4c2c152b80 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c2bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c2c0b0ee0 0x7f4c2c15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c23fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c2c0cbad0 0x7f4c2c0cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c2bfff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c2c0b0ee0 0x7f4c2c15a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:48514/0 (socket says 172.21.15.172:48514) 2024-03-16T19:32:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c23fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c2c0cbad0 0x7f4c2c0cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:58384/0 (socket says 172.21.15.172:58384) 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c2bfff640 1 -- 172.21.15.172:0/961390231 learned_addr learned my addr 172.21.15.172:0/961390231 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.310+0000 7f4c30e47640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2c152b80 0x7f4c2c1f0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c23fff640 1 -- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2c152b80 msgr2=0x7f4c2c1f0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c23fff640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2c152b80 0x7f4c2c1f0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c23fff640 1 -- 172.21.15.172:0/961390231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c2c0b0ee0 msgr2=0x7f4c2c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c23fff640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c2c0b0ee0 0x7f4c2c15a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c23fff640 1 -- 172.21.15.172:0/961390231 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c0094d0 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c23fff640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c2c0cbad0 0x7f4c2c0cd530 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400e820 tx=0x7f4c1400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c297fa640 1 -- 172.21.15.172:0/961390231 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1400c940 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c297fa640 1 -- 172.21.15.172:0/961390231 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1400caa0 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c1ef910 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c297fa640 1 -- 172.21.15.172:0/961390231 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1400ce30 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.311+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c0cdf00 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.312+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c2c1596a0 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.312+0000 7f4c297fa640 1 -- 172.21.15.172:0/961390231 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4c14013470 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.313+0000 7f4c297fa640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4c0c06b3b0 0x7f4c0c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.313+0000 7f4c297fa640 1 -- 172.21.15.172:0/961390231 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f4c14089e20 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.313+0000 7f4c2bfff640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4c0c06b3b0 0x7f4c0c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.313+0000 7f4c297fa640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4c0c073730 0x7f4c0c075bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.313+0000 7f4c297fa640 1 -- 172.21.15.172:0/961390231 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c0c0762a0 con 0x7f4c0c073730 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.313+0000 7f4c297fa640 1 -- 172.21.15.172:0/961390231 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c1408c4e0 con 0x7f4c2c0cbad0 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.313+0000 7f4c30e47640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4c0c073730 0x7f4c0c075bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.313+0000 7f4c2bfff640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4c0c06b3b0 0x7f4c0c06d870 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0090b0 tx=0x7f4c1c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.314+0000 7f4c30e47640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4c0c073730 0x7f4c0c075bd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.315+0000 7f4c297fa640 1 -- 172.21.15.172:0/961390231 <== osd.7 v2:172.21.15.174:6800/339941177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c0c0762a0 con 0x7f4c0c073730 2024-03-16T19:32:29.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.339+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 --> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4c2c0681c0 con 0x7f4c0c073730 2024-03-16T19:32:29.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c297fa640 1 -- 172.21.15.172:0/961390231 <== osd.7 v2:172.21.15.174:6800/339941177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4c2c0681c0 con 0x7f4c0c073730 2024-03-16T19:32:29.320 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:32:29.320 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-16T19:32:29.320 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:32:29.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4c0c073730 msgr2=0x7f4c0c075bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:6800/339941177,v1:172.21.15.174:6801/339941177] conn(0x7f4c0c073730 0x7f4c0c075bd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4c0c06b3b0 msgr2=0x7f4c0c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4c0c06b3b0 0x7f4c0c06d870 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0090b0 tx=0x7f4c1c008030 comp rx=0 tx=0).stop 2024-03-16T19:32:29.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c2c0cbad0 msgr2=0x7f4c2c0cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 --2- 172.21.15.172:0/961390231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c2c0cbad0 0x7f4c2c0cd530 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400e820 tx=0x7f4c1400ece0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c30e47640 1 -- 172.21.15.172:0/961390231 reap_dead start 2024-03-16T19:32:29.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 shutdown_connections 2024-03-16T19:32:29.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 >> 172.21.15.172:0/961390231 conn(0x7f4c2c06f940 msgr2=0x7f4c2c15c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:29.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 shutdown_connections 2024-03-16T19:32:29.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.340+0000 7f4c328aa640 1 -- 172.21.15.172:0/961390231 wait complete. 2024-03-16T19:32:29.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-16T19:32:29.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.488+0000 7f7dae8b5640 1 Processor -- start 2024-03-16T19:32:29.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.488+0000 7f7dae8b5640 1 -- start start 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.489+0000 7f7dae8b5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7da8154bd0 0x7f7da80ca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.489+0000 7f7dae8b5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.489+0000 7f7dae8b5640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da806f430 con 0x7f7da80ce970 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.489+0000 7f7dae8b5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da806db50 con 0x7f7da8154bd0 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.489+0000 7f7dae8b5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da8154fb0 con 0x7f7da80c8b50 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.490+0000 7f7dace52640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7da80ce970 0x7f7da815abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:33324/0 (socket says 172.21.15.172:33324) 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.490+0000 7f7da7fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7da8154bd0 0x7f7da80ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.490+0000 7f7dace52640 1 -- 172.21.15.172:0/992174049 learned_addr learned my addr 172.21.15.172:0/992174049 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.490+0000 7f7da77fe640 1 --2- 172.21.15.172:0/992174049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.490+0000 7f7da7fff640 1 -- 172.21.15.172:0/992174049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 msgr2=0x7f7da80ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.490+0000 7f7da7fff640 1 --2- 172.21.15.172:0/992174049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.490+0000 7f7da7fff640 1 -- 172.21.15.172:0/992174049 >> v1:172.21.15.172:6789/0 conn(0x7f7da80ce970 legacy=0x7f7da815abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.490+0000 7f7da7fff640 1 -- 172.21.15.172:0/992174049 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da815ed70 con 0x7f7da8154bd0 2024-03-16T19:32:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.490+0000 7f7da77fe640 1 --2- 172.21.15.172:0/992174049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:32:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.491+0000 7f7da7fff640 1 --2- 172.21.15.172:0/992174049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 0x7f7da80ca450 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7d980097a0 tx=0x7f7d9802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42af197d60a9a021 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.491+0000 7f7da6ffd640 1 -- 172.21.15.172:0/992174049 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98030610 con 0x7f7da8154bd0 2024-03-16T19:32:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.491+0000 7f7da6ffd640 1 -- 172.21.15.172:0/992174049 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d98030770 con 0x7f7da8154bd0 2024-03-16T19:32:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.491+0000 7f7da6ffd640 1 -- 172.21.15.172:0/992174049 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98030b00 con 0x7f7da8154bd0 2024-03-16T19:32:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.492+0000 7f7dae8b5640 1 -- 172.21.15.172:0/992174049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 msgr2=0x7f7da80ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.492+0000 7f7dae8b5640 1 --2- 172.21.15.172:0/992174049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 0x7f7da80ca450 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7d980097a0 tx=0x7f7d9802b960 comp rx=0 tx=0).stop 2024-03-16T19:32:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.492+0000 7f7dae8b5640 1 -- 172.21.15.172:0/992174049 shutdown_connections 2024-03-16T19:32:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.492+0000 7f7dae8b5640 1 --2- 172.21.15.172:0/992174049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.492+0000 7f7dae8b5640 1 --2- 172.21.15.172:0/992174049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 0x7f7da80ca450 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.492+0000 7f7dae8b5640 1 -- 172.21.15.172:0/992174049 >> 172.21.15.172:0/992174049 conn(0x7f7da8066e00 msgr2=0x7f7da80671e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.492+0000 7f7dae8b5640 1 -- 172.21.15.172:0/992174049 shutdown_connections 2024-03-16T19:32:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.492+0000 7f7dae8b5640 1 -- 172.21.15.172:0/992174049 wait complete. 2024-03-16T19:32:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.493+0000 7f7dae8b5640 1 Processor -- start 2024-03-16T19:32:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.494+0000 7f7dae8b5640 1 -- start start 2024-03-16T19:32:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.494+0000 7f7dae8b5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.494+0000 7f7dae8b5640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7da80ce970 0x7f7da80c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.494+0000 7f7dae8b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 0x7f7da80c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dae8b5640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da80b0390 con 0x7f7da80ce970 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dae8b5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da80af2b0 con 0x7f7da8154bd0 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dae8b5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da80af550 con 0x7f7da80c8b50 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7da77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7da80ce970 0x7f7da80c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dace52640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 0x7f7da80c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7da77fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7da80ce970 0x7f7da80c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:48526/0 (socket says 172.21.15.172:48526) 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dace52640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 0x7f7da80c69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:58410/0 (socket says 172.21.15.172:58410) 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7da7fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80b0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:40828/0 (socket says 172.21.15.172:40828) 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dace52640 1 -- 172.21.15.172:0/3122312450 learned_addr learned my addr 172.21.15.172:0/3122312450 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:29.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dace52640 1 -- 172.21.15.172:0/3122312450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 msgr2=0x7f7da80b0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dace52640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80b0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dace52640 1 -- 172.21.15.172:0/3122312450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7da80ce970 msgr2=0x7f7da80c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dace52640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7da80ce970 0x7f7da80c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dace52640 1 -- 172.21.15.172:0/3122312450 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98009400 con 0x7f7da8154bd0 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dace52640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 0x7f7da80c69b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7da000cbf0 tx=0x7f7da000a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7da4ff9640 1 -- 172.21.15.172:0/3122312450 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da000ad40 con 0x7f7da8154bd0 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7da4ff9640 1 -- 172.21.15.172:0/3122312450 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da0012400 con 0x7f7da8154bd0 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7da4ff9640 1 -- 172.21.15.172:0/3122312450 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da000aea0 con 0x7f7da8154bd0 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.495+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da80af810 con 0x7f7da8154bd0 2024-03-16T19:32:29.477 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.496+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da80afdd0 con 0x7f7da8154bd0 2024-03-16T19:32:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.497+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da80ca970 con 0x7f7da8154bd0 2024-03-16T19:32:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.497+0000 7f7da4ff9640 1 -- 172.21.15.172:0/3122312450 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7da0007070 con 0x7f7da8154bd0 2024-03-16T19:32:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.497+0000 7f7da4ff9640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d9406b3b0 0x7f7d9406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.498+0000 7f7da4ff9640 1 -- 172.21.15.172:0/3122312450 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f7da0089c80 con 0x7f7da8154bd0 2024-03-16T19:32:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.498+0000 7f7da7fff640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d9406b3b0 0x7f7d9406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.498+0000 7f7da7fff640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d9406b3b0 0x7f7d9406d870 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7d98036000 tx=0x7f7d9803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.481 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.502+0000 7f7da4ff9640 1 -- 172.21.15.172:0/3122312450 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7da80ca970 con 0x7f7da8154bd0 2024-03-16T19:32:29.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.761+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f7da815df50 con 0x7f7d9406b3b0 2024-03-16T19:32:29.741 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.762+0000 7f7da4ff9640 1 -- 172.21.15.172:0/3122312450 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7d9803f040 con 0x7f7d9406b3b0 2024-03-16T19:32:29.741 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 7 to deep-scrub 2024-03-16T19:32:29.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d9406b3b0 msgr2=0x7f7d9406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d9406b3b0 0x7f7d9406d870 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7d98036000 tx=0x7f7d9803b040 comp rx=0 tx=0).stop 2024-03-16T19:32:29.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 msgr2=0x7f7da80c69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.744 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 0x7f7da80c69b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7da000cbf0 tx=0x7f7da000a430 comp rx=0 tx=0).stop 2024-03-16T19:32:29.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 shutdown_connections 2024-03-16T19:32:29.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da8154bd0 0x7f7da80c69b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7da80ce970 0x7f7da80c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7d9406b3b0 0x7f7d9406d870 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 --2- 172.21.15.172:0/3122312450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da80c8b50 0x7f7da80b0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 >> 172.21.15.172:0/3122312450 conn(0x7f7da8066e00 msgr2=0x7f7da8070ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:29.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 shutdown_connections 2024-03-16T19:32:29.745 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.764+0000 7f7dae8b5640 1 -- 172.21.15.172:0/3122312450 wait complete. 2024-03-16T19:32:29.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:29.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.897+0000 7ff6a27ae640 1 Processor -- start 2024-03-16T19:32:29.877 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.897+0000 7ff6a27ae640 1 -- start start 2024-03-16T19:32:29.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.898+0000 7ff6a27ae640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff69c06bb10 0x7ff69c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.898+0000 7ff6a27ae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c0cc760 0x7ff69c158b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.898+0000 7ff6a27ae640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69c0c5090 con 0x7ff69c06bef0 2024-03-16T19:32:29.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.898+0000 7ff6a27ae640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69c0caee0 con 0x7ff69c06bb10 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.898+0000 7ff6a27ae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c15f330 con 0x7ff69c0cc760 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.898+0000 7ff69bfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff69c06bb10 0x7ff69c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.898+0000 7ff69bfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff69c06bb10 0x7ff69c0cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:58422/0 (socket says 172.21.15.172:58422) 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.898+0000 7ff69bfff640 1 -- 172.21.15.172:0/2276346657 learned_addr learned my addr 172.21.15.172:0/2276346657 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.898+0000 7ff6a0d4b640 1 --2- 172.21.15.172:0/2276346657 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c0cc760 0x7ff69c158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.899+0000 7ff69bfff640 1 -- 172.21.15.172:0/2276346657 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c0cc760 msgr2=0x7ff69c158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.899+0000 7ff69bfff640 1 --2- 172.21.15.172:0/2276346657 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c0cc760 0x7ff69c158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.899+0000 7ff69bfff640 1 -- 172.21.15.172:0/2276346657 >> v1:172.21.15.174:6789/0 conn(0x7ff69c06bef0 legacy=0x7ff69c0cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.899+0000 7ff69bfff640 1 -- 172.21.15.172:0/2276346657 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c15f5f0 con 0x7ff69c06bb10 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.899+0000 7ff69affd640 1 -- 172.21.15.172:0/2276346657 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310798141 0 0) 0x7ff69c0c5090 con 0x7ff69c06bef0 2024-03-16T19:32:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.899+0000 7ff6a0d4b640 1 --2- 172.21.15.172:0/2276346657 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c0cc760 0x7ff69c158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.899+0000 7ff69bfff640 1 --2- 172.21.15.172:0/2276346657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c06bb10 0x7ff69c0cb370 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009870 tx=0x7ff68c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd76a6107b4941cd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.900+0000 7ff69affd640 1 -- 172.21.15.172:0/2276346657 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c030840 con 0x7ff69c06bb10 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.900+0000 7ff69affd640 1 -- 172.21.15.172:0/2276346657 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c0309a0 con 0x7ff69c06bb10 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.900+0000 7ff69affd640 1 -- 172.21.15.172:0/2276346657 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c030d30 con 0x7ff69c06bb10 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.900+0000 7ff6a27ae640 1 -- 172.21.15.172:0/2276346657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c06bb10 msgr2=0x7ff69c0cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.900+0000 7ff6a27ae640 1 --2- 172.21.15.172:0/2276346657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c06bb10 0x7ff69c0cb370 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009870 tx=0x7ff68c02b980 comp rx=0 tx=0).stop 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.900+0000 7ff6a27ae640 1 -- 172.21.15.172:0/2276346657 shutdown_connections 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.900+0000 7ff6a27ae640 1 --2- 172.21.15.172:0/2276346657 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c0cc760 0x7ff69c158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.900+0000 7ff6a27ae640 1 --2- 172.21.15.172:0/2276346657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c06bb10 0x7ff69c0cb370 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.900+0000 7ff6a27ae640 1 -- 172.21.15.172:0/2276346657 >> 172.21.15.172:0/2276346657 conn(0x7ff69c067250 msgr2=0x7ff69c0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.901+0000 7ff6a27ae640 1 -- 172.21.15.172:0/2276346657 shutdown_connections 2024-03-16T19:32:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.901+0000 7ff6a27ae640 1 -- 172.21.15.172:0/2276346657 wait complete. 2024-03-16T19:32:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.903+0000 7ff6a27ae640 1 Processor -- start 2024-03-16T19:32:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.903+0000 7ff6a27ae640 1 -- start start 2024-03-16T19:32:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff6a27ae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c06bb10 0x7ff69c15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff6a27ae640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff69c06bef0 0x7ff69c15b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff6a27ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c0cc760 0x7ff69c0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff69b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff69c06bef0 0x7ff69c15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff69b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff69c06bef0 0x7ff69c15b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:48536/0 (socket says 172.21.15.172:48536) 2024-03-16T19:32:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff69bfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c06bb10 0x7ff69c15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff6a27ae640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c15ba70 con 0x7ff69c06bef0 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff69bfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c06bb10 0x7ff69c15bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:40834/0 (socket says 172.21.15.172:40834) 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff6a27ae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0b52b0 con 0x7ff69c0cc760 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff69bfff640 1 -- 172.21.15.172:0/1409693870 learned_addr learned my addr 172.21.15.172:0/1409693870 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0b6110 con 0x7ff69c06bb10 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.904+0000 7ff6a0d4b640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c0cc760 0x7ff69c0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.905+0000 7ff6a0d4b640 1 -- 172.21.15.172:0/1409693870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c06bb10 msgr2=0x7ff69c15bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.905+0000 7ff6a0d4b640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c06bb10 0x7ff69c15bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.905+0000 7ff6a0d4b640 1 -- 172.21.15.172:0/1409693870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff69c06bef0 msgr2=0x7ff69c15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.905+0000 7ff6a0d4b640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff69c06bef0 0x7ff69c15b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.905+0000 7ff6a0d4b640 1 -- 172.21.15.172:0/1409693870 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff694009a70 con 0x7ff69c0cc760 2024-03-16T19:32:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.905+0000 7ff6a0d4b640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c0cc760 0x7ff69c0c75f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff694009db0 tx=0x7ff69400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.905+0000 7ff698ff9640 1 -- 172.21.15.172:0/1409693870 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69400cd30 con 0x7ff69c0cc760 2024-03-16T19:32:29.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.905+0000 7ff698ff9640 1 -- 172.21.15.172:0/1409693870 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69400f040 con 0x7ff69c0cc760 2024-03-16T19:32:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.906+0000 7ff698ff9640 1 -- 172.21.15.172:0/1409693870 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694003bd0 con 0x7ff69c0cc760 2024-03-16T19:32:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.906+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68c0094d0 con 0x7ff69c0cc760 2024-03-16T19:32:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.906+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c0b57e0 con 0x7ff69c0cc760 2024-03-16T19:32:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.907+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c0b62a0 con 0x7ff69c0cc760 2024-03-16T19:32:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.908+0000 7ff698ff9640 1 -- 172.21.15.172:0/1409693870 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ff694014480 con 0x7ff69c0cc760 2024-03-16T19:32:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.908+0000 7ff698ff9640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff68806b3b0 0x7ff68806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.908+0000 7ff698ff9640 1 -- 172.21.15.172:0/1409693870 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7ff69408a670 con 0x7ff69c0cc760 2024-03-16T19:32:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.908+0000 7ff69bfff640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff68806b3b0 0x7ff68806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:29.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.909+0000 7ff69bfff640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff68806b3b0 0x7ff68806d870 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff68c02b9c0 tx=0x7ff68c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:29.892 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:29.912+0000 7ff698ff9640 1 -- 172.21.15.172:0/1409693870 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6940552d0 con 0x7ff69c0cc760 2024-03-16T19:32:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.128+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff69c15fbb0 con 0x7ff68806b3b0 2024-03-16T19:32:30.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.134+0000 7ff698ff9640 1 -- 172.21.15.172:0/1409693870 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251370 (secure 0 0 0) 0x7ff68c030640 con 0x7ff68806b3b0 2024-03-16T19:32:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:30.117 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:30.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.137+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff68806b3b0 msgr2=0x7ff68806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:30.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.137+0000 7ff6a27ae640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff68806b3b0 0x7ff68806d870 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff68c02b9c0 tx=0x7ff68c008030 comp rx=0 tx=0).stop 2024-03-16T19:32:30.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.137+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c0cc760 msgr2=0x7ff69c0c75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:30.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.137+0000 7ff6a27ae640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c0cc760 0x7ff69c0c75f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff694009db0 tx=0x7ff69400c490 comp rx=0 tx=0).stop 2024-03-16T19:32:30.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.138+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 shutdown_connections 2024-03-16T19:32:30.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.138+0000 7ff6a27ae640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff69c0cc760 0x7ff69c0c75f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:30.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.138+0000 7ff6a27ae640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff69c06bef0 0x7ff69c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:30.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.138+0000 7ff6a27ae640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ff68806b3b0 0x7ff68806d870 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:30.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.138+0000 7ff6a27ae640 1 --2- 172.21.15.172:0/1409693870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff69c06bb10 0x7ff69c15bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.138+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 >> 172.21.15.172:0/1409693870 conn(0x7ff69c067250 msgr2=0x7ff69c158900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.138+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 shutdown_connections 2024-03-16T19:32:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:30.138+0000 7ff6a27ae640 1 -- 172.21.15.172:0/1409693870 wait complete. 2024-03-16T19:32:30.132 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-03-16T19:32:28.234818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232600,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":221359,"kb_avail":754742120,"statfs":{"total":773094113280,"available":772855930880,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":226672043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,17,0,0,0,0,9],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":173,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002226"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":187,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.702253+0000","last_change":"2024-03-16T19:29:26.360000+0000","last_active":"2024-03-16T19:32:21.702253+0000","last_peered":"2024-03-16T19:32:21.702253+0000","last_clean":"2024-03-16T19:32:21.702253+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:21.702253+0000","last_undegraded":"2024-03-16T19:32:21.702253+0000","last_fullsized":"2024-03-16T19:32:21.702253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:36:59.576733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.696862+0000","last_change":"2024-03-16T19:31:49.979490+0000","last_active":"2024-03-16T19:32:21.696862+0000","last_peered":"2024-03-16T19:32:21.696862+0000","last_clean":"2024-03-16T19:32:21.696862+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:21.696862+0000","last_undegraded":"2024-03-16T19:32:21.696862+0000","last_fullsized":"2024-03-16T19:32:21.696862+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:30.459611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":104,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.697929+0000","last_change":"2024-03-16T19:31:51.808087+0000","last_active":"2024-03-16T19:32:21.697929+0000","last_peered":"2024-03-16T19:32:21.697929+0000","last_clean":"2024-03-16T19:32:21.697929+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:21.697929+0000","last_undegraded":"2024-03-16T19:32:21.697929+0000","last_fullsized":"2024-03-16T19:32:21.697929+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.716598+0000","last_change":"2024-03-16T19:31:26.808712+0000","last_active":"2024-03-16T19:32:21.716598+0000","last_peered":"2024-03-16T19:32:21.716598+0000","last_clean":"2024-03-16T19:32:21.716598+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:21.716598+0000","last_undegraded":"2024-03-16T19:32:21.716598+0000","last_fullsized":"2024-03-16T19:32:21.716598+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:52.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.697626+0000","last_change":"2024-03-16T19:31:49.976859+0000","last_active":"2024-03-16T19:32:21.697626+0000","last_peered":"2024-03-16T19:32:21.697626+0000","last_clean":"2024-03-16T19:32:21.697626+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:21.697626+0000","last_undegraded":"2024-03-16T19:32:21.697626+0000","last_fullsized":"2024-03-16T19:32:21.697626+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:20.196193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.715701+0000","last_change":"2024-03-16T19:31:49.982237+0000","last_active":"2024-03-16T19:32:21.715701+0000","last_peered":"2024-03-16T19:32:21.715701+0000","last_clean":"2024-03-16T19:32:21.715701+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:21.715701+0000","last_undegraded":"2024-03-16T19:32:21.715701+0000","last_fullsized":"2024-03-16T19:32:21.715701+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:48:22.689930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.688120+0000","last_change":"2024-03-16T19:31:49.981160+0000","last_active":"2024-03-16T19:32:21.688120+0000","last_peered":"2024-03-16T19:32:21.688120+0000","last_clean":"2024-03-16T19:32:21.688120+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:21.688120+0000","last_undegraded":"2024-03-16T19:32:21.688120+0000","last_fullsized":"2024-03-16T19:32:21.688120+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:03.493863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.701277+0000","last_change":"2024-03-16T19:31:26.809442+0000","last_active":"2024-03-16T19:32:21.701277+0000","last_peered":"2024-03-16T19:32:21.701277+0000","last_clean":"2024-03-16T19:32:21.701277+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:21.701277+0000","last_undegraded":"2024-03-16T19:32:21.701277+0000","last_fullsized":"2024-03-16T19:32:21.701277+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.713390+0000","last_change":"2024-03-16T19:29:26.363593+0000","last_active":"2024-03-16T19:32:21.713390+0000","last_peered":"2024-03-16T19:32:21.713390+0000","last_clean":"2024-03-16T19:32:21.713390+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:21.713390+0000","last_undegraded":"2024-03-16T19:32:21.713390+0000","last_fullsized":"2024-03-16T19:32:21.713390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:31:56.378629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.716257+0000","last_change":"2024-03-16T19:29:26.366889+0000","last_active":"2024-03-16T19:32:21.716257+0000","last_peered":"2024-03-16T19:32:21.716257+0000","last_clean":"2024-03-16T19:32:21.716257+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:21.716257+0000","last_undegraded":"2024-03-16T19:32:21.716257+0000","last_fullsized":"2024-03-16T19:32:21.716257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:06.168148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.713976+0000","last_change":"2024-03-16T19:29:26.366896+0000","last_active":"2024-03-16T19:32:21.713976+0000","last_peered":"2024-03-16T19:32:21.713976+0000","last_clean":"2024-03-16T19:32:21.713976+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:21.713976+0000","last_undegraded":"2024-03-16T19:32:21.713976+0000","last_fullsized":"2024-03-16T19:32:21.713976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:49.479259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.696150+0000","last_change":"2024-03-16T19:29:26.361540+0000","last_active":"2024-03-16T19:32:21.696150+0000","last_peered":"2024-03-16T19:32:21.696150+0000","last_clean":"2024-03-16T19:32:21.696150+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:21.696150+0000","last_undegraded":"2024-03-16T19:32:21.696150+0000","last_fullsized":"2024-03-16T19:32:21.696150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.697622+0000","last_change":"2024-03-16T19:31:26.814689+0000","last_active":"2024-03-16T19:32:21.697622+0000","last_peered":"2024-03-16T19:32:21.697622+0000","last_clean":"2024-03-16T19:32:21.697622+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:21.697622+0000","last_undegraded":"2024-03-16T19:32:21.697622+0000","last_fullsized":"2024-03-16T19:32:21.697622+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.699264+0000","last_change":"2024-03-16T19:31:51.810173+0000","last_active":"2024-03-16T19:32:21.699264+0000","last_peered":"2024-03-16T19:32:21.699264+0000","last_clean":"2024-03-16T19:32:21.699264+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:21.699264+0000","last_undegraded":"2024-03-16T19:32:21.699264+0000","last_fullsized":"2024-03-16T19:32:21.699264+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.687370+0000","last_change":"2024-03-16T19:29:26.351540+0000","last_active":"2024-03-16T19:32:21.687370+0000","last_peered":"2024-03-16T19:32:21.687370+0000","last_clean":"2024-03-16T19:32:21.687370+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:21.687370+0000","last_undegraded":"2024-03-16T19:32:21.687370+0000","last_fullsized":"2024-03-16T19:32:21.687370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:35:45.641840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.715757+0000","last_change":"2024-03-16T19:31:49.973156+0000","last_active":"2024-03-16T19:32:21.715757+0000","last_peered":"2024-03-16T19:32:21.715757+0000","last_clean":"2024-03-16T19:32:21.715757+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:21.715757+0000","last_undegraded":"2024-03-16T19:32:21.715757+0000","last_fullsized":"2024-03-16T19:32:21.715757+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:11.249529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.713745+0000","last_change":"2024-03-16T19:29:26.359036+0000","last_active":"2024-03-16T19:32:21.713745+0000","last_peered":"2024-03-16T19:32:21.713745+0000","last_clean":"2024-03-16T19:32:21.713745+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:21.713745+0000","last_undegraded":"2024-03-16T19:32:21.713745+0000","last_fullsized":"2024-03-16T19:32:21.713745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:27.996209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.701977+0000","last_change":"2024-03-16T19:31:49.981036+0000","last_active":"2024-03-16T19:32:21.701977+0000","last_peered":"2024-03-16T19:32:21.701977+0000","last_clean":"2024-03-16T19:32:21.701977+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:21.701977+0000","last_undegraded":"2024-03-16T19:32:21.701977+0000","last_fullsized":"2024-03-16T19:32:21.701977+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.699642+0000","last_change":"2024-03-16T19:31:49.976896+0000","last_active":"2024-03-16T19:32:21.699642+0000","last_peered":"2024-03-16T19:32:21.699642+0000","last_clean":"2024-03-16T19:32:21.699642+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:21.699642+0000","last_undegraded":"2024-03-16T19:32:21.699642+0000","last_fullsized":"2024-03-16T19:32:21.699642+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:08:44.705040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.708181+0000","last_change":"2024-03-16T19:29:26.359417+0000","last_active":"2024-03-16T19:32:21.708181+0000","last_peered":"2024-03-16T19:32:21.708181+0000","last_clean":"2024-03-16T19:32:21.708181+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:21.708181+0000","last_undegraded":"2024-03-16T19:32:21.708181+0000","last_fullsized":"2024-03-16T19:32:21.708181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:53.589773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202018+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202018+0000","last_peered":"2024-03-16T19:32:26.202018+0000","last_clean":"2024-03-16T19:32:26.202018+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202018+0000","last_undegraded":"2024-03-16T19:32:26.202018+0000","last_fullsized":"2024-03-16T19:32:26.202018+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.713973+0000","last_change":"2024-03-16T19:31:49.981070+0000","last_active":"2024-03-16T19:32:21.713973+0000","last_peered":"2024-03-16T19:32:21.713973+0000","last_clean":"2024-03-16T19:32:21.713973+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:21.713973+0000","last_undegraded":"2024-03-16T19:32:21.713973+0000","last_fullsized":"2024-03-16T19:32:21.713973+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.701973+0000","last_change":"2024-03-16T19:31:26.809204+0000","last_active":"2024-03-16T19:32:21.701973+0000","last_peered":"2024-03-16T19:32:21.701973+0000","last_clean":"2024-03-16T19:32:21.701973+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:21.701973+0000","last_undegraded":"2024-03-16T19:32:21.701973+0000","last_fullsized":"2024-03-16T19:32:21.701973+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.713185+0000","last_change":"2024-03-16T19:31:26.808824+0000","last_active":"2024-03-16T19:32:21.713185+0000","last_peered":"2024-03-16T19:32:21.713185+0000","last_clean":"2024-03-16T19:32:21.713185+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:21.713185+0000","last_undegraded":"2024-03-16T19:32:21.713185+0000","last_fullsized":"2024-03-16T19:32:21.713185+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:00:24.837559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":163,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.700929+0000","last_change":"2024-03-16T19:31:27.802263+0000","last_active":"2024-03-16T19:32:21.700929+0000","last_peered":"2024-03-16T19:32:21.700929+0000","last_clean":"2024-03-16T19:32:21.700929+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:21.700929+0000","last_undegraded":"2024-03-16T19:32:21.700929+0000","last_fullsized":"2024-03-16T19:32:21.700929+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.713793+0000","last_change":"2024-03-16T19:29:26.363573+0000","last_active":"2024-03-16T19:32:21.713793+0000","last_peered":"2024-03-16T19:32:21.713793+0000","last_clean":"2024-03-16T19:32:21.713793+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:21.713793+0000","last_undegraded":"2024-03-16T19:32:21.713793+0000","last_fullsized":"2024-03-16T19:32:21.713793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:38:17.306453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.698915+0000","last_change":"2024-03-16T19:31:26.812170+0000","last_active":"2024-03-16T19:32:21.698915+0000","last_peered":"2024-03-16T19:32:21.698915+0000","last_clean":"2024-03-16T19:32:21.698915+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:21.698915+0000","last_undegraded":"2024-03-16T19:32:21.698915+0000","last_fullsized":"2024-03-16T19:32:21.698915+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:20.671880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.700323+0000","last_change":"2024-03-16T19:31:26.815049+0000","last_active":"2024-03-16T19:32:21.700323+0000","last_peered":"2024-03-16T19:32:21.700323+0000","last_clean":"2024-03-16T19:32:21.700323+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:21.700323+0000","last_undegraded":"2024-03-16T19:32:21.700323+0000","last_fullsized":"2024-03-16T19:32:21.700323+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.699674+0000","last_change":"2024-03-16T19:31:52.262272+0000","last_active":"2024-03-16T19:32:21.699674+0000","last_peered":"2024-03-16T19:32:21.699674+0000","last_clean":"2024-03-16T19:32:21.699674+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:21.699674+0000","last_undegraded":"2024-03-16T19:32:21.699674+0000","last_fullsized":"2024-03-16T19:32:21.699674+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.700435+0000","last_change":"2024-03-16T19:31:26.813728+0000","last_active":"2024-03-16T19:32:21.700435+0000","last_peered":"2024-03-16T19:32:21.700435+0000","last_clean":"2024-03-16T19:32:21.700435+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:21.700435+0000","last_undegraded":"2024-03-16T19:32:21.700435+0000","last_fullsized":"2024-03-16T19:32:21.700435+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:57:39.745904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216652+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216652+0000","last_peered":"2024-03-16T19:32:27.216652+0000","last_clean":"2024-03-16T19:32:27.216652+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216652+0000","last_undegraded":"2024-03-16T19:32:27.216652+0000","last_fullsized":"2024-03-16T19:32:27.216652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.706815+0000","last_change":"2024-03-16T19:31:49.972154+0000","last_active":"2024-03-16T19:32:21.706815+0000","last_peered":"2024-03-16T19:32:21.706815+0000","last_clean":"2024-03-16T19:32:21.706815+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:21.706815+0000","last_undegraded":"2024-03-16T19:32:21.706815+0000","last_fullsized":"2024-03-16T19:32:21.706815+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:51.987261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002739+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002739+0000","last_peered":"2024-03-16T19:32:28.002739+0000","last_clean":"2024-03-16T19:32:28.002739+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002739+0000","last_undegraded":"2024-03-16T19:32:28.002739+0000","last_fullsized":"2024-03-16T19:32:28.002739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207538+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207538+0000","last_peered":"2024-03-16T19:32:28.207538+0000","last_clean":"2024-03-16T19:32:28.207538+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207538+0000","last_undegraded":"2024-03-16T19:32:28.207538+0000","last_fullsized":"2024-03-16T19:32:28.207538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.700814+0000","last_change":"2024-03-16T19:31:51.812717+0000","last_active":"2024-03-16T19:32:21.700814+0000","last_peered":"2024-03-16T19:32:21.700814+0000","last_clean":"2024-03-16T19:32:21.700814+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:21.700814+0000","last_undegraded":"2024-03-16T19:32:21.700814+0000","last_fullsized":"2024-03-16T19:32:21.700814+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287611+0000","last_change":"2024-03-16T19:29:25.347488+0000","last_active":"2024-03-16T19:32:09.287611+0000","last_peered":"2024-03-16T19:32:09.287611+0000","last_clean":"2024-03-16T19:32:09.287611+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:09.287611+0000","last_undegraded":"2024-03-16T19:32:09.287611+0000","last_fullsized":"2024-03-16T19:32:09.287611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:49.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":68,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086113+0000","last_change":"2024-03-16T19:31:26.810533+0000","last_active":"2024-03-16T19:32:08.086113+0000","last_peered":"2024-03-16T19:32:08.086113+0000","last_clean":"2024-03-16T19:32:08.086113+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:08.086113+0000","last_undegraded":"2024-03-16T19:32:08.086113+0000","last_fullsized":"2024-03-16T19:32:08.086113+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.700423+0000","last_change":"2024-03-16T19:31:49.972302+0000","last_active":"2024-03-16T19:32:21.700423+0000","last_peered":"2024-03-16T19:32:21.700423+0000","last_clean":"2024-03-16T19:32:21.700423+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:21.700423+0000","last_undegraded":"2024-03-16T19:32:21.700423+0000","last_fullsized":"2024-03-16T19:32:21.700423+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083540+0000","last_change":"2024-03-16T19:29:25.338355+0000","last_active":"2024-03-16T19:32:08.083540+0000","last_peered":"2024-03-16T19:32:08.083540+0000","last_clean":"2024-03-16T19:32:08.083540+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:08.083540+0000","last_undegraded":"2024-03-16T19:32:08.083540+0000","last_fullsized":"2024-03-16T19:32:08.083540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:51.862147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.700403+0000","last_change":"2024-03-16T19:31:26.812375+0000","last_active":"2024-03-16T19:32:21.700403+0000","last_peered":"2024-03-16T19:32:21.700403+0000","last_clean":"2024-03-16T19:32:21.700403+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:21.700403+0000","last_undegraded":"2024-03-16T19:32:21.700403+0000","last_fullsized":"2024-03-16T19:32:21.700403+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:35.534853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287793+0000","last_change":"2024-03-16T19:29:25.343797+0000","last_active":"2024-03-16T19:32:09.287793+0000","last_peered":"2024-03-16T19:32:09.287793+0000","last_clean":"2024-03-16T19:32:09.287793+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:09.287793+0000","last_undegraded":"2024-03-16T19:32:09.287793+0000","last_fullsized":"2024-03-16T19:32:09.287793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:14:33.399242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.716244+0000","last_change":"2024-03-16T19:29:26.361502+0000","last_active":"2024-03-16T19:32:21.716244+0000","last_peered":"2024-03-16T19:32:21.716244+0000","last_clean":"2024-03-16T19:32:21.716244+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:21.716244+0000","last_undegraded":"2024-03-16T19:32:21.716244+0000","last_fullsized":"2024-03-16T19:32:21.716244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:33:40.702418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083846+0000","last_change":"2024-03-16T19:29:25.340973+0000","last_active":"2024-03-16T19:32:08.083846+0000","last_peered":"2024-03-16T19:32:08.083846+0000","last_clean":"2024-03-16T19:32:08.083846+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:08.083846+0000","last_undegraded":"2024-03-16T19:32:08.083846+0000","last_fullsized":"2024-03-16T19:32:08.083846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:15:27.712611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.697465+0000","last_change":"2024-03-16T19:31:51.809323+0000","last_active":"2024-03-16T19:32:21.697465+0000","last_peered":"2024-03-16T19:32:21.697465+0000","last_clean":"2024-03-16T19:32:21.697465+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:21.697465+0000","last_undegraded":"2024-03-16T19:32:21.697465+0000","last_fullsized":"2024-03-16T19:32:21.697465+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093565+0000","last_change":"2024-03-16T19:29:25.339488+0000","last_active":"2024-03-16T19:32:08.093565+0000","last_peered":"2024-03-16T19:32:08.093565+0000","last_clean":"2024-03-16T19:32:08.093565+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:08.093565+0000","last_undegraded":"2024-03-16T19:32:08.093565+0000","last_fullsized":"2024-03-16T19:32:08.093565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:35.757647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":22,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.714144+0000","last_change":"2024-03-16T19:32:08.938817+0000","last_active":"2024-03-16T19:32:21.714144+0000","last_peered":"2024-03-16T19:32:21.714144+0000","last_clean":"2024-03-16T19:32:21.714144+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:21.714144+0000","last_undegraded":"2024-03-16T19:32:21.714144+0000","last_fullsized":"2024-03-16T19:32:21.714144+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'25","reported_seq":235,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.669749+0000","last_change":"2024-03-16T19:31:26.831312+0000","last_active":"2024-03-16T19:32:21.669749+0000","last_peered":"2024-03-16T19:32:21.669749+0000","last_clean":"2024-03-16T19:32:21.669749+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:21.669749+0000","last_undegraded":"2024-03-16T19:32:21.669749+0000","last_fullsized":"2024-03-16T19:32:21.669749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:05.605219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.699283+0000","last_change":"2024-03-16T19:31:26.810489+0000","last_active":"2024-03-16T19:32:21.699283+0000","last_peered":"2024-03-16T19:32:21.699283+0000","last_clean":"2024-03-16T19:32:21.699283+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:21.699283+0000","last_undegraded":"2024-03-16T19:32:21.699283+0000","last_fullsized":"2024-03-16T19:32:21.699283+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:24.767545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351613+0000","last_change":"2024-03-16T19:31:51.813002+0000","last_active":"2024-03-16T19:32:08.351613+0000","last_peered":"2024-03-16T19:32:08.351613+0000","last_clean":"2024-03-16T19:32:08.351613+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:08.351613+0000","last_undegraded":"2024-03-16T19:32:08.351613+0000","last_fullsized":"2024-03-16T19:32:08.351613+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197866+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197866+0000","last_peered":"2024-03-16T19:32:27.197866+0000","last_clean":"2024-03-16T19:32:27.197866+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197866+0000","last_undegraded":"2024-03-16T19:32:27.197866+0000","last_fullsized":"2024-03-16T19:32:27.197866+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083847+0000","last_change":"2024-03-16T19:31:26.808770+0000","last_active":"2024-03-16T19:32:08.083847+0000","last_peered":"2024-03-16T19:32:08.083847+0000","last_clean":"2024-03-16T19:32:08.083847+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:08.083847+0000","last_undegraded":"2024-03-16T19:32:08.083847+0000","last_fullsized":"2024-03-16T19:32:08.083847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:15.243777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.705294+0000","last_change":"2024-03-16T19:29:26.360662+0000","last_active":"2024-03-16T19:32:21.705294+0000","last_peered":"2024-03-16T19:32:21.705294+0000","last_clean":"2024-03-16T19:32:21.705294+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:21.705294+0000","last_undegraded":"2024-03-16T19:32:21.705294+0000","last_fullsized":"2024-03-16T19:32:21.705294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:52:45.999568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287610+0000","last_change":"2024-03-16T19:29:25.342530+0000","last_active":"2024-03-16T19:32:09.287610+0000","last_peered":"2024-03-16T19:32:09.287610+0000","last_clean":"2024-03-16T19:32:09.287610+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:09.287610+0000","last_undegraded":"2024-03-16T19:32:09.287610+0000","last_fullsized":"2024-03-16T19:32:09.287610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:54:39.963327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.706192+0000","last_change":"2024-03-16T19:31:49.971850+0000","last_active":"2024-03-16T19:32:21.706192+0000","last_peered":"2024-03-16T19:32:21.706192+0000","last_clean":"2024-03-16T19:32:21.706192+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:21.706192+0000","last_undegraded":"2024-03-16T19:32:21.706192+0000","last_fullsized":"2024-03-16T19:32:21.706192+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:34.834168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":59,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087418+0000","last_change":"2024-03-16T19:31:49.975757+0000","last_active":"2024-03-16T19:32:08.087418+0000","last_peered":"2024-03-16T19:32:08.087418+0000","last_clean":"2024-03-16T19:32:08.087418+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:08.087418+0000","last_undegraded":"2024-03-16T19:32:08.087418+0000","last_fullsized":"2024-03-16T19:32:08.087418+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.704682+0000","last_change":"2024-03-16T19:29:26.361122+0000","last_active":"2024-03-16T19:32:21.704682+0000","last_peered":"2024-03-16T19:32:21.704682+0000","last_clean":"2024-03-16T19:32:21.704682+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:21.704682+0000","last_undegraded":"2024-03-16T19:32:21.704682+0000","last_fullsized":"2024-03-16T19:32:21.704682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:06.832667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.084364+0000","last_change":"2024-03-16T19:31:49.989827+0000","last_active":"2024-03-16T19:32:08.084364+0000","last_peered":"2024-03-16T19:32:08.084364+0000","last_clean":"2024-03-16T19:32:08.084364+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:08.084364+0000","last_undegraded":"2024-03-16T19:32:08.084364+0000","last_fullsized":"2024-03-16T19:32:08.084364+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.704408+0000","last_change":"2024-03-16T19:29:26.361386+0000","last_active":"2024-03-16T19:32:21.704408+0000","last_peered":"2024-03-16T19:32:21.704408+0000","last_clean":"2024-03-16T19:32:21.704408+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:21.704408+0000","last_undegraded":"2024-03-16T19:32:21.704408+0000","last_fullsized":"2024-03-16T19:32:21.704408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:24:44.054403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":56,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086967+0000","last_change":"2024-03-16T19:31:50.985467+0000","last_active":"2024-03-16T19:32:08.086967+0000","last_peered":"2024-03-16T19:32:08.086967+0000","last_clean":"2024-03-16T19:32:08.086967+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:08.086967+0000","last_undegraded":"2024-03-16T19:32:08.086967+0000","last_fullsized":"2024-03-16T19:32:08.086967+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.708334+0000","last_change":"2024-03-16T19:29:26.362530+0000","last_active":"2024-03-16T19:32:21.708334+0000","last_peered":"2024-03-16T19:32:21.708334+0000","last_clean":"2024-03-16T19:32:21.708334+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:21.708334+0000","last_undegraded":"2024-03-16T19:32:21.708334+0000","last_fullsized":"2024-03-16T19:32:21.708334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:34:08.468508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093703+0000","last_change":"2024-03-16T19:29:25.346664+0000","last_active":"2024-03-16T19:32:08.093703+0000","last_peered":"2024-03-16T19:32:08.093703+0000","last_clean":"2024-03-16T19:32:08.093703+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:08.093703+0000","last_undegraded":"2024-03-16T19:32:08.093703+0000","last_fullsized":"2024-03-16T19:32:08.093703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:37.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.713069+0000","last_change":"2024-03-16T19:31:26.808782+0000","last_active":"2024-03-16T19:32:21.713069+0000","last_peered":"2024-03-16T19:32:21.713069+0000","last_clean":"2024-03-16T19:32:21.713069+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:21.713069+0000","last_undegraded":"2024-03-16T19:32:21.713069+0000","last_fullsized":"2024-03-16T19:32:21.713069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:22:39.405302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.086718+0000","last_change":"2024-03-16T19:31:49.974620+0000","last_active":"2024-03-16T19:32:08.086718+0000","last_peered":"2024-03-16T19:32:08.086718+0000","last_clean":"2024-03-16T19:32:08.086718+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:08.086718+0000","last_undegraded":"2024-03-16T19:32:08.086718+0000","last_fullsized":"2024-03-16T19:32:08.086718+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":156,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:22.363509+0000","last_change":"2024-03-16T19:30:13.514454+0000","last_active":"2024-03-16T19:32:22.363509+0000","last_peered":"2024-03-16T19:32:22.363509+0000","last_clean":"2024-03-16T19:32:22.363509+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:22.363509+0000","last_undegraded":"2024-03-16T19:32:22.363509+0000","last_fullsized":"2024-03-16T19:32:22.363509+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:49.068021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":363,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937413+0000","last_change":"2024-03-16T19:31:52.807808+0000","last_active":"2024-03-16T19:32:08.937413+0000","last_peered":"2024-03-16T19:32:08.937413+0000","last_clean":"2024-03-16T19:32:08.937413+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:08.937413+0000","last_undegraded":"2024-03-16T19:32:08.937413+0000","last_fullsized":"2024-03-16T19:32:08.937413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:15.221658+0000","last_clean_scrub_stamp":"2024-03-16T19:29:15.221658+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:41:38.789143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083631+0000","last_change":"2024-03-16T19:31:49.975286+0000","last_active":"2024-03-16T19:32:08.083631+0000","last_peered":"2024-03-16T19:32:08.083631+0000","last_clean":"2024-03-16T19:32:08.083631+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:08.083631+0000","last_undegraded":"2024-03-16T19:32:08.083631+0000","last_fullsized":"2024-03-16T19:32:08.083631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:24:05.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.701683+0000","last_change":"2024-03-16T19:31:26.809120+0000","last_active":"2024-03-16T19:32:21.701683+0000","last_peered":"2024-03-16T19:32:21.701683+0000","last_clean":"2024-03-16T19:32:21.701683+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:21.701683+0000","last_undegraded":"2024-03-16T19:32:21.701683+0000","last_fullsized":"2024-03-16T19:32:21.701683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937520+0000","last_change":"2024-03-16T19:29:25.348087+0000","last_active":"2024-03-16T19:32:08.937520+0000","last_peered":"2024-03-16T19:32:08.937520+0000","last_clean":"2024-03-16T19:32:08.937520+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:08.937520+0000","last_undegraded":"2024-03-16T19:32:08.937520+0000","last_fullsized":"2024-03-16T19:32:08.937520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:35:30.718328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.700598+0000","last_change":"2024-03-16T19:31:51.812184+0000","last_active":"2024-03-16T19:32:21.700598+0000","last_peered":"2024-03-16T19:32:21.700598+0000","last_clean":"2024-03-16T19:32:21.700598+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:21.700598+0000","last_undegraded":"2024-03-16T19:32:21.700598+0000","last_fullsized":"2024-03-16T19:32:21.700598+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"84'2","reported_seq":53,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.087177+0000","last_change":"2024-03-16T19:31:50.991729+0000","last_active":"2024-03-16T19:32:08.087177+0000","last_peered":"2024-03-16T19:32:08.087177+0000","last_clean":"2024-03-16T19:32:08.087177+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:08.087177+0000","last_undegraded":"2024-03-16T19:32:08.087177+0000","last_fullsized":"2024-03-16T19:32:08.087177+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.096035+0000","last_change":"2024-03-16T19:31:49.976472+0000","last_active":"2024-03-16T19:32:08.096035+0000","last_peered":"2024-03-16T19:32:08.096035+0000","last_clean":"2024-03-16T19:32:08.096035+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:08.096035+0000","last_undegraded":"2024-03-16T19:32:08.096035+0000","last_fullsized":"2024-03-16T19:32:08.096035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:05:36.990107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.083939+0000","last_change":"2024-03-16T19:29:25.341760+0000","last_active":"2024-03-16T19:32:08.083939+0000","last_peered":"2024-03-16T19:32:08.083939+0000","last_clean":"2024-03-16T19:32:08.083939+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:08.083939+0000","last_undegraded":"2024-03-16T19:32:08.083939+0000","last_fullsized":"2024-03-16T19:32:08.083939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:13.495989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.691966+0000","last_change":"2024-03-16T19:31:49.980933+0000","last_active":"2024-03-16T19:32:21.691966+0000","last_peered":"2024-03-16T19:32:21.691966+0000","last_clean":"2024-03-16T19:32:21.691966+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:21.691966+0000","last_undegraded":"2024-03-16T19:32:21.691966+0000","last_fullsized":"2024-03-16T19:32:21.691966+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093299+0000","last_change":"2024-03-16T19:31:49.979956+0000","last_active":"2024-03-16T19:32:08.093299+0000","last_peered":"2024-03-16T19:32:08.093299+0000","last_clean":"2024-03-16T19:32:08.093299+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:08.093299+0000","last_undegraded":"2024-03-16T19:32:08.093299+0000","last_fullsized":"2024-03-16T19:32:08.093299+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:33:18.614773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.715753+0000","last_change":"2024-03-16T19:31:26.808959+0000","last_active":"2024-03-16T19:32:21.715753+0000","last_peered":"2024-03-16T19:32:21.715753+0000","last_clean":"2024-03-16T19:32:21.715753+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:21.715753+0000","last_undegraded":"2024-03-16T19:32:21.715753+0000","last_fullsized":"2024-03-16T19:32:21.715753+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:47:13.119788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937906+0000","last_change":"2024-03-16T19:31:49.982381+0000","last_active":"2024-03-16T19:32:08.937906+0000","last_peered":"2024-03-16T19:32:08.937906+0000","last_clean":"2024-03-16T19:32:08.937906+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:08.937906+0000","last_undegraded":"2024-03-16T19:32:08.937906+0000","last_fullsized":"2024-03-16T19:32:08.937906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:44.105214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.093171+0000","last_change":"2024-03-16T19:31:49.975789+0000","last_active":"2024-03-16T19:32:08.093171+0000","last_peered":"2024-03-16T19:32:08.093171+0000","last_clean":"2024-03-16T19:32:08.093171+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:08.093171+0000","last_undegraded":"2024-03-16T19:32:08.093171+0000","last_fullsized":"2024-03-16T19:32:08.093171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:23.128822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094697+0000","last_change":"2024-03-16T19:29:25.349636+0000","last_active":"2024-03-16T19:32:08.094697+0000","last_peered":"2024-03-16T19:32:08.094697+0000","last_clean":"2024-03-16T19:32:08.094697+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:08.094697+0000","last_undegraded":"2024-03-16T19:32:08.094697+0000","last_fullsized":"2024-03-16T19:32:08.094697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:50:01.702964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.703305+0000","last_change":"2024-03-16T19:31:26.810543+0000","last_active":"2024-03-16T19:32:21.703305+0000","last_peered":"2024-03-16T19:32:21.703305+0000","last_clean":"2024-03-16T19:32:21.703305+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:21.703305+0000","last_undegraded":"2024-03-16T19:32:21.703305+0000","last_fullsized":"2024-03-16T19:32:21.703305+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'3","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:09.287797+0000","last_change":"2024-03-16T19:29:25.347085+0000","last_active":"2024-03-16T19:32:09.287797+0000","last_peered":"2024-03-16T19:32:09.287797+0000","last_clean":"2024-03-16T19:32:09.287797+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:09.287797+0000","last_undegraded":"2024-03-16T19:32:09.287797+0000","last_fullsized":"2024-03-16T19:32:09.287797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:05:40.859560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.699754+0000","last_change":"2024-03-16T19:31:51.810179+0000","last_active":"2024-03-16T19:32:21.699754+0000","last_peered":"2024-03-16T19:32:21.699754+0000","last_clean":"2024-03-16T19:32:21.699754+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:21.699754+0000","last_undegraded":"2024-03-16T19:32:21.699754+0000","last_fullsized":"2024-03-16T19:32:21.699754+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937470+0000","last_change":"2024-03-16T19:31:49.980944+0000","last_active":"2024-03-16T19:32:08.937470+0000","last_peered":"2024-03-16T19:32:08.937470+0000","last_clean":"2024-03-16T19:32:08.937470+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:08.937470+0000","last_undegraded":"2024-03-16T19:32:08.937470+0000","last_fullsized":"2024-03-16T19:32:08.937470+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:53.500473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.703080+0000","last_change":"2024-03-16T19:31:49.974099+0000","last_active":"2024-03-16T19:32:21.703080+0000","last_peered":"2024-03-16T19:32:21.703080+0000","last_clean":"2024-03-16T19:32:21.703080+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:21.703080+0000","last_undegraded":"2024-03-16T19:32:21.703080+0000","last_fullsized":"2024-03-16T19:32:21.703080+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.094246+0000","last_change":"2024-03-16T19:29:25.346786+0000","last_active":"2024-03-16T19:32:08.094246+0000","last_peered":"2024-03-16T19:32:08.094246+0000","last_clean":"2024-03-16T19:32:08.094246+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:08.094246+0000","last_undegraded":"2024-03-16T19:32:08.094246+0000","last_fullsized":"2024-03-16T19:32:08.094246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:52.757346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":106,"state":"active+clean+laggy","last_fresh":"2024-03-16T19:32:21.688916+0000","last_change":"2024-03-16T19:32:21.688916+0000","last_active":"2024-03-16T19:32:21.688916+0000","last_peered":"2024-03-16T19:32:21.688916+0000","last_clean":"2024-03-16T19:32:21.688916+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:32:21.688916+0000","last_undegraded":"2024-03-16T19:32:21.688916+0000","last_fullsized":"2024-03-16T19:32:21.688916+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.095090+0000","last_change":"2024-03-16T19:31:26.815051+0000","last_active":"2024-03-16T19:32:08.095090+0000","last_peered":"2024-03-16T19:32:08.095090+0000","last_clean":"2024-03-16T19:32:08.095090+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:08.095090+0000","last_undegraded":"2024-03-16T19:32:08.095090+0000","last_fullsized":"2024-03-16T19:32:08.095090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:32.726000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.699285+0000","last_change":"2024-03-16T19:31:49.984725+0000","last_active":"2024-03-16T19:32:21.699285+0000","last_peered":"2024-03-16T19:32:21.699285+0000","last_clean":"2024-03-16T19:32:21.699285+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:21.699285+0000","last_undegraded":"2024-03-16T19:32:21.699285+0000","last_fullsized":"2024-03-16T19:32:21.699285+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937781+0000","last_change":"2024-03-16T19:29:25.348066+0000","last_active":"2024-03-16T19:32:08.937781+0000","last_peered":"2024-03-16T19:32:08.937781+0000","last_clean":"2024-03-16T19:32:08.937781+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:08.937781+0000","last_undegraded":"2024-03-16T19:32:08.937781+0000","last_fullsized":"2024-03-16T19:32:08.937781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:14:04.840279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.708154+0000","last_change":"2024-03-16T19:29:26.359275+0000","last_active":"2024-03-16T19:32:21.708154+0000","last_peered":"2024-03-16T19:32:21.708154+0000","last_clean":"2024-03-16T19:32:21.708154+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:21.708154+0000","last_undegraded":"2024-03-16T19:32:21.708154+0000","last_fullsized":"2024-03-16T19:32:21.708154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:59:27.951870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.716771+0000","last_change":"2024-03-16T19:29:26.355500+0000","last_active":"2024-03-16T19:32:21.716771+0000","last_peered":"2024-03-16T19:32:21.716771+0000","last_clean":"2024-03-16T19:32:21.716771+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:21.716771+0000","last_undegraded":"2024-03-16T19:32:21.716771+0000","last_fullsized":"2024-03-16T19:32:21.716771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:54:05.970283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937766+0000","last_change":"2024-03-16T19:29:25.346723+0000","last_active":"2024-03-16T19:32:08.937766+0000","last_peered":"2024-03-16T19:32:08.937766+0000","last_clean":"2024-03-16T19:32:08.937766+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:08.937766+0000","last_undegraded":"2024-03-16T19:32:08.937766+0000","last_fullsized":"2024-03-16T19:32:08.937766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:33:24.769755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.716530+0000","last_change":"2024-03-16T19:29:26.357878+0000","last_active":"2024-03-16T19:32:21.716530+0000","last_peered":"2024-03-16T19:32:21.716530+0000","last_clean":"2024-03-16T19:32:21.716530+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:21.716530+0000","last_undegraded":"2024-03-16T19:32:21.716530+0000","last_fullsized":"2024-03-16T19:32:21.716530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:23:55.701697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"84'3","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.937841+0000","last_change":"2024-03-16T19:29:25.349867+0000","last_active":"2024-03-16T19:32:08.937841+0000","last_peered":"2024-03-16T19:32:08.937841+0000","last_clean":"2024-03-16T19:32:08.937841+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:08.937841+0000","last_undegraded":"2024-03-16T19:32:08.937841+0000","last_fullsized":"2024-03-16T19:32:08.937841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:18:23.893112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":171,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.699872+0000","last_change":"2024-03-16T19:31:51.810498+0000","last_active":"2024-03-16T19:32:21.699872+0000","last_peered":"2024-03-16T19:32:21.699872+0000","last_clean":"2024-03-16T19:32:21.699872+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:21.699872+0000","last_undegraded":"2024-03-16T19:32:21.699872+0000","last_fullsized":"2024-03-16T19:32:21.699872+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.088522+0000","last_change":"2024-03-16T19:31:49.984782+0000","last_active":"2024-03-16T19:32:08.088522+0000","last_peered":"2024-03-16T19:32:08.088522+0000","last_clean":"2024-03-16T19:32:08.088522+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:08.088522+0000","last_undegraded":"2024-03-16T19:32:08.088522+0000","last_fullsized":"2024-03-16T19:32:08.088522+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":166,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:21.697545+0000","last_change":"2024-03-16T19:31:27.859066+0000","last_active":"2024-03-16T19:32:21.697545+0000","last_peered":"2024-03-16T19:32:21.697545+0000","last_clean":"2024-03-16T19:32:21.697545+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:21.697545+0000","last_undegraded":"2024-03-16T19:32:21.697545+0000","last_fullsized":"2024-03-16T19:32:21.697545+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:08.351546+0000","last_change":"2024-03-16T19:31:51.809145+0000","last_active":"2024-03-16T19:32:08.351546+0000","last_peered":"2024-03-16T19:32:08.351546+0000","last_clean":"2024-03-16T19:32:08.351546+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:08.351546+0000","last_undegraded":"2024-03-16T19:32:08.351546+0000","last_fullsized":"2024-03-16T19:32:08.351546+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":477,"ondisk_log_size":477,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28832,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343008,"statfs":{"total":96636764160,"available":96607240192,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122058,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":8192},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:30.139 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.139 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.139 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.139 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.139 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.139 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.139 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.139 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.139 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.140 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.140 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.140 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.140 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.140 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.140 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.140 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.140 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.140 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.141 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.142 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.142 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.142 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.142 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.142 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.142 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.142 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.142 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.143 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.143 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.143 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.143 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.143 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.143 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.143 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.143 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.143 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.144 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-16T19:29:15.221658+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=15, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.145 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.146 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.147 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.148 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.148 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.148 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.148 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.148 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:30.148 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:32:50.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:32:50.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.315+0000 7f85127e2640 1 Processor -- start 2024-03-16T19:32:50.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.315+0000 7f85127e2640 1 -- start start 2024-03-16T19:32:50.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.315+0000 7f85127e2640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f850c151b00 0x7f850c0c6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:50.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f85127e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c0c3d20 0x7f850c0c6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:50.297 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f85127e2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f850c158240 0x7f850c0c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f85127e2640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0c4a40 con 0x7f850c151b00 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f85127e2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0c73e0 con 0x7f850c158240 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f85127e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0c6040 con 0x7f850c0c3d20 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f850bfff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f850c151b00 0x7f850c0c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f850bfff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f850c151b00 0x7f850c0c6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:38742/0 (socket says 172.21.15.172:38742) 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f850bfff640 1 -- 172.21.15.172:0/551418107 learned_addr learned my addr 172.21.15.172:0/551418107 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f850b7fe640 1 --2- 172.21.15.172:0/551418107 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c0c3d20 0x7f850c0c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f8510d7f640 1 --2- 172.21.15.172:0/551418107 >> v2:172.21.15.174:3300/0 conn(0x7f850c158240 0x7f850c0c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f850bfff640 1 -- 172.21.15.172:0/551418107 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c0c3d20 msgr2=0x7f850c0c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f850bfff640 1 --2- 172.21.15.172:0/551418107 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c0c3d20 0x7f850c0c6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f850bfff640 1 -- 172.21.15.172:0/551418107 >> v2:172.21.15.174:3300/0 conn(0x7f850c158240 msgr2=0x7f850c0c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:32:50.298 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f850bfff640 1 --2- 172.21.15.172:0/551418107 >> v2:172.21.15.174:3300/0 conn(0x7f850c158240 0x7f850c0c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.299 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.316+0000 7f850bfff640 1 -- 172.21.15.172:0/551418107 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0c57a0 con 0x7f850c151b00 2024-03-16T19:32:50.299 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.317+0000 7f850bfff640 1 --2- 172.21.15.172:0/551418107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c151b00 0x7f850c0c6440 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f850c06ca40 tx=0x7f84fc02db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51390217a6c28d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:50.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.317+0000 7f850affd640 1 -- 172.21.15.172:0/551418107 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc02f8e0 con 0x7f850c151b00 2024-03-16T19:32:50.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.317+0000 7f850affd640 1 -- 172.21.15.172:0/551418107 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc02fa40 con 0x7f850c151b00 2024-03-16T19:32:50.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.317+0000 7f850affd640 1 -- 172.21.15.172:0/551418107 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc02fdd0 con 0x7f850c151b00 2024-03-16T19:32:50.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.318+0000 7f85127e2640 1 -- 172.21.15.172:0/551418107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c151b00 msgr2=0x7f850c0c6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:50.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.318+0000 7f85127e2640 1 --2- 172.21.15.172:0/551418107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c151b00 0x7f850c0c6440 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f850c06ca40 tx=0x7f84fc02db20 comp rx=0 tx=0).stop 2024-03-16T19:32:50.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.318+0000 7f85127e2640 1 -- 172.21.15.172:0/551418107 shutdown_connections 2024-03-16T19:32:50.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.318+0000 7f85127e2640 1 --2- 172.21.15.172:0/551418107 >> v2:172.21.15.174:3300/0 conn(0x7f850c158240 0x7f850c0c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.318+0000 7f85127e2640 1 --2- 172.21.15.172:0/551418107 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c0c3d20 0x7f850c0c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.300 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.318+0000 7f85127e2640 1 --2- 172.21.15.172:0/551418107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c151b00 0x7f850c0c6440 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.318+0000 7f85127e2640 1 -- 172.21.15.172:0/551418107 >> 172.21.15.172:0/551418107 conn(0x7f850c0cc820 msgr2=0x7f850c0ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:50.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.318+0000 7f85127e2640 1 -- 172.21.15.172:0/551418107 shutdown_connections 2024-03-16T19:32:50.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.318+0000 7f85127e2640 1 -- 172.21.15.172:0/551418107 wait complete. 2024-03-16T19:32:50.301 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.320+0000 7f85127e2640 1 Processor -- start 2024-03-16T19:32:50.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.320+0000 7f85127e2640 1 -- start start 2024-03-16T19:32:50.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f85127e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f850c0c3d20 0x7f850c0724e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:50.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f85127e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c151b00 0x7f850c1ec380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:50.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f85127e2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c158240 0x7f850c1545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:50.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f85127e2640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c147cc0 con 0x7f850c158240 2024-03-16T19:32:50.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f850b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c151b00 0x7f850c1ec380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:50.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f850b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c151b00 0x7f850c1ec380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48450/0 (socket says 172.21.15.172:48450) 2024-03-16T19:32:50.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f850b7fe640 1 -- 172.21.15.172:0/2297443463 learned_addr learned my addr 172.21.15.172:0/2297443463 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f850bfff640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f850c0c3d20 0x7f850c0724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f85127e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c149db0 con 0x7f850c0c3d20 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1ebc50 con 0x7f850c151b00 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f8510d7f640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c158240 0x7f850c1545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f850b7fe640 1 -- 172.21.15.172:0/2297443463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f850c0c3d20 msgr2=0x7f850c0724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f850b7fe640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f850c0c3d20 0x7f850c0724e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f850b7fe640 1 -- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c158240 msgr2=0x7f850c1545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f850b7fe640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c158240 0x7f850c1545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.321+0000 7f850b7fe640 1 -- 172.21.15.172:0/2297443463 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc0094d0 con 0x7f850c151b00 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.322+0000 7f850b7fe640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c151b00 0x7f850c1ec380 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f850000e860 tx=0x7f850000ed20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.322+0000 7f8508ff9640 1 -- 172.21.15.172:0/2297443463 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85000095f0 con 0x7f850c151b00 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.322+0000 7f8508ff9640 1 -- 172.21.15.172:0/2297443463 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850000c940 con 0x7f850c151b00 2024-03-16T19:32:50.304 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.323+0000 7f8508ff9640 1 -- 172.21.15.172:0/2297443463 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850000ccd0 con 0x7f850c151b00 2024-03-16T19:32:50.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.323+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c1ebef0 con 0x7f850c151b00 2024-03-16T19:32:50.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.323+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c156790 con 0x7f850c151b00 2024-03-16T19:32:50.305 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.324+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c1ec080 con 0x7f850c151b00 2024-03-16T19:32:50.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.324+0000 7f8508ff9640 1 -- 172.21.15.172:0/2297443463 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8500013470 con 0x7f850c151b00 2024-03-16T19:32:50.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.324+0000 7f8508ff9640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f84f806b3e0 0x7f84f806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:32:50.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.324+0000 7f8508ff9640 1 -- 172.21.15.172:0/2297443463 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f8500089ed0 con 0x7f850c151b00 2024-03-16T19:32:50.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.325+0000 7f850bfff640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f84f806b3e0 0x7f84f806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:32:50.306 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.325+0000 7f850bfff640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f84f806b3e0 0x7f84f806d8a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f84fc038000 tx=0x7f84fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:32:50.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.329+0000 7f8508ff9640 1 -- 172.21.15.172:0/2297443463 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8500054b30 con 0x7f850c151b00 2024-03-16T19:32:50.556 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.576+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f850c0c5d60 con 0x7f84f806b3e0 2024-03-16T19:32:50.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.587+0000 7f8508ff9640 1 -- 172.21.15.172:0/2297443463 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252901 (secure 0 0 0) 0x7f84fc02fbb0 con 0x7f84f806b3e0 2024-03-16T19:32:50.570 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:32:50.572 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:32:50.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f84f806b3e0 msgr2=0x7f84f806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:50.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f84f806b3e0 0x7f84f806d8a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f84fc038000 tx=0x7f84fc008030 comp rx=0 tx=0).stop 2024-03-16T19:32:50.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c151b00 msgr2=0x7f850c1ec380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:32:50.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c151b00 0x7f850c1ec380 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f850000e860 tx=0x7f850000ed20 comp rx=0 tx=0).stop 2024-03-16T19:32:50.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 shutdown_connections 2024-03-16T19:32:50.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f850c158240 0x7f850c1545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f84f806b3e0 0x7f84f806d8a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f850c151b00 0x7f850c1ec380 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.573 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 --2- 172.21.15.172:0/2297443463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f850c0c3d20 0x7f850c0724e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:32:50.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 >> 172.21.15.172:0/2297443463 conn(0x7f850c0cc820 msgr2=0x7f850c0c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:32:50.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 shutdown_connections 2024-03-16T19:32:50.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:32:50.592+0000 7f85127e2640 1 -- 172.21.15.172:0/2297443463 wait complete. 2024-03-16T19:32:50.585 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-03-16T19:32:50.239232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":65536},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002403"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.221290+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:32:36.221290+0000","last_peered":"2024-03-16T19:32:36.221290+0000","last_clean":"2024-03-16T19:32:36.221290+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:36.221290+0000","last_undegraded":"2024-03-16T19:32:36.221290+0000","last_fullsized":"2024-03-16T19:32:36.221290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.681904+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:32:36.681904+0000","last_peered":"2024-03-16T19:32:36.681904+0000","last_clean":"2024-03-16T19:32:36.681904+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:36.681904+0000","last_undegraded":"2024-03-16T19:32:36.681904+0000","last_fullsized":"2024-03-16T19:32:36.681904+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":112,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.236899+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:32:28.236899+0000","last_peered":"2024-03-16T19:32:28.236899+0000","last_clean":"2024-03-16T19:32:28.236899+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:28.236899+0000","last_undegraded":"2024-03-16T19:32:28.236899+0000","last_fullsized":"2024-03-16T19:32:28.236899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.687101+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:32:35.687101+0000","last_peered":"2024-03-16T19:32:35.687101+0000","last_clean":"2024-03-16T19:32:35.687101+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:35.687101+0000","last_undegraded":"2024-03-16T19:32:35.687101+0000","last_fullsized":"2024-03-16T19:32:35.687101+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193158+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193158+0000","last_peered":"2024-03-16T19:32:42.193158+0000","last_clean":"2024-03-16T19:32:42.193158+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193158+0000","last_undegraded":"2024-03-16T19:32:42.193158+0000","last_fullsized":"2024-03-16T19:32:42.193158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501727+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501727+0000","last_peered":"2024-03-16T19:32:43.501727+0000","last_clean":"2024-03-16T19:32:43.501727+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501727+0000","last_undegraded":"2024-03-16T19:32:43.501727+0000","last_fullsized":"2024-03-16T19:32:43.501727+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180894+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180894+0000","last_peered":"2024-03-16T19:32:45.180894+0000","last_clean":"2024-03-16T19:32:45.180894+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180894+0000","last_undegraded":"2024-03-16T19:32:45.180894+0000","last_fullsized":"2024-03-16T19:32:45.180894+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142579+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142579+0000","last_peered":"2024-03-16T19:32:31.142579+0000","last_clean":"2024-03-16T19:32:31.142579+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142579+0000","last_undegraded":"2024-03-16T19:32:31.142579+0000","last_fullsized":"2024-03-16T19:32:31.142579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050015+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050015+0000","last_peered":"2024-03-16T19:32:30.050015+0000","last_clean":"2024-03-16T19:32:30.050015+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050015+0000","last_undegraded":"2024-03-16T19:32:30.050015+0000","last_fullsized":"2024-03-16T19:32:30.050015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906468+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906468+0000","last_peered":"2024-03-16T19:32:36.906468+0000","last_clean":"2024-03-16T19:32:36.906468+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906468+0000","last_undegraded":"2024-03-16T19:32:36.906468+0000","last_fullsized":"2024-03-16T19:32:36.906468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946447+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946447+0000","last_peered":"2024-03-16T19:32:37.946447+0000","last_clean":"2024-03-16T19:32:37.946447+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946447+0000","last_undegraded":"2024-03-16T19:32:37.946447+0000","last_fullsized":"2024-03-16T19:32:37.946447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012343+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012343+0000","last_peered":"2024-03-16T19:32:29.012343+0000","last_clean":"2024-03-16T19:32:29.012343+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012343+0000","last_undegraded":"2024-03-16T19:32:29.012343+0000","last_fullsized":"2024-03-16T19:32:29.012343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085947+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085947+0000","last_peered":"2024-03-16T19:32:34.085947+0000","last_clean":"2024-03-16T19:32:34.085947+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085947+0000","last_undegraded":"2024-03-16T19:32:34.085947+0000","last_fullsized":"2024-03-16T19:32:34.085947+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194777+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194777+0000","last_peered":"2024-03-16T19:32:31.194777+0000","last_clean":"2024-03-16T19:32:31.194777+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194777+0000","last_undegraded":"2024-03-16T19:32:31.194777+0000","last_fullsized":"2024-03-16T19:32:31.194777+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622130+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622130+0000","last_peered":"2024-03-16T19:32:30.622130+0000","last_clean":"2024-03-16T19:32:30.622130+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622130+0000","last_undegraded":"2024-03-16T19:32:30.622130+0000","last_fullsized":"2024-03-16T19:32:30.622130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.527986+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.527986+0000","last_peered":"2024-03-16T19:32:41.527986+0000","last_clean":"2024-03-16T19:32:41.527986+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.527986+0000","last_undegraded":"2024-03-16T19:32:41.527986+0000","last_fullsized":"2024-03-16T19:32:41.527986+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668862+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668862+0000","last_peered":"2024-03-16T19:32:33.668862+0000","last_clean":"2024-03-16T19:32:33.668862+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668862+0000","last_undegraded":"2024-03-16T19:32:33.668862+0000","last_fullsized":"2024-03-16T19:32:33.668862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":70,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158073+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158073+0000","last_peered":"2024-03-16T19:32:30.158073+0000","last_clean":"2024-03-16T19:32:30.158073+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158073+0000","last_undegraded":"2024-03-16T19:32:30.158073+0000","last_fullsized":"2024-03-16T19:32:30.158073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198963+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198963+0000","last_peered":"2024-03-16T19:32:43.198963+0000","last_clean":"2024-03-16T19:32:43.198963+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198963+0000","last_undegraded":"2024-03-16T19:32:43.198963+0000","last_fullsized":"2024-03-16T19:32:43.198963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874908+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874908+0000","last_peered":"2024-03-16T19:32:34.874908+0000","last_clean":"2024-03-16T19:32:34.874908+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874908+0000","last_undegraded":"2024-03-16T19:32:34.874908+0000","last_fullsized":"2024-03-16T19:32:34.874908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202018+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202018+0000","last_peered":"2024-03-16T19:32:26.202018+0000","last_clean":"2024-03-16T19:32:26.202018+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202018+0000","last_undegraded":"2024-03-16T19:32:26.202018+0000","last_fullsized":"2024-03-16T19:32:26.202018+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113786+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113786+0000","last_peered":"2024-03-16T19:32:31.113786+0000","last_clean":"2024-03-16T19:32:31.113786+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113786+0000","last_undegraded":"2024-03-16T19:32:31.113786+0000","last_fullsized":"2024-03-16T19:32:31.113786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189258+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189258+0000","last_peered":"2024-03-16T19:32:30.189258+0000","last_clean":"2024-03-16T19:32:30.189258+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189258+0000","last_undegraded":"2024-03-16T19:32:30.189258+0000","last_fullsized":"2024-03-16T19:32:30.189258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633685+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633685+0000","last_peered":"2024-03-16T19:32:38.633685+0000","last_clean":"2024-03-16T19:32:38.633685+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633685+0000","last_undegraded":"2024-03-16T19:32:38.633685+0000","last_fullsized":"2024-03-16T19:32:38.633685+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":171,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030001+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030001+0000","last_peered":"2024-03-16T19:32:36.030001+0000","last_clean":"2024-03-16T19:32:36.030001+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030001+0000","last_undegraded":"2024-03-16T19:32:36.030001+0000","last_fullsized":"2024-03-16T19:32:36.030001+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881947+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881947+0000","last_peered":"2024-03-16T19:32:35.881947+0000","last_clean":"2024-03-16T19:32:35.881947+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881947+0000","last_undegraded":"2024-03-16T19:32:35.881947+0000","last_fullsized":"2024-03-16T19:32:35.881947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651732+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651732+0000","last_peered":"2024-03-16T19:32:31.651732+0000","last_clean":"2024-03-16T19:32:31.651732+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651732+0000","last_undegraded":"2024-03-16T19:32:31.651732+0000","last_fullsized":"2024-03-16T19:32:31.651732+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":80,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.051201+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:32:35.051201+0000","last_peered":"2024-03-16T19:32:35.051201+0000","last_clean":"2024-03-16T19:32:35.051201+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:35.051201+0000","last_undegraded":"2024-03-16T19:32:35.051201+0000","last_fullsized":"2024-03-16T19:32:35.051201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":97,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209531+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209531+0000","last_peered":"2024-03-16T19:32:37.209531+0000","last_clean":"2024-03-16T19:32:37.209531+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209531+0000","last_undegraded":"2024-03-16T19:32:37.209531+0000","last_fullsized":"2024-03-16T19:32:37.209531+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.540693+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:32:40.540693+0000","last_peered":"2024-03-16T19:32:40.540693+0000","last_clean":"2024-03-16T19:32:40.540693+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:40.540693+0000","last_undegraded":"2024-03-16T19:32:40.540693+0000","last_fullsized":"2024-03-16T19:32:40.540693+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216652+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216652+0000","last_peered":"2024-03-16T19:32:27.216652+0000","last_clean":"2024-03-16T19:32:27.216652+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216652+0000","last_undegraded":"2024-03-16T19:32:27.216652+0000","last_fullsized":"2024-03-16T19:32:27.216652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.897651+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:32:40.897651+0000","last_peered":"2024-03-16T19:32:40.897651+0000","last_clean":"2024-03-16T19:32:40.897651+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:40.897651+0000","last_undegraded":"2024-03-16T19:32:40.897651+0000","last_fullsized":"2024-03-16T19:32:40.897651+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002739+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002739+0000","last_peered":"2024-03-16T19:32:28.002739+0000","last_clean":"2024-03-16T19:32:28.002739+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002739+0000","last_undegraded":"2024-03-16T19:32:28.002739+0000","last_fullsized":"2024-03-16T19:32:28.002739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207538+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207538+0000","last_peered":"2024-03-16T19:32:28.207538+0000","last_clean":"2024-03-16T19:32:28.207538+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207538+0000","last_undegraded":"2024-03-16T19:32:28.207538+0000","last_fullsized":"2024-03-16T19:32:28.207538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265363+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265363+0000","last_peered":"2024-03-16T19:32:35.265363+0000","last_clean":"2024-03-16T19:32:35.265363+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265363+0000","last_undegraded":"2024-03-16T19:32:35.265363+0000","last_fullsized":"2024-03-16T19:32:35.265363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913639+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913639+0000","last_peered":"2024-03-16T19:32:32.913639+0000","last_clean":"2024-03-16T19:32:32.913639+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913639+0000","last_undegraded":"2024-03-16T19:32:32.913639+0000","last_fullsized":"2024-03-16T19:32:32.913639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105259+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105259+0000","last_peered":"2024-03-16T19:32:32.105259+0000","last_clean":"2024-03-16T19:32:32.105259+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105259+0000","last_undegraded":"2024-03-16T19:32:32.105259+0000","last_fullsized":"2024-03-16T19:32:32.105259+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062743+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062743+0000","last_peered":"2024-03-16T19:32:38.062743+0000","last_clean":"2024-03-16T19:32:38.062743+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062743+0000","last_undegraded":"2024-03-16T19:32:38.062743+0000","last_fullsized":"2024-03-16T19:32:38.062743+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.685985+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.685985+0000","last_peered":"2024-03-16T19:32:28.685985+0000","last_clean":"2024-03-16T19:32:28.685985+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.685985+0000","last_undegraded":"2024-03-16T19:32:28.685985+0000","last_fullsized":"2024-03-16T19:32:28.685985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691219+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691219+0000","last_peered":"2024-03-16T19:32:32.691219+0000","last_clean":"2024-03-16T19:32:32.691219+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691219+0000","last_undegraded":"2024-03-16T19:32:32.691219+0000","last_fullsized":"2024-03-16T19:32:32.691219+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944568+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944568+0000","last_peered":"2024-03-16T19:32:30.944568+0000","last_clean":"2024-03-16T19:32:30.944568+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944568+0000","last_undegraded":"2024-03-16T19:32:30.944568+0000","last_fullsized":"2024-03-16T19:32:30.944568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706768+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706768+0000","last_peered":"2024-03-16T19:32:34.706768+0000","last_clean":"2024-03-16T19:32:34.706768+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706768+0000","last_undegraded":"2024-03-16T19:32:34.706768+0000","last_fullsized":"2024-03-16T19:32:34.706768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646626+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646626+0000","last_peered":"2024-03-16T19:32:29.646626+0000","last_clean":"2024-03-16T19:32:29.646626+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646626+0000","last_undegraded":"2024-03-16T19:32:29.646626+0000","last_fullsized":"2024-03-16T19:32:29.646626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223824+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223824+0000","last_peered":"2024-03-16T19:32:30.223824+0000","last_clean":"2024-03-16T19:32:30.223824+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223824+0000","last_undegraded":"2024-03-16T19:32:30.223824+0000","last_fullsized":"2024-03-16T19:32:30.223824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538373+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538373+0000","last_peered":"2024-03-16T19:32:26.538373+0000","last_clean":"2024-03-16T19:32:26.538373+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538373+0000","last_undegraded":"2024-03-16T19:32:26.538373+0000","last_fullsized":"2024-03-16T19:32:26.538373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":30,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076122+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076122+0000","last_peered":"2024-03-16T19:32:35.076122+0000","last_clean":"2024-03-16T19:32:35.076122+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076122+0000","last_undegraded":"2024-03-16T19:32:35.076122+0000","last_fullsized":"2024-03-16T19:32:35.076122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":246,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954015+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954015+0000","last_peered":"2024-03-16T19:32:38.954015+0000","last_clean":"2024-03-16T19:32:38.954015+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954015+0000","last_undegraded":"2024-03-16T19:32:38.954015+0000","last_fullsized":"2024-03-16T19:32:38.954015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179908+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179908+0000","last_peered":"2024-03-16T19:32:41.179908+0000","last_clean":"2024-03-16T19:32:41.179908+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179908+0000","last_undegraded":"2024-03-16T19:32:41.179908+0000","last_fullsized":"2024-03-16T19:32:41.179908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245654+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245654+0000","last_peered":"2024-03-16T19:32:36.245654+0000","last_clean":"2024-03-16T19:32:36.245654+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245654+0000","last_undegraded":"2024-03-16T19:32:36.245654+0000","last_fullsized":"2024-03-16T19:32:36.245654+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197866+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197866+0000","last_peered":"2024-03-16T19:32:27.197866+0000","last_clean":"2024-03-16T19:32:27.197866+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197866+0000","last_undegraded":"2024-03-16T19:32:27.197866+0000","last_fullsized":"2024-03-16T19:32:27.197866+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696935+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696935+0000","last_peered":"2024-03-16T19:32:36.696935+0000","last_clean":"2024-03-16T19:32:36.696935+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696935+0000","last_undegraded":"2024-03-16T19:32:36.696935+0000","last_fullsized":"2024-03-16T19:32:36.696935+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190715+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190715+0000","last_peered":"2024-03-16T19:32:37.190715+0000","last_clean":"2024-03-16T19:32:37.190715+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190715+0000","last_undegraded":"2024-03-16T19:32:37.190715+0000","last_fullsized":"2024-03-16T19:32:37.190715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938090+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938090+0000","last_peered":"2024-03-16T19:32:29.938090+0000","last_clean":"2024-03-16T19:32:29.938090+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938090+0000","last_undegraded":"2024-03-16T19:32:29.938090+0000","last_fullsized":"2024-03-16T19:32:29.938090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932702+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932702+0000","last_peered":"2024-03-16T19:32:39.932702+0000","last_clean":"2024-03-16T19:32:39.932702+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932702+0000","last_undegraded":"2024-03-16T19:32:39.932702+0000","last_fullsized":"2024-03-16T19:32:39.932702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":64,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:32:32.981143+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:32:32.981143+0000","last_peered":"2024-03-16T19:32:32.981143+0000","last_clean":"2024-03-16T19:32:32.981143+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:32.981143+0000","last_undegraded":"2024-03-16T19:32:32.981143+0000","last_fullsized":"2024-03-16T19:32:32.981143+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 18s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176411+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176411+0000","last_peered":"2024-03-16T19:32:38.176411+0000","last_clean":"2024-03-16T19:32:38.176411+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176411+0000","last_undegraded":"2024-03-16T19:32:38.176411+0000","last_fullsized":"2024-03-16T19:32:38.176411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":214,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982041+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982041+0000","last_peered":"2024-03-16T19:32:31.982041+0000","last_clean":"2024-03-16T19:32:31.982041+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982041+0000","last_undegraded":"2024-03-16T19:32:31.982041+0000","last_fullsized":"2024-03-16T19:32:31.982041+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149057+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149057+0000","last_peered":"2024-03-16T19:32:39.149057+0000","last_clean":"2024-03-16T19:32:39.149057+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149057+0000","last_undegraded":"2024-03-16T19:32:39.149057+0000","last_fullsized":"2024-03-16T19:32:39.149057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":65,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108534+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108534+0000","last_peered":"2024-03-16T19:32:32.108534+0000","last_clean":"2024-03-16T19:32:32.108534+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108534+0000","last_undegraded":"2024-03-16T19:32:32.108534+0000","last_fullsized":"2024-03-16T19:32:32.108534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059643+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059643+0000","last_peered":"2024-03-16T19:32:31.059643+0000","last_clean":"2024-03-16T19:32:31.059643+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059643+0000","last_undegraded":"2024-03-16T19:32:31.059643+0000","last_fullsized":"2024-03-16T19:32:31.059643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568517+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568517+0000","last_peered":"2024-03-16T19:32:27.568517+0000","last_clean":"2024-03-16T19:32:27.568517+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568517+0000","last_undegraded":"2024-03-16T19:32:27.568517+0000","last_fullsized":"2024-03-16T19:32:27.568517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651622+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651622+0000","last_peered":"2024-03-16T19:32:37.651622+0000","last_clean":"2024-03-16T19:32:37.651622+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651622+0000","last_undegraded":"2024-03-16T19:32:37.651622+0000","last_fullsized":"2024-03-16T19:32:37.651622+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205810+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205810+0000","last_peered":"2024-03-16T19:32:28.205810+0000","last_clean":"2024-03-16T19:32:28.205810+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205810+0000","last_undegraded":"2024-03-16T19:32:28.205810+0000","last_fullsized":"2024-03-16T19:32:28.205810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":165,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166583+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166583+0000","last_peered":"2024-03-16T19:32:40.166583+0000","last_clean":"2024-03-16T19:32:40.166583+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166583+0000","last_undegraded":"2024-03-16T19:32:40.166583+0000","last_fullsized":"2024-03-16T19:32:40.166583+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":374,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:47.115724+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:32:47.115724+0000","last_peered":"2024-03-16T19:32:47.115724+0000","last_clean":"2024-03-16T19:32:47.115724+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:47.115724+0000","last_undegraded":"2024-03-16T19:32:47.115724+0000","last_fullsized":"2024-03-16T19:32:47.115724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.532556+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:32:42.532556+0000","last_peered":"2024-03-16T19:32:42.532556+0000","last_clean":"2024-03-16T19:32:42.532556+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:42.532556+0000","last_undegraded":"2024-03-16T19:32:42.532556+0000","last_fullsized":"2024-03-16T19:32:42.532556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.219236+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:32:29.219236+0000","last_peered":"2024-03-16T19:32:29.219236+0000","last_clean":"2024-03-16T19:32:29.219236+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:29.219236+0000","last_undegraded":"2024-03-16T19:32:29.219236+0000","last_fullsized":"2024-03-16T19:32:29.219236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.162280+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:32:34.162280+0000","last_peered":"2024-03-16T19:32:34.162280+0000","last_clean":"2024-03-16T19:32:34.162280+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:34.162280+0000","last_undegraded":"2024-03-16T19:32:34.162280+0000","last_fullsized":"2024-03-16T19:32:34.162280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240482+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240482+0000","last_peered":"2024-03-16T19:32:34.240482+0000","last_clean":"2024-03-16T19:32:34.240482+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240482+0000","last_undegraded":"2024-03-16T19:32:34.240482+0000","last_fullsized":"2024-03-16T19:32:34.240482+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.144993+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.144993+0000","last_peered":"2024-03-16T19:32:33.144993+0000","last_clean":"2024-03-16T19:32:33.144993+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.144993+0000","last_undegraded":"2024-03-16T19:32:33.144993+0000","last_fullsized":"2024-03-16T19:32:33.144993+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":190,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:32:35.703093+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:32:35.703093+0000","last_peered":"2024-03-16T19:32:35.703093+0000","last_clean":"2024-03-16T19:32:35.703093+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:35.703093+0000","last_undegraded":"2024-03-16T19:32:35.703093+0000","last_fullsized":"2024-03-16T19:32:35.703093+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.692376+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:32:30.692376+0000","last_peered":"2024-03-16T19:32:30.692376+0000","last_clean":"2024-03-16T19:32:30.692376+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:30.692376+0000","last_undegraded":"2024-03-16T19:32:30.692376+0000","last_fullsized":"2024-03-16T19:32:30.692376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.998665+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:32:30.998665+0000","last_peered":"2024-03-16T19:32:30.998665+0000","last_clean":"2024-03-16T19:32:30.998665+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:30.998665+0000","last_undegraded":"2024-03-16T19:32:30.998665+0000","last_fullsized":"2024-03-16T19:32:30.998665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.721499+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:32:37.721499+0000","last_peered":"2024-03-16T19:32:37.721499+0000","last_clean":"2024-03-16T19:32:37.721499+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:37.721499+0000","last_undegraded":"2024-03-16T19:32:37.721499+0000","last_fullsized":"2024-03-16T19:32:37.721499+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.589644+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:32:39.589644+0000","last_peered":"2024-03-16T19:32:39.589644+0000","last_clean":"2024-03-16T19:32:39.589644+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:39.589644+0000","last_undegraded":"2024-03-16T19:32:39.589644+0000","last_fullsized":"2024-03-16T19:32:39.589644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:46.142279+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:32:46.142279+0000","last_peered":"2024-03-16T19:32:46.142279+0000","last_clean":"2024-03-16T19:32:46.142279+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:46.142279+0000","last_undegraded":"2024-03-16T19:32:46.142279+0000","last_fullsized":"2024-03-16T19:32:46.142279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":190,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:32:34.733330+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:32:34.733330+0000","last_peered":"2024-03-16T19:32:34.733330+0000","last_clean":"2024-03-16T19:32:34.733330+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:34.733330+0000","last_undegraded":"2024-03-16T19:32:34.733330+0000","last_fullsized":"2024-03-16T19:32:34.733330+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586853+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586853+0000","last_peered":"2024-03-16T19:32:29.586853+0000","last_clean":"2024-03-16T19:32:29.586853+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586853+0000","last_undegraded":"2024-03-16T19:32:29.586853+0000","last_fullsized":"2024-03-16T19:32:29.586853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.125468+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:32:33.125468+0000","last_peered":"2024-03-16T19:32:33.125468+0000","last_clean":"2024-03-16T19:32:33.125468+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:33.125468+0000","last_undegraded":"2024-03-16T19:32:33.125468+0000","last_fullsized":"2024-03-16T19:32:33.125468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910633+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910633+0000","last_peered":"2024-03-16T19:32:31.910633+0000","last_clean":"2024-03-16T19:32:31.910633+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910633+0000","last_undegraded":"2024-03-16T19:32:31.910633+0000","last_fullsized":"2024-03-16T19:32:31.910633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188001+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188001+0000","last_peered":"2024-03-16T19:32:32.188001+0000","last_clean":"2024-03-16T19:32:32.188001+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188001+0000","last_undegraded":"2024-03-16T19:32:32.188001+0000","last_fullsized":"2024-03-16T19:32:32.188001+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190102+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190102+0000","last_peered":"2024-03-16T19:32:44.190102+0000","last_clean":"2024-03-16T19:32:44.190102+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190102+0000","last_undegraded":"2024-03-16T19:32:44.190102+0000","last_fullsized":"2024-03-16T19:32:44.190102+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026127+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026127+0000","last_peered":"2024-03-16T19:32:30.026127+0000","last_clean":"2024-03-16T19:32:30.026127+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026127+0000","last_undegraded":"2024-03-16T19:32:30.026127+0000","last_fullsized":"2024-03-16T19:32:30.026127+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604768+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604768+0000","last_peered":"2024-03-16T19:32:28.604768+0000","last_clean":"2024-03-16T19:32:28.604768+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604768+0000","last_undegraded":"2024-03-16T19:32:28.604768+0000","last_fullsized":"2024-03-16T19:32:28.604768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:32:34.100708+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:32:34.100708+0000","last_peered":"2024-03-16T19:32:34.100708+0000","last_clean":"2024-03-16T19:32:34.100708+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:32:34.100708+0000","last_undegraded":"2024-03-16T19:32:34.100708+0000","last_fullsized":"2024-03-16T19:32:34.100708+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699866+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699866+0000","last_peered":"2024-03-16T19:32:33.699866+0000","last_clean":"2024-03-16T19:32:33.699866+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699866+0000","last_undegraded":"2024-03-16T19:32:33.699866+0000","last_fullsized":"2024-03-16T19:32:33.699866+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030418+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030418+0000","last_peered":"2024-03-16T19:32:39.030418+0000","last_clean":"2024-03-16T19:32:39.030418+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030418+0000","last_undegraded":"2024-03-16T19:32:39.030418+0000","last_fullsized":"2024-03-16T19:32:39.030418+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152403+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152403+0000","last_peered":"2024-03-16T19:32:33.152403+0000","last_clean":"2024-03-16T19:32:33.152403+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152403+0000","last_undegraded":"2024-03-16T19:32:33.152403+0000","last_fullsized":"2024-03-16T19:32:33.152403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889893+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889893+0000","last_peered":"2024-03-16T19:32:33.889893+0000","last_clean":"2024-03-16T19:32:33.889893+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889893+0000","last_undegraded":"2024-03-16T19:32:33.889893+0000","last_fullsized":"2024-03-16T19:32:33.889893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695467+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695467+0000","last_peered":"2024-03-16T19:32:31.695467+0000","last_clean":"2024-03-16T19:32:31.695467+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695467+0000","last_undegraded":"2024-03-16T19:32:31.695467+0000","last_fullsized":"2024-03-16T19:32:31.695467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102602+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102602+0000","last_peered":"2024-03-16T19:32:32.102602+0000","last_clean":"2024-03-16T19:32:32.102602+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102602+0000","last_undegraded":"2024-03-16T19:32:32.102602+0000","last_fullsized":"2024-03-16T19:32:32.102602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696329+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696329+0000","last_peered":"2024-03-16T19:32:32.696329+0000","last_clean":"2024-03-16T19:32:32.696329+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696329+0000","last_undegraded":"2024-03-16T19:32:32.696329+0000","last_fullsized":"2024-03-16T19:32:32.696329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177533+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177533+0000","last_peered":"2024-03-16T19:32:35.177533+0000","last_clean":"2024-03-16T19:32:35.177533+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177533+0000","last_undegraded":"2024-03-16T19:32:35.177533+0000","last_fullsized":"2024-03-16T19:32:35.177533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230753+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230753+0000","last_peered":"2024-03-16T19:32:33.230753+0000","last_clean":"2024-03-16T19:32:33.230753+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230753+0000","last_undegraded":"2024-03-16T19:32:33.230753+0000","last_fullsized":"2024-03-16T19:32:33.230753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.046496+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:32:40.046496+0000","last_peered":"2024-03-16T19:32:40.046496+0000","last_clean":"2024-03-16T19:32:40.046496+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:40.046496+0000","last_undegraded":"2024-03-16T19:32:40.046496+0000","last_fullsized":"2024-03-16T19:32:40.046496+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":174,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.063406+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:32:37.063406+0000","last_peered":"2024-03-16T19:32:37.063406+0000","last_clean":"2024-03-16T19:32:37.063406+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:37.063406+0000","last_undegraded":"2024-03-16T19:32:37.063406+0000","last_fullsized":"2024-03-16T19:32:37.063406+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272623+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272623+0000","last_peered":"2024-03-16T19:32:29.272623+0000","last_clean":"2024-03-16T19:32:29.272623+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272623+0000","last_undegraded":"2024-03-16T19:32:29.272623+0000","last_fullsized":"2024-03-16T19:32:29.272623+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":65536},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":65536},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:32:50.589 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:50.590 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:50.590 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:50.590 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:32:50.590 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:32:56.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:32:56.584 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:33:10.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:33:10.714 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.735+0000 7fbc302d4640 1 Processor -- start 2024-03-16T19:33:10.715 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.735+0000 7fbc302d4640 1 -- start start 2024-03-16T19:33:10.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.736+0000 7fbc302d4640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbc2806aed0 0x7fbc2806dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:10.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.736+0000 7fbc302d4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbc2806e2b0 0x7fbc2815e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:10.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.736+0000 7fbc302d4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2806ba50 con 0x7fbc28154bd0 2024-03-16T19:33:10.716 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.736+0000 7fbc302d4640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2806e6c0 con 0x7fbc2806aed0 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.736+0000 7fbc302d4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2815e0d0 con 0x7fbc2806e2b0 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.736+0000 7fbc2d86f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fbc2806aed0 0x7fbc2806dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.736+0000 7fbc2e070640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fbc28154bd0 0x7fbc280c7b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:49736/0 (socket says 172.21.15.172:49736) 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.736+0000 7fbc2e070640 1 -- 172.21.15.172:0/2163714467 learned_addr learned my addr 172.21.15.172:0/2163714467 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.736+0000 7fbc2e871640 1 --2- 172.21.15.172:0/2163714467 >> v2:172.21.15.174:3300/0 conn(0x7fbc2806e2b0 0x7fbc2815e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.737+0000 7fbc2e871640 1 -- 172.21.15.172:0/2163714467 >> v2:172.21.15.172:3300/0 conn(0x7fbc2806aed0 msgr2=0x7fbc2806dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.737+0000 7fbc2e871640 1 --2- 172.21.15.172:0/2163714467 >> v2:172.21.15.172:3300/0 conn(0x7fbc2806aed0 0x7fbc2806dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.737+0000 7fbc2e871640 1 -- 172.21.15.172:0/2163714467 >> v1:172.21.15.172:6789/0 conn(0x7fbc28154bd0 legacy=0x7fbc280c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.737+0000 7fbc2e871640 1 -- 172.21.15.172:0/2163714467 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2806bd30 con 0x7fbc2806e2b0 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.737+0000 7fbc2d06e640 1 -- 172.21.15.172:0/2163714467 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512102056 0 0) 0x7fbc2806ba50 con 0x7fbc28154bd0 2024-03-16T19:33:10.717 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.737+0000 7fbc2d86f640 1 --2- 172.21.15.172:0/2163714467 >> v2:172.21.15.172:3300/0 conn(0x7fbc2806aed0 0x7fbc2806dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:33:10.718 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.737+0000 7fbc2e871640 1 --2- 172.21.15.172:0/2163714467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc2806e2b0 0x7fbc2815e520 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbc280c4510 tx=0x7fbc2002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7d7f68f879b6d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc2d06e640 1 -- 172.21.15.172:0/2163714467 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2002cd00 con 0x7fbc2806e2b0 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc2d06e640 1 -- 172.21.15.172:0/2163714467 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2002ce60 con 0x7fbc2806e2b0 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc2d06e640 1 -- 172.21.15.172:0/2163714467 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20034660 con 0x7fbc2806e2b0 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc302d4640 1 -- 172.21.15.172:0/2163714467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc2806e2b0 msgr2=0x7fbc2815e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc302d4640 1 --2- 172.21.15.172:0/2163714467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc2806e2b0 0x7fbc2815e520 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbc280c4510 tx=0x7fbc2002bf50 comp rx=0 tx=0).stop 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc302d4640 1 -- 172.21.15.172:0/2163714467 shutdown_connections 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc302d4640 1 --2- 172.21.15.172:0/2163714467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc2806e2b0 0x7fbc2815e520 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc302d4640 1 --2- 172.21.15.172:0/2163714467 >> v2:172.21.15.172:3300/0 conn(0x7fbc2806aed0 0x7fbc2806dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc302d4640 1 -- 172.21.15.172:0/2163714467 >> 172.21.15.172:0/2163714467 conn(0x7fbc28071be0 msgr2=0x7fbc28071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc302d4640 1 -- 172.21.15.172:0/2163714467 shutdown_connections 2024-03-16T19:33:10.719 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.738+0000 7fbc302d4640 1 -- 172.21.15.172:0/2163714467 wait complete. 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc302d4640 1 Processor -- start 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc302d4640 1 -- start start 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc302d4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc2806aed0 0x7fbc281f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc302d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc2806e2b0 0x7fbc281f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc302d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc28154bd0 0x7fbc28153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc302d4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2815aca0 con 0x7fbc2806aed0 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc302d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2815b970 con 0x7fbc28154bd0 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc302d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc281f0d80 con 0x7fbc2806e2b0 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc2e070640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc2806aed0 0x7fbc281f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc2e070640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc2806aed0 0x7fbc281f0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57172/0 (socket says 172.21.15.172:57172) 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.740+0000 7fbc2e070640 1 -- 172.21.15.172:0/3048431142 learned_addr learned my addr 172.21.15.172:0/3048431142 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:33:10.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc2e871640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc28154bd0 0x7fbc28153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc2d86f640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc2806e2b0 0x7fbc281f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc2e871640 1 -- 172.21.15.172:0/3048431142 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc2806e2b0 msgr2=0x7fbc281f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc2e871640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc2806e2b0 0x7fbc281f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc2e871640 1 -- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc2806aed0 msgr2=0x7fbc281f0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc2e871640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc2806aed0 0x7fbc281f0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc2e871640 1 -- 172.21.15.172:0/3048431142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc18009400 con 0x7fbc28154bd0 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc2e070640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc2806aed0 0x7fbc281f0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc2e871640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc28154bd0 0x7fbc28153540 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbc20033000 tx=0x7fbc20004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc1effd640 1 -- 172.21.15.172:0/3048431142 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2003d600 con 0x7fbc28154bd0 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc1effd640 1 -- 172.21.15.172:0/3048431142 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2003d760 con 0x7fbc28154bd0 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc20009a70 con 0x7fbc28154bd0 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc1effd640 1 -- 172.21.15.172:0/3048431142 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2003db10 con 0x7fbc28154bd0 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.741+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc280c6350 con 0x7fbc28154bd0 2024-03-16T19:33:10.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.742+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc280681c0 con 0x7fbc28154bd0 2024-03-16T19:33:10.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.743+0000 7fbc1effd640 1 -- 172.21.15.172:0/3048431142 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbc2003b040 con 0x7fbc28154bd0 2024-03-16T19:33:10.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.743+0000 7fbc1effd640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc0c06b3e0 0x7fbc0c06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:10.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.743+0000 7fbc1effd640 1 -- 172.21.15.172:0/3048431142 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fbc200ab5f0 con 0x7fbc28154bd0 2024-03-16T19:33:10.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.743+0000 7fbc2e070640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc0c06b3e0 0x7fbc0c06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:10.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.744+0000 7fbc2e070640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc0c06b3e0 0x7fbc0c06d8a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbc24007b10 tx=0x7fbc24007a60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:33:10.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.748+0000 7fbc1effd640 1 -- 172.21.15.172:0/3048431142 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc200761d0 con 0x7fbc28154bd0 2024-03-16T19:33:10.936 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.956+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc2806c2f0 con 0x7fbc0c06b3e0 2024-03-16T19:33:10.944 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.965+0000 7fbc1effd640 1 -- 172.21.15.172:0/3048431142 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252913 (secure 0 0 0) 0x7fbc24004aa0 con 0x7fbc0c06b3e0 2024-03-16T19:33:10.947 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:33:10.948 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc0c06b3e0 msgr2=0x7fbc0c06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc0c06b3e0 0x7fbc0c06d8a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbc24007b10 tx=0x7fbc24007a60 comp rx=0 tx=0).stop 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc28154bd0 msgr2=0x7fbc28153540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc28154bd0 0x7fbc28153540 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbc20033000 tx=0x7fbc20004a40 comp rx=0 tx=0).stop 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 shutdown_connections 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc28154bd0 0x7fbc28153540 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc0c06b3e0 0x7fbc0c06d8a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc2806e2b0 0x7fbc281f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 --2- 172.21.15.172:0/3048431142 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc2806aed0 0x7fbc281f0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 >> 172.21.15.172:0/3048431142 conn(0x7fbc28071be0 msgr2=0x7fbc280cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:33:10.950 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 shutdown_connections 2024-03-16T19:33:10.951 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:10.969+0000 7fbc302d4640 1 -- 172.21.15.172:0/3048431142 wait complete. 2024-03-16T19:33:10.963 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-03-16T19:33:10.243196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002343"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.221290+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:32:36.221290+0000","last_peered":"2024-03-16T19:32:36.221290+0000","last_clean":"2024-03-16T19:32:36.221290+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:36.221290+0000","last_undegraded":"2024-03-16T19:32:36.221290+0000","last_fullsized":"2024-03-16T19:32:36.221290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.681904+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:32:36.681904+0000","last_peered":"2024-03-16T19:32:36.681904+0000","last_clean":"2024-03-16T19:32:36.681904+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:36.681904+0000","last_undegraded":"2024-03-16T19:32:36.681904+0000","last_fullsized":"2024-03-16T19:32:36.681904+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":112,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.236899+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:32:28.236899+0000","last_peered":"2024-03-16T19:32:28.236899+0000","last_clean":"2024-03-16T19:32:28.236899+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:28.236899+0000","last_undegraded":"2024-03-16T19:32:28.236899+0000","last_fullsized":"2024-03-16T19:32:28.236899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.687101+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:32:35.687101+0000","last_peered":"2024-03-16T19:32:35.687101+0000","last_clean":"2024-03-16T19:32:35.687101+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:35.687101+0000","last_undegraded":"2024-03-16T19:32:35.687101+0000","last_fullsized":"2024-03-16T19:32:35.687101+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193158+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193158+0000","last_peered":"2024-03-16T19:32:42.193158+0000","last_clean":"2024-03-16T19:32:42.193158+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193158+0000","last_undegraded":"2024-03-16T19:32:42.193158+0000","last_fullsized":"2024-03-16T19:32:42.193158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501727+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501727+0000","last_peered":"2024-03-16T19:32:43.501727+0000","last_clean":"2024-03-16T19:32:43.501727+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501727+0000","last_undegraded":"2024-03-16T19:32:43.501727+0000","last_fullsized":"2024-03-16T19:32:43.501727+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180894+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180894+0000","last_peered":"2024-03-16T19:32:45.180894+0000","last_clean":"2024-03-16T19:32:45.180894+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180894+0000","last_undegraded":"2024-03-16T19:32:45.180894+0000","last_fullsized":"2024-03-16T19:32:45.180894+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142579+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142579+0000","last_peered":"2024-03-16T19:32:31.142579+0000","last_clean":"2024-03-16T19:32:31.142579+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142579+0000","last_undegraded":"2024-03-16T19:32:31.142579+0000","last_fullsized":"2024-03-16T19:32:31.142579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050015+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050015+0000","last_peered":"2024-03-16T19:32:30.050015+0000","last_clean":"2024-03-16T19:32:30.050015+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050015+0000","last_undegraded":"2024-03-16T19:32:30.050015+0000","last_fullsized":"2024-03-16T19:32:30.050015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906468+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906468+0000","last_peered":"2024-03-16T19:32:36.906468+0000","last_clean":"2024-03-16T19:32:36.906468+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906468+0000","last_undegraded":"2024-03-16T19:32:36.906468+0000","last_fullsized":"2024-03-16T19:32:36.906468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946447+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946447+0000","last_peered":"2024-03-16T19:32:37.946447+0000","last_clean":"2024-03-16T19:32:37.946447+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946447+0000","last_undegraded":"2024-03-16T19:32:37.946447+0000","last_fullsized":"2024-03-16T19:32:37.946447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012343+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012343+0000","last_peered":"2024-03-16T19:32:29.012343+0000","last_clean":"2024-03-16T19:32:29.012343+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012343+0000","last_undegraded":"2024-03-16T19:32:29.012343+0000","last_fullsized":"2024-03-16T19:32:29.012343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085947+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085947+0000","last_peered":"2024-03-16T19:32:34.085947+0000","last_clean":"2024-03-16T19:32:34.085947+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085947+0000","last_undegraded":"2024-03-16T19:32:34.085947+0000","last_fullsized":"2024-03-16T19:32:34.085947+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194777+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194777+0000","last_peered":"2024-03-16T19:32:31.194777+0000","last_clean":"2024-03-16T19:32:31.194777+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194777+0000","last_undegraded":"2024-03-16T19:32:31.194777+0000","last_fullsized":"2024-03-16T19:32:31.194777+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622130+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622130+0000","last_peered":"2024-03-16T19:32:30.622130+0000","last_clean":"2024-03-16T19:32:30.622130+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622130+0000","last_undegraded":"2024-03-16T19:32:30.622130+0000","last_fullsized":"2024-03-16T19:32:30.622130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.527986+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.527986+0000","last_peered":"2024-03-16T19:32:41.527986+0000","last_clean":"2024-03-16T19:32:41.527986+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.527986+0000","last_undegraded":"2024-03-16T19:32:41.527986+0000","last_fullsized":"2024-03-16T19:32:41.527986+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668862+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668862+0000","last_peered":"2024-03-16T19:32:33.668862+0000","last_clean":"2024-03-16T19:32:33.668862+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668862+0000","last_undegraded":"2024-03-16T19:32:33.668862+0000","last_fullsized":"2024-03-16T19:32:33.668862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":70,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158073+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158073+0000","last_peered":"2024-03-16T19:32:30.158073+0000","last_clean":"2024-03-16T19:32:30.158073+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158073+0000","last_undegraded":"2024-03-16T19:32:30.158073+0000","last_fullsized":"2024-03-16T19:32:30.158073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198963+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198963+0000","last_peered":"2024-03-16T19:32:43.198963+0000","last_clean":"2024-03-16T19:32:43.198963+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198963+0000","last_undegraded":"2024-03-16T19:32:43.198963+0000","last_fullsized":"2024-03-16T19:32:43.198963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874908+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874908+0000","last_peered":"2024-03-16T19:32:34.874908+0000","last_clean":"2024-03-16T19:32:34.874908+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874908+0000","last_undegraded":"2024-03-16T19:32:34.874908+0000","last_fullsized":"2024-03-16T19:32:34.874908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202018+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202018+0000","last_peered":"2024-03-16T19:32:26.202018+0000","last_clean":"2024-03-16T19:32:26.202018+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202018+0000","last_undegraded":"2024-03-16T19:32:26.202018+0000","last_fullsized":"2024-03-16T19:32:26.202018+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113786+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113786+0000","last_peered":"2024-03-16T19:32:31.113786+0000","last_clean":"2024-03-16T19:32:31.113786+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113786+0000","last_undegraded":"2024-03-16T19:32:31.113786+0000","last_fullsized":"2024-03-16T19:32:31.113786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189258+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189258+0000","last_peered":"2024-03-16T19:32:30.189258+0000","last_clean":"2024-03-16T19:32:30.189258+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189258+0000","last_undegraded":"2024-03-16T19:32:30.189258+0000","last_fullsized":"2024-03-16T19:32:30.189258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633685+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633685+0000","last_peered":"2024-03-16T19:32:38.633685+0000","last_clean":"2024-03-16T19:32:38.633685+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633685+0000","last_undegraded":"2024-03-16T19:32:38.633685+0000","last_fullsized":"2024-03-16T19:32:38.633685+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":171,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030001+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030001+0000","last_peered":"2024-03-16T19:32:36.030001+0000","last_clean":"2024-03-16T19:32:36.030001+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030001+0000","last_undegraded":"2024-03-16T19:32:36.030001+0000","last_fullsized":"2024-03-16T19:32:36.030001+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881947+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881947+0000","last_peered":"2024-03-16T19:32:35.881947+0000","last_clean":"2024-03-16T19:32:35.881947+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881947+0000","last_undegraded":"2024-03-16T19:32:35.881947+0000","last_fullsized":"2024-03-16T19:32:35.881947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651732+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651732+0000","last_peered":"2024-03-16T19:32:31.651732+0000","last_clean":"2024-03-16T19:32:31.651732+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651732+0000","last_undegraded":"2024-03-16T19:32:31.651732+0000","last_fullsized":"2024-03-16T19:32:31.651732+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":80,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.051201+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:32:35.051201+0000","last_peered":"2024-03-16T19:32:35.051201+0000","last_clean":"2024-03-16T19:32:35.051201+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:35.051201+0000","last_undegraded":"2024-03-16T19:32:35.051201+0000","last_fullsized":"2024-03-16T19:32:35.051201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":97,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209531+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209531+0000","last_peered":"2024-03-16T19:32:37.209531+0000","last_clean":"2024-03-16T19:32:37.209531+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209531+0000","last_undegraded":"2024-03-16T19:32:37.209531+0000","last_fullsized":"2024-03-16T19:32:37.209531+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.540693+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:32:40.540693+0000","last_peered":"2024-03-16T19:32:40.540693+0000","last_clean":"2024-03-16T19:32:40.540693+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:40.540693+0000","last_undegraded":"2024-03-16T19:32:40.540693+0000","last_fullsized":"2024-03-16T19:32:40.540693+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216652+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216652+0000","last_peered":"2024-03-16T19:32:27.216652+0000","last_clean":"2024-03-16T19:32:27.216652+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216652+0000","last_undegraded":"2024-03-16T19:32:27.216652+0000","last_fullsized":"2024-03-16T19:32:27.216652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.897651+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:32:40.897651+0000","last_peered":"2024-03-16T19:32:40.897651+0000","last_clean":"2024-03-16T19:32:40.897651+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:40.897651+0000","last_undegraded":"2024-03-16T19:32:40.897651+0000","last_fullsized":"2024-03-16T19:32:40.897651+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002739+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002739+0000","last_peered":"2024-03-16T19:32:28.002739+0000","last_clean":"2024-03-16T19:32:28.002739+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002739+0000","last_undegraded":"2024-03-16T19:32:28.002739+0000","last_fullsized":"2024-03-16T19:32:28.002739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207538+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207538+0000","last_peered":"2024-03-16T19:32:28.207538+0000","last_clean":"2024-03-16T19:32:28.207538+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207538+0000","last_undegraded":"2024-03-16T19:32:28.207538+0000","last_fullsized":"2024-03-16T19:32:28.207538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265363+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265363+0000","last_peered":"2024-03-16T19:32:35.265363+0000","last_clean":"2024-03-16T19:32:35.265363+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265363+0000","last_undegraded":"2024-03-16T19:32:35.265363+0000","last_fullsized":"2024-03-16T19:32:35.265363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913639+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913639+0000","last_peered":"2024-03-16T19:32:32.913639+0000","last_clean":"2024-03-16T19:32:32.913639+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913639+0000","last_undegraded":"2024-03-16T19:32:32.913639+0000","last_fullsized":"2024-03-16T19:32:32.913639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105259+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105259+0000","last_peered":"2024-03-16T19:32:32.105259+0000","last_clean":"2024-03-16T19:32:32.105259+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105259+0000","last_undegraded":"2024-03-16T19:32:32.105259+0000","last_fullsized":"2024-03-16T19:32:32.105259+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062743+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062743+0000","last_peered":"2024-03-16T19:32:38.062743+0000","last_clean":"2024-03-16T19:32:38.062743+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062743+0000","last_undegraded":"2024-03-16T19:32:38.062743+0000","last_fullsized":"2024-03-16T19:32:38.062743+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.685985+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.685985+0000","last_peered":"2024-03-16T19:32:28.685985+0000","last_clean":"2024-03-16T19:32:28.685985+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.685985+0000","last_undegraded":"2024-03-16T19:32:28.685985+0000","last_fullsized":"2024-03-16T19:32:28.685985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691219+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691219+0000","last_peered":"2024-03-16T19:32:32.691219+0000","last_clean":"2024-03-16T19:32:32.691219+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691219+0000","last_undegraded":"2024-03-16T19:32:32.691219+0000","last_fullsized":"2024-03-16T19:32:32.691219+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944568+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944568+0000","last_peered":"2024-03-16T19:32:30.944568+0000","last_clean":"2024-03-16T19:32:30.944568+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944568+0000","last_undegraded":"2024-03-16T19:32:30.944568+0000","last_fullsized":"2024-03-16T19:32:30.944568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706768+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706768+0000","last_peered":"2024-03-16T19:32:34.706768+0000","last_clean":"2024-03-16T19:32:34.706768+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706768+0000","last_undegraded":"2024-03-16T19:32:34.706768+0000","last_fullsized":"2024-03-16T19:32:34.706768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646626+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646626+0000","last_peered":"2024-03-16T19:32:29.646626+0000","last_clean":"2024-03-16T19:32:29.646626+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646626+0000","last_undegraded":"2024-03-16T19:32:29.646626+0000","last_fullsized":"2024-03-16T19:32:29.646626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223824+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223824+0000","last_peered":"2024-03-16T19:32:30.223824+0000","last_clean":"2024-03-16T19:32:30.223824+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223824+0000","last_undegraded":"2024-03-16T19:32:30.223824+0000","last_fullsized":"2024-03-16T19:32:30.223824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538373+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538373+0000","last_peered":"2024-03-16T19:32:26.538373+0000","last_clean":"2024-03-16T19:32:26.538373+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538373+0000","last_undegraded":"2024-03-16T19:32:26.538373+0000","last_fullsized":"2024-03-16T19:32:26.538373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":30,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076122+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076122+0000","last_peered":"2024-03-16T19:32:35.076122+0000","last_clean":"2024-03-16T19:32:35.076122+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076122+0000","last_undegraded":"2024-03-16T19:32:35.076122+0000","last_fullsized":"2024-03-16T19:32:35.076122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":246,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954015+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954015+0000","last_peered":"2024-03-16T19:32:38.954015+0000","last_clean":"2024-03-16T19:32:38.954015+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954015+0000","last_undegraded":"2024-03-16T19:32:38.954015+0000","last_fullsized":"2024-03-16T19:32:38.954015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179908+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179908+0000","last_peered":"2024-03-16T19:32:41.179908+0000","last_clean":"2024-03-16T19:32:41.179908+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179908+0000","last_undegraded":"2024-03-16T19:32:41.179908+0000","last_fullsized":"2024-03-16T19:32:41.179908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245654+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245654+0000","last_peered":"2024-03-16T19:32:36.245654+0000","last_clean":"2024-03-16T19:32:36.245654+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245654+0000","last_undegraded":"2024-03-16T19:32:36.245654+0000","last_fullsized":"2024-03-16T19:32:36.245654+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197866+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197866+0000","last_peered":"2024-03-16T19:32:27.197866+0000","last_clean":"2024-03-16T19:32:27.197866+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197866+0000","last_undegraded":"2024-03-16T19:32:27.197866+0000","last_fullsized":"2024-03-16T19:32:27.197866+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696935+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696935+0000","last_peered":"2024-03-16T19:32:36.696935+0000","last_clean":"2024-03-16T19:32:36.696935+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696935+0000","last_undegraded":"2024-03-16T19:32:36.696935+0000","last_fullsized":"2024-03-16T19:32:36.696935+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190715+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190715+0000","last_peered":"2024-03-16T19:32:37.190715+0000","last_clean":"2024-03-16T19:32:37.190715+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190715+0000","last_undegraded":"2024-03-16T19:32:37.190715+0000","last_fullsized":"2024-03-16T19:32:37.190715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938090+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938090+0000","last_peered":"2024-03-16T19:32:29.938090+0000","last_clean":"2024-03-16T19:32:29.938090+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938090+0000","last_undegraded":"2024-03-16T19:32:29.938090+0000","last_fullsized":"2024-03-16T19:32:29.938090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932702+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932702+0000","last_peered":"2024-03-16T19:32:39.932702+0000","last_clean":"2024-03-16T19:32:39.932702+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932702+0000","last_undegraded":"2024-03-16T19:32:39.932702+0000","last_fullsized":"2024-03-16T19:32:39.932702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":65,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:32:47.982348+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:32:47.982348+0000","last_peered":"2024-03-16T19:32:47.982348+0000","last_clean":"2024-03-16T19:32:47.982348+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:32:47.982348+0000","last_undegraded":"2024-03-16T19:32:47.982348+0000","last_fullsized":"2024-03-16T19:32:47.982348+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 33s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176411+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176411+0000","last_peered":"2024-03-16T19:32:38.176411+0000","last_clean":"2024-03-16T19:32:38.176411+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176411+0000","last_undegraded":"2024-03-16T19:32:38.176411+0000","last_fullsized":"2024-03-16T19:32:38.176411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":214,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982041+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982041+0000","last_peered":"2024-03-16T19:32:31.982041+0000","last_clean":"2024-03-16T19:32:31.982041+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982041+0000","last_undegraded":"2024-03-16T19:32:31.982041+0000","last_fullsized":"2024-03-16T19:32:31.982041+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149057+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149057+0000","last_peered":"2024-03-16T19:32:39.149057+0000","last_clean":"2024-03-16T19:32:39.149057+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149057+0000","last_undegraded":"2024-03-16T19:32:39.149057+0000","last_fullsized":"2024-03-16T19:32:39.149057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":65,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108534+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108534+0000","last_peered":"2024-03-16T19:32:32.108534+0000","last_clean":"2024-03-16T19:32:32.108534+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108534+0000","last_undegraded":"2024-03-16T19:32:32.108534+0000","last_fullsized":"2024-03-16T19:32:32.108534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059643+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059643+0000","last_peered":"2024-03-16T19:32:31.059643+0000","last_clean":"2024-03-16T19:32:31.059643+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059643+0000","last_undegraded":"2024-03-16T19:32:31.059643+0000","last_fullsized":"2024-03-16T19:32:31.059643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568517+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568517+0000","last_peered":"2024-03-16T19:32:27.568517+0000","last_clean":"2024-03-16T19:32:27.568517+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568517+0000","last_undegraded":"2024-03-16T19:32:27.568517+0000","last_fullsized":"2024-03-16T19:32:27.568517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651622+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651622+0000","last_peered":"2024-03-16T19:32:37.651622+0000","last_clean":"2024-03-16T19:32:37.651622+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651622+0000","last_undegraded":"2024-03-16T19:32:37.651622+0000","last_fullsized":"2024-03-16T19:32:37.651622+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205810+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205810+0000","last_peered":"2024-03-16T19:32:28.205810+0000","last_clean":"2024-03-16T19:32:28.205810+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205810+0000","last_undegraded":"2024-03-16T19:32:28.205810+0000","last_fullsized":"2024-03-16T19:32:28.205810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":165,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166583+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166583+0000","last_peered":"2024-03-16T19:32:40.166583+0000","last_clean":"2024-03-16T19:32:40.166583+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166583+0000","last_undegraded":"2024-03-16T19:32:40.166583+0000","last_fullsized":"2024-03-16T19:32:40.166583+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":374,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:47.115724+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:32:47.115724+0000","last_peered":"2024-03-16T19:32:47.115724+0000","last_clean":"2024-03-16T19:32:47.115724+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:47.115724+0000","last_undegraded":"2024-03-16T19:32:47.115724+0000","last_fullsized":"2024-03-16T19:32:47.115724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.532556+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:32:42.532556+0000","last_peered":"2024-03-16T19:32:42.532556+0000","last_clean":"2024-03-16T19:32:42.532556+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:42.532556+0000","last_undegraded":"2024-03-16T19:32:42.532556+0000","last_fullsized":"2024-03-16T19:32:42.532556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.219236+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:32:29.219236+0000","last_peered":"2024-03-16T19:32:29.219236+0000","last_clean":"2024-03-16T19:32:29.219236+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:29.219236+0000","last_undegraded":"2024-03-16T19:32:29.219236+0000","last_fullsized":"2024-03-16T19:32:29.219236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.162280+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:32:34.162280+0000","last_peered":"2024-03-16T19:32:34.162280+0000","last_clean":"2024-03-16T19:32:34.162280+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:34.162280+0000","last_undegraded":"2024-03-16T19:32:34.162280+0000","last_fullsized":"2024-03-16T19:32:34.162280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240482+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240482+0000","last_peered":"2024-03-16T19:32:34.240482+0000","last_clean":"2024-03-16T19:32:34.240482+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240482+0000","last_undegraded":"2024-03-16T19:32:34.240482+0000","last_fullsized":"2024-03-16T19:32:34.240482+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.144993+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.144993+0000","last_peered":"2024-03-16T19:32:33.144993+0000","last_clean":"2024-03-16T19:32:33.144993+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.144993+0000","last_undegraded":"2024-03-16T19:32:33.144993+0000","last_fullsized":"2024-03-16T19:32:33.144993+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":192,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:32:53.237851+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:32:53.237851+0000","last_peered":"2024-03-16T19:32:53.237851+0000","last_clean":"2024-03-16T19:32:53.237851+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:32:53.237851+0000","last_undegraded":"2024-03-16T19:32:53.237851+0000","last_fullsized":"2024-03-16T19:32:53.237851+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 32s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.692376+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:32:30.692376+0000","last_peered":"2024-03-16T19:32:30.692376+0000","last_clean":"2024-03-16T19:32:30.692376+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:30.692376+0000","last_undegraded":"2024-03-16T19:32:30.692376+0000","last_fullsized":"2024-03-16T19:32:30.692376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.998665+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:32:30.998665+0000","last_peered":"2024-03-16T19:32:30.998665+0000","last_clean":"2024-03-16T19:32:30.998665+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:30.998665+0000","last_undegraded":"2024-03-16T19:32:30.998665+0000","last_fullsized":"2024-03-16T19:32:30.998665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.721499+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:32:37.721499+0000","last_peered":"2024-03-16T19:32:37.721499+0000","last_clean":"2024-03-16T19:32:37.721499+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:37.721499+0000","last_undegraded":"2024-03-16T19:32:37.721499+0000","last_fullsized":"2024-03-16T19:32:37.721499+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.589644+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:32:39.589644+0000","last_peered":"2024-03-16T19:32:39.589644+0000","last_clean":"2024-03-16T19:32:39.589644+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:39.589644+0000","last_undegraded":"2024-03-16T19:32:39.589644+0000","last_fullsized":"2024-03-16T19:32:39.589644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:46.142279+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:32:46.142279+0000","last_peered":"2024-03-16T19:32:46.142279+0000","last_clean":"2024-03-16T19:32:46.142279+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:46.142279+0000","last_undegraded":"2024-03-16T19:32:46.142279+0000","last_fullsized":"2024-03-16T19:32:46.142279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":192,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:32:53.237898+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:32:53.237898+0000","last_peered":"2024-03-16T19:32:53.237898+0000","last_clean":"2024-03-16T19:32:53.237898+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:32:53.237898+0000","last_undegraded":"2024-03-16T19:32:53.237898+0000","last_fullsized":"2024-03-16T19:32:53.237898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 33s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586853+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586853+0000","last_peered":"2024-03-16T19:32:29.586853+0000","last_clean":"2024-03-16T19:32:29.586853+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586853+0000","last_undegraded":"2024-03-16T19:32:29.586853+0000","last_fullsized":"2024-03-16T19:32:29.586853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.125468+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:32:33.125468+0000","last_peered":"2024-03-16T19:32:33.125468+0000","last_clean":"2024-03-16T19:32:33.125468+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:33.125468+0000","last_undegraded":"2024-03-16T19:32:33.125468+0000","last_fullsized":"2024-03-16T19:32:33.125468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910633+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910633+0000","last_peered":"2024-03-16T19:32:31.910633+0000","last_clean":"2024-03-16T19:32:31.910633+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910633+0000","last_undegraded":"2024-03-16T19:32:31.910633+0000","last_fullsized":"2024-03-16T19:32:31.910633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188001+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188001+0000","last_peered":"2024-03-16T19:32:32.188001+0000","last_clean":"2024-03-16T19:32:32.188001+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188001+0000","last_undegraded":"2024-03-16T19:32:32.188001+0000","last_fullsized":"2024-03-16T19:32:32.188001+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190102+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190102+0000","last_peered":"2024-03-16T19:32:44.190102+0000","last_clean":"2024-03-16T19:32:44.190102+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190102+0000","last_undegraded":"2024-03-16T19:32:44.190102+0000","last_fullsized":"2024-03-16T19:32:44.190102+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026127+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026127+0000","last_peered":"2024-03-16T19:32:30.026127+0000","last_clean":"2024-03-16T19:32:30.026127+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026127+0000","last_undegraded":"2024-03-16T19:32:30.026127+0000","last_fullsized":"2024-03-16T19:32:30.026127+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604768+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604768+0000","last_peered":"2024-03-16T19:32:28.604768+0000","last_clean":"2024-03-16T19:32:28.604768+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604768+0000","last_undegraded":"2024-03-16T19:32:28.604768+0000","last_fullsized":"2024-03-16T19:32:28.604768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:32:52.982874+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:32:52.982874+0000","last_peered":"2024-03-16T19:32:52.982874+0000","last_clean":"2024-03-16T19:32:52.982874+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:32:52.982874+0000","last_undegraded":"2024-03-16T19:32:52.982874+0000","last_fullsized":"2024-03-16T19:32:52.982874+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 33s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699866+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699866+0000","last_peered":"2024-03-16T19:32:33.699866+0000","last_clean":"2024-03-16T19:32:33.699866+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699866+0000","last_undegraded":"2024-03-16T19:32:33.699866+0000","last_fullsized":"2024-03-16T19:32:33.699866+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030418+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030418+0000","last_peered":"2024-03-16T19:32:39.030418+0000","last_clean":"2024-03-16T19:32:39.030418+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030418+0000","last_undegraded":"2024-03-16T19:32:39.030418+0000","last_fullsized":"2024-03-16T19:32:39.030418+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152403+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152403+0000","last_peered":"2024-03-16T19:32:33.152403+0000","last_clean":"2024-03-16T19:32:33.152403+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152403+0000","last_undegraded":"2024-03-16T19:32:33.152403+0000","last_fullsized":"2024-03-16T19:32:33.152403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889893+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889893+0000","last_peered":"2024-03-16T19:32:33.889893+0000","last_clean":"2024-03-16T19:32:33.889893+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889893+0000","last_undegraded":"2024-03-16T19:32:33.889893+0000","last_fullsized":"2024-03-16T19:32:33.889893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695467+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695467+0000","last_peered":"2024-03-16T19:32:31.695467+0000","last_clean":"2024-03-16T19:32:31.695467+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695467+0000","last_undegraded":"2024-03-16T19:32:31.695467+0000","last_fullsized":"2024-03-16T19:32:31.695467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102602+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102602+0000","last_peered":"2024-03-16T19:32:32.102602+0000","last_clean":"2024-03-16T19:32:32.102602+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102602+0000","last_undegraded":"2024-03-16T19:32:32.102602+0000","last_fullsized":"2024-03-16T19:32:32.102602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696329+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696329+0000","last_peered":"2024-03-16T19:32:32.696329+0000","last_clean":"2024-03-16T19:32:32.696329+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696329+0000","last_undegraded":"2024-03-16T19:32:32.696329+0000","last_fullsized":"2024-03-16T19:32:32.696329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177533+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177533+0000","last_peered":"2024-03-16T19:32:35.177533+0000","last_clean":"2024-03-16T19:32:35.177533+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177533+0000","last_undegraded":"2024-03-16T19:32:35.177533+0000","last_fullsized":"2024-03-16T19:32:35.177533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230753+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230753+0000","last_peered":"2024-03-16T19:32:33.230753+0000","last_clean":"2024-03-16T19:32:33.230753+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230753+0000","last_undegraded":"2024-03-16T19:32:33.230753+0000","last_fullsized":"2024-03-16T19:32:33.230753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.046496+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:32:40.046496+0000","last_peered":"2024-03-16T19:32:40.046496+0000","last_clean":"2024-03-16T19:32:40.046496+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:40.046496+0000","last_undegraded":"2024-03-16T19:32:40.046496+0000","last_fullsized":"2024-03-16T19:32:40.046496+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":174,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.063406+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:32:37.063406+0000","last_peered":"2024-03-16T19:32:37.063406+0000","last_clean":"2024-03-16T19:32:37.063406+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:37.063406+0000","last_undegraded":"2024-03-16T19:32:37.063406+0000","last_fullsized":"2024-03-16T19:32:37.063406+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272623+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272623+0000","last_peered":"2024-03-16T19:32:29.272623+0000","last_clean":"2024-03-16T19:32:29.272623+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272623+0000","last_undegraded":"2024-03-16T19:32:29.272623+0000","last_fullsized":"2024-03-16T19:32:29.272623+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574900,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122066,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:33:10.968 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:10.968 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:10.968 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:10.968 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:10.969 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:33:26.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:33:26.714 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:33:30.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:33:31.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.119+0000 7fe07085c640 1 Processor -- start 2024-03-16T19:33:31.100 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.119+0000 7fe07085c640 1 -- start start 2024-03-16T19:33:31.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.120+0000 7fe07085c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c154cb0 0x7fe06c06dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:31.101 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.120+0000 7fe07085c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe06c0c8ba0 0x7fe06c0cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.120+0000 7fe07085c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c0cbba0 con 0x7fe06c06fe10 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.120+0000 7fe07085c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06c0c91f0 con 0x7fe06c0c8ba0 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe07085c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c06c180 con 0x7fe06c154cb0 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe06a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c154cb0 0x7fe06c06dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe069d9b640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe06c06fe10 0x7fe06c15de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:39790/0 (socket says 172.21.15.172:39790) 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe069d9b640 1 -- 172.21.15.172:0/583449033 learned_addr learned my addr 172.21.15.172:0/583449033 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe06ad9d640 1 --2- 172.21.15.172:0/583449033 >> v2:172.21.15.174:3300/0 conn(0x7fe06c0c8ba0 0x7fe06c0cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe06a59c640 1 -- 172.21.15.172:0/583449033 >> v2:172.21.15.174:3300/0 conn(0x7fe06c0c8ba0 msgr2=0x7fe06c0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe06a59c640 1 --2- 172.21.15.172:0/583449033 >> v2:172.21.15.174:3300/0 conn(0x7fe06c0c8ba0 0x7fe06c0cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe06959a640 1 -- 172.21.15.172:0/583449033 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821611943 0 0) 0x7fe06c0cbba0 con 0x7fe06c06fe10 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe06a59c640 1 -- 172.21.15.172:0/583449033 >> v1:172.21.15.174:6789/0 conn(0x7fe06c06fe10 legacy=0x7fe06c15de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.121+0000 7fe06a59c640 1 -- 172.21.15.172:0/583449033 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c06c460 con 0x7fe06c154cb0 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.122+0000 7fe06a59c640 1 --2- 172.21.15.172:0/583449033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c154cb0 0x7fe06c06dc20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0097a0 tx=0x7fe05c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5cfc4e0ffd1676f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.122+0000 7fe06959a640 1 -- 172.21.15.172:0/583449033 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c02f050 con 0x7fe06c154cb0 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.122+0000 7fe06959a640 1 -- 172.21.15.172:0/583449033 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c031700 con 0x7fe06c154cb0 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.122+0000 7fe06959a640 1 -- 172.21.15.172:0/583449033 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c02f050 con 0x7fe06c154cb0 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.123+0000 7fe07085c640 1 -- 172.21.15.172:0/583449033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c154cb0 msgr2=0x7fe06c06dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.123+0000 7fe07085c640 1 --2- 172.21.15.172:0/583449033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c154cb0 0x7fe06c06dc20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0097a0 tx=0x7fe05c02b7a0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.123+0000 7fe07085c640 1 -- 172.21.15.172:0/583449033 shutdown_connections 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.123+0000 7fe07085c640 1 --2- 172.21.15.172:0/583449033 >> v2:172.21.15.174:3300/0 conn(0x7fe06c0c8ba0 0x7fe06c0cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.123+0000 7fe07085c640 1 --2- 172.21.15.172:0/583449033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c154cb0 0x7fe06c06dc20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.123+0000 7fe07085c640 1 -- 172.21.15.172:0/583449033 >> 172.21.15.172:0/583449033 conn(0x7fe06c070860 msgr2=0x7fe06c070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.123+0000 7fe07085c640 1 -- 172.21.15.172:0/583449033 shutdown_connections 2024-03-16T19:33:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.123+0000 7fe07085c640 1 -- 172.21.15.172:0/583449033 wait complete. 2024-03-16T19:33:31.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.125+0000 7fe07085c640 1 Processor -- start 2024-03-16T19:33:31.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.125+0000 7fe07085c640 1 -- start start 2024-03-16T19:33:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe07085c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c06fe10 0x7fe06c14c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe07085c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06c0c8ba0 0x7fe06c14d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe07085c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe06c154cb0 0x7fe06c15a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe07085c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c150a00 con 0x7fe06c0c8ba0 2024-03-16T19:33:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe07085c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c14aa90 con 0x7fe06c154cb0 2024-03-16T19:33:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe07085c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c14ad70 con 0x7fe06c06fe10 2024-03-16T19:33:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe069d9b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06c0c8ba0 0x7fe06c14d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c06fe10 0x7fe06c14c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe06c154cb0 0x7fe06c15a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c06fe10 0x7fe06c14c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56036/0 (socket says 172.21.15.172:56036) 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06a59c640 1 -- 172.21.15.172:0/1592308418 learned_addr learned my addr 172.21.15.172:0/1592308418 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06a59c640 1 -- 172.21.15.172:0/1592308418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe06c154cb0 msgr2=0x7fe06c15a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06a59c640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe06c154cb0 0x7fe06c15a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06a59c640 1 -- 172.21.15.172:0/1592308418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06c0c8ba0 msgr2=0x7fe06c14d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06a59c640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06c0c8ba0 0x7fe06c14d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06a59c640 1 -- 172.21.15.172:0/1592308418 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c009400 con 0x7fe06c06fe10 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06ad9d640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe06c154cb0 0x7fe06c15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.126+0000 7fe06a59c640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c06fe10 0x7fe06c14c790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe05c036040 tx=0x7fe05c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.127+0000 7fe05affd640 1 -- 172.21.15.172:0/1592308418 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c006e10 con 0x7fe06c06fe10 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.127+0000 7fe05affd640 1 -- 172.21.15.172:0/1592308418 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c0044c0 con 0x7fe06c06fe10 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.127+0000 7fe05affd640 1 -- 172.21.15.172:0/1592308418 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c003a40 con 0x7fe06c06fe10 2024-03-16T19:33:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.127+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06c14b050 con 0x7fe06c06fe10 2024-03-16T19:33:31.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.127+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06c14b610 con 0x7fe06c06fe10 2024-03-16T19:33:31.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.128+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06c0724c0 con 0x7fe06c06fe10 2024-03-16T19:33:31.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.128+0000 7fe05affd640 1 -- 172.21.15.172:0/1592308418 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fe05c03a070 con 0x7fe06c06fe10 2024-03-16T19:33:31.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.129+0000 7fe05affd640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe04c06b3b0 0x7fe04c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:31.109 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.129+0000 7fe05affd640 1 -- 172.21.15.172:0/1592308418 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fe05c0ab7c0 con 0x7fe06c06fe10 2024-03-16T19:33:31.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.129+0000 7fe069d9b640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe04c06b3b0 0x7fe04c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:31.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.130+0000 7fe069d9b640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe04c06b3b0 0x7fe04c06d870 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe06c150fb0 tx=0x7fe060007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:33:31.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.133+0000 7fe05affd640 1 -- 172.21.15.172:0/1592308418 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe05c0763a0 con 0x7fe06c06fe10 2024-03-16T19:33:31.394 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.414+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe06c0c9790 con 0x7fe04c06b3b0 2024-03-16T19:33:31.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.423+0000 7fe05affd640 1 -- 172.21.15.172:0/1592308418 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252913 (secure 0 0 0) 0x7fe060004c70 con 0x7fe04c06b3b0 2024-03-16T19:33:31.405 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:33:31.407 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:33:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe04c06b3b0 msgr2=0x7fe04c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe04c06b3b0 0x7fe04c06d870 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe06c150fb0 tx=0x7fe060007d50 comp rx=0 tx=0).stop 2024-03-16T19:33:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c06fe10 msgr2=0x7fe06c14c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c06fe10 0x7fe06c14c790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe05c036040 tx=0x7fe05c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 shutdown_connections 2024-03-16T19:33:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe06c154cb0 0x7fe06c15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06c0c8ba0 0x7fe06c14d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fe04c06b3b0 0x7fe04c06d870 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 --2- 172.21.15.172:0/1592308418 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe06c06fe10 0x7fe06c14c790 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:31.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 >> 172.21.15.172:0/1592308418 conn(0x7fe06c070860 msgr2=0x7fe06c15d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:33:31.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 shutdown_connections 2024-03-16T19:33:31.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:31.427+0000 7fe07085c640 1 -- 172.21.15.172:0/1592308418 wait complete. 2024-03-16T19:33:31.420 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-03-16T19:33:30.247148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002476"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.221290+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:32:36.221290+0000","last_peered":"2024-03-16T19:32:36.221290+0000","last_clean":"2024-03-16T19:32:36.221290+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:36.221290+0000","last_undegraded":"2024-03-16T19:32:36.221290+0000","last_fullsized":"2024-03-16T19:32:36.221290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.681904+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:32:36.681904+0000","last_peered":"2024-03-16T19:32:36.681904+0000","last_clean":"2024-03-16T19:32:36.681904+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:36.681904+0000","last_undegraded":"2024-03-16T19:32:36.681904+0000","last_fullsized":"2024-03-16T19:32:36.681904+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":112,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.236899+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:32:28.236899+0000","last_peered":"2024-03-16T19:32:28.236899+0000","last_clean":"2024-03-16T19:32:28.236899+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:28.236899+0000","last_undegraded":"2024-03-16T19:32:28.236899+0000","last_fullsized":"2024-03-16T19:32:28.236899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.687123+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:32:35.687123+0000","last_peered":"2024-03-16T19:32:35.687123+0000","last_clean":"2024-03-16T19:32:35.687123+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:35.687123+0000","last_undegraded":"2024-03-16T19:32:35.687123+0000","last_fullsized":"2024-03-16T19:32:35.687123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193158+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193158+0000","last_peered":"2024-03-16T19:32:42.193158+0000","last_clean":"2024-03-16T19:32:42.193158+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193158+0000","last_undegraded":"2024-03-16T19:32:42.193158+0000","last_fullsized":"2024-03-16T19:32:42.193158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501727+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501727+0000","last_peered":"2024-03-16T19:32:43.501727+0000","last_clean":"2024-03-16T19:32:43.501727+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501727+0000","last_undegraded":"2024-03-16T19:32:43.501727+0000","last_fullsized":"2024-03-16T19:32:43.501727+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180894+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180894+0000","last_peered":"2024-03-16T19:32:45.180894+0000","last_clean":"2024-03-16T19:32:45.180894+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180894+0000","last_undegraded":"2024-03-16T19:32:45.180894+0000","last_fullsized":"2024-03-16T19:32:45.180894+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142579+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142579+0000","last_peered":"2024-03-16T19:32:31.142579+0000","last_clean":"2024-03-16T19:32:31.142579+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142579+0000","last_undegraded":"2024-03-16T19:32:31.142579+0000","last_fullsized":"2024-03-16T19:32:31.142579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050015+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050015+0000","last_peered":"2024-03-16T19:32:30.050015+0000","last_clean":"2024-03-16T19:32:30.050015+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050015+0000","last_undegraded":"2024-03-16T19:32:30.050015+0000","last_fullsized":"2024-03-16T19:32:30.050015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906468+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906468+0000","last_peered":"2024-03-16T19:32:36.906468+0000","last_clean":"2024-03-16T19:32:36.906468+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906468+0000","last_undegraded":"2024-03-16T19:32:36.906468+0000","last_fullsized":"2024-03-16T19:32:36.906468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946447+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946447+0000","last_peered":"2024-03-16T19:32:37.946447+0000","last_clean":"2024-03-16T19:32:37.946447+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946447+0000","last_undegraded":"2024-03-16T19:32:37.946447+0000","last_fullsized":"2024-03-16T19:32:37.946447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012343+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012343+0000","last_peered":"2024-03-16T19:32:29.012343+0000","last_clean":"2024-03-16T19:32:29.012343+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012343+0000","last_undegraded":"2024-03-16T19:32:29.012343+0000","last_fullsized":"2024-03-16T19:32:29.012343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085947+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085947+0000","last_peered":"2024-03-16T19:32:34.085947+0000","last_clean":"2024-03-16T19:32:34.085947+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085947+0000","last_undegraded":"2024-03-16T19:32:34.085947+0000","last_fullsized":"2024-03-16T19:32:34.085947+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194777+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194777+0000","last_peered":"2024-03-16T19:32:31.194777+0000","last_clean":"2024-03-16T19:32:31.194777+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194777+0000","last_undegraded":"2024-03-16T19:32:31.194777+0000","last_fullsized":"2024-03-16T19:32:31.194777+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622130+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622130+0000","last_peered":"2024-03-16T19:32:30.622130+0000","last_clean":"2024-03-16T19:32:30.622130+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622130+0000","last_undegraded":"2024-03-16T19:32:30.622130+0000","last_fullsized":"2024-03-16T19:32:30.622130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.527986+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.527986+0000","last_peered":"2024-03-16T19:32:41.527986+0000","last_clean":"2024-03-16T19:32:41.527986+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.527986+0000","last_undegraded":"2024-03-16T19:32:41.527986+0000","last_fullsized":"2024-03-16T19:32:41.527986+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668862+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668862+0000","last_peered":"2024-03-16T19:32:33.668862+0000","last_clean":"2024-03-16T19:32:33.668862+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668862+0000","last_undegraded":"2024-03-16T19:32:33.668862+0000","last_fullsized":"2024-03-16T19:32:33.668862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":70,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158073+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158073+0000","last_peered":"2024-03-16T19:32:30.158073+0000","last_clean":"2024-03-16T19:32:30.158073+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158073+0000","last_undegraded":"2024-03-16T19:32:30.158073+0000","last_fullsized":"2024-03-16T19:32:30.158073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198963+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198963+0000","last_peered":"2024-03-16T19:32:43.198963+0000","last_clean":"2024-03-16T19:32:43.198963+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198963+0000","last_undegraded":"2024-03-16T19:32:43.198963+0000","last_fullsized":"2024-03-16T19:32:43.198963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874908+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874908+0000","last_peered":"2024-03-16T19:32:34.874908+0000","last_clean":"2024-03-16T19:32:34.874908+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874908+0000","last_undegraded":"2024-03-16T19:32:34.874908+0000","last_fullsized":"2024-03-16T19:32:34.874908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202018+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202018+0000","last_peered":"2024-03-16T19:32:26.202018+0000","last_clean":"2024-03-16T19:32:26.202018+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202018+0000","last_undegraded":"2024-03-16T19:32:26.202018+0000","last_fullsized":"2024-03-16T19:32:26.202018+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113786+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113786+0000","last_peered":"2024-03-16T19:32:31.113786+0000","last_clean":"2024-03-16T19:32:31.113786+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113786+0000","last_undegraded":"2024-03-16T19:32:31.113786+0000","last_fullsized":"2024-03-16T19:32:31.113786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189258+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189258+0000","last_peered":"2024-03-16T19:32:30.189258+0000","last_clean":"2024-03-16T19:32:30.189258+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189258+0000","last_undegraded":"2024-03-16T19:32:30.189258+0000","last_fullsized":"2024-03-16T19:32:30.189258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633685+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633685+0000","last_peered":"2024-03-16T19:32:38.633685+0000","last_clean":"2024-03-16T19:32:38.633685+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633685+0000","last_undegraded":"2024-03-16T19:32:38.633685+0000","last_fullsized":"2024-03-16T19:32:38.633685+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":171,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030001+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030001+0000","last_peered":"2024-03-16T19:32:36.030001+0000","last_clean":"2024-03-16T19:32:36.030001+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030001+0000","last_undegraded":"2024-03-16T19:32:36.030001+0000","last_fullsized":"2024-03-16T19:32:36.030001+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881947+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881947+0000","last_peered":"2024-03-16T19:32:35.881947+0000","last_clean":"2024-03-16T19:32:35.881947+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881947+0000","last_undegraded":"2024-03-16T19:32:35.881947+0000","last_fullsized":"2024-03-16T19:32:35.881947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651732+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651732+0000","last_peered":"2024-03-16T19:32:31.651732+0000","last_clean":"2024-03-16T19:32:31.651732+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651732+0000","last_undegraded":"2024-03-16T19:32:31.651732+0000","last_fullsized":"2024-03-16T19:32:31.651732+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":80,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.051201+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:32:35.051201+0000","last_peered":"2024-03-16T19:32:35.051201+0000","last_clean":"2024-03-16T19:32:35.051201+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:35.051201+0000","last_undegraded":"2024-03-16T19:32:35.051201+0000","last_fullsized":"2024-03-16T19:32:35.051201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":97,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209531+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209531+0000","last_peered":"2024-03-16T19:32:37.209531+0000","last_clean":"2024-03-16T19:32:37.209531+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209531+0000","last_undegraded":"2024-03-16T19:32:37.209531+0000","last_fullsized":"2024-03-16T19:32:37.209531+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.540693+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:32:40.540693+0000","last_peered":"2024-03-16T19:32:40.540693+0000","last_clean":"2024-03-16T19:32:40.540693+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:40.540693+0000","last_undegraded":"2024-03-16T19:32:40.540693+0000","last_fullsized":"2024-03-16T19:32:40.540693+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216652+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216652+0000","last_peered":"2024-03-16T19:32:27.216652+0000","last_clean":"2024-03-16T19:32:27.216652+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216652+0000","last_undegraded":"2024-03-16T19:32:27.216652+0000","last_fullsized":"2024-03-16T19:32:27.216652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.897651+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:32:40.897651+0000","last_peered":"2024-03-16T19:32:40.897651+0000","last_clean":"2024-03-16T19:32:40.897651+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:40.897651+0000","last_undegraded":"2024-03-16T19:32:40.897651+0000","last_fullsized":"2024-03-16T19:32:40.897651+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002739+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002739+0000","last_peered":"2024-03-16T19:32:28.002739+0000","last_clean":"2024-03-16T19:32:28.002739+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002739+0000","last_undegraded":"2024-03-16T19:32:28.002739+0000","last_fullsized":"2024-03-16T19:32:28.002739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207538+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207538+0000","last_peered":"2024-03-16T19:32:28.207538+0000","last_clean":"2024-03-16T19:32:28.207538+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207538+0000","last_undegraded":"2024-03-16T19:32:28.207538+0000","last_fullsized":"2024-03-16T19:32:28.207538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265363+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265363+0000","last_peered":"2024-03-16T19:32:35.265363+0000","last_clean":"2024-03-16T19:32:35.265363+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265363+0000","last_undegraded":"2024-03-16T19:32:35.265363+0000","last_fullsized":"2024-03-16T19:32:35.265363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913639+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913639+0000","last_peered":"2024-03-16T19:32:32.913639+0000","last_clean":"2024-03-16T19:32:32.913639+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913639+0000","last_undegraded":"2024-03-16T19:32:32.913639+0000","last_fullsized":"2024-03-16T19:32:32.913639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105259+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105259+0000","last_peered":"2024-03-16T19:32:32.105259+0000","last_clean":"2024-03-16T19:32:32.105259+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105259+0000","last_undegraded":"2024-03-16T19:32:32.105259+0000","last_fullsized":"2024-03-16T19:32:32.105259+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062743+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062743+0000","last_peered":"2024-03-16T19:32:38.062743+0000","last_clean":"2024-03-16T19:32:38.062743+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062743+0000","last_undegraded":"2024-03-16T19:32:38.062743+0000","last_fullsized":"2024-03-16T19:32:38.062743+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.685985+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.685985+0000","last_peered":"2024-03-16T19:32:28.685985+0000","last_clean":"2024-03-16T19:32:28.685985+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.685985+0000","last_undegraded":"2024-03-16T19:32:28.685985+0000","last_fullsized":"2024-03-16T19:32:28.685985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691219+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691219+0000","last_peered":"2024-03-16T19:32:32.691219+0000","last_clean":"2024-03-16T19:32:32.691219+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691219+0000","last_undegraded":"2024-03-16T19:32:32.691219+0000","last_fullsized":"2024-03-16T19:32:32.691219+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944568+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944568+0000","last_peered":"2024-03-16T19:32:30.944568+0000","last_clean":"2024-03-16T19:32:30.944568+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944568+0000","last_undegraded":"2024-03-16T19:32:30.944568+0000","last_fullsized":"2024-03-16T19:32:30.944568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706768+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706768+0000","last_peered":"2024-03-16T19:32:34.706768+0000","last_clean":"2024-03-16T19:32:34.706768+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706768+0000","last_undegraded":"2024-03-16T19:32:34.706768+0000","last_fullsized":"2024-03-16T19:32:34.706768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646626+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646626+0000","last_peered":"2024-03-16T19:32:29.646626+0000","last_clean":"2024-03-16T19:32:29.646626+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646626+0000","last_undegraded":"2024-03-16T19:32:29.646626+0000","last_fullsized":"2024-03-16T19:32:29.646626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223824+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223824+0000","last_peered":"2024-03-16T19:32:30.223824+0000","last_clean":"2024-03-16T19:32:30.223824+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223824+0000","last_undegraded":"2024-03-16T19:32:30.223824+0000","last_fullsized":"2024-03-16T19:32:30.223824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538373+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538373+0000","last_peered":"2024-03-16T19:32:26.538373+0000","last_clean":"2024-03-16T19:32:26.538373+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538373+0000","last_undegraded":"2024-03-16T19:32:26.538373+0000","last_fullsized":"2024-03-16T19:32:26.538373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":30,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076122+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076122+0000","last_peered":"2024-03-16T19:32:35.076122+0000","last_clean":"2024-03-16T19:32:35.076122+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076122+0000","last_undegraded":"2024-03-16T19:32:35.076122+0000","last_fullsized":"2024-03-16T19:32:35.076122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":246,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954015+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954015+0000","last_peered":"2024-03-16T19:32:38.954015+0000","last_clean":"2024-03-16T19:32:38.954015+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954015+0000","last_undegraded":"2024-03-16T19:32:38.954015+0000","last_fullsized":"2024-03-16T19:32:38.954015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179908+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179908+0000","last_peered":"2024-03-16T19:32:41.179908+0000","last_clean":"2024-03-16T19:32:41.179908+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179908+0000","last_undegraded":"2024-03-16T19:32:41.179908+0000","last_fullsized":"2024-03-16T19:32:41.179908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245654+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245654+0000","last_peered":"2024-03-16T19:32:36.245654+0000","last_clean":"2024-03-16T19:32:36.245654+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245654+0000","last_undegraded":"2024-03-16T19:32:36.245654+0000","last_fullsized":"2024-03-16T19:32:36.245654+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197866+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197866+0000","last_peered":"2024-03-16T19:32:27.197866+0000","last_clean":"2024-03-16T19:32:27.197866+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197866+0000","last_undegraded":"2024-03-16T19:32:27.197866+0000","last_fullsized":"2024-03-16T19:32:27.197866+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696935+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696935+0000","last_peered":"2024-03-16T19:32:36.696935+0000","last_clean":"2024-03-16T19:32:36.696935+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696935+0000","last_undegraded":"2024-03-16T19:32:36.696935+0000","last_fullsized":"2024-03-16T19:32:36.696935+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190715+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190715+0000","last_peered":"2024-03-16T19:32:37.190715+0000","last_clean":"2024-03-16T19:32:37.190715+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190715+0000","last_undegraded":"2024-03-16T19:32:37.190715+0000","last_fullsized":"2024-03-16T19:32:37.190715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938090+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938090+0000","last_peered":"2024-03-16T19:32:29.938090+0000","last_clean":"2024-03-16T19:32:29.938090+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938090+0000","last_undegraded":"2024-03-16T19:32:29.938090+0000","last_fullsized":"2024-03-16T19:32:29.938090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932702+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932702+0000","last_peered":"2024-03-16T19:32:39.932702+0000","last_clean":"2024-03-16T19:32:39.932702+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932702+0000","last_undegraded":"2024-03-16T19:32:39.932702+0000","last_fullsized":"2024-03-16T19:32:39.932702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":66,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:33:02.984706+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:33:02.984706+0000","last_peered":"2024-03-16T19:33:02.984706+0000","last_clean":"2024-03-16T19:33:02.984706+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:33:02.984706+0000","last_undegraded":"2024-03-16T19:33:02.984706+0000","last_fullsized":"2024-03-16T19:33:02.984706+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 48s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176411+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176411+0000","last_peered":"2024-03-16T19:32:38.176411+0000","last_clean":"2024-03-16T19:32:38.176411+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176411+0000","last_undegraded":"2024-03-16T19:32:38.176411+0000","last_fullsized":"2024-03-16T19:32:38.176411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":214,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982041+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982041+0000","last_peered":"2024-03-16T19:32:31.982041+0000","last_clean":"2024-03-16T19:32:31.982041+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982041+0000","last_undegraded":"2024-03-16T19:32:31.982041+0000","last_fullsized":"2024-03-16T19:32:31.982041+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149057+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149057+0000","last_peered":"2024-03-16T19:32:39.149057+0000","last_clean":"2024-03-16T19:32:39.149057+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149057+0000","last_undegraded":"2024-03-16T19:32:39.149057+0000","last_fullsized":"2024-03-16T19:32:39.149057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":65,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108534+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108534+0000","last_peered":"2024-03-16T19:32:32.108534+0000","last_clean":"2024-03-16T19:32:32.108534+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108534+0000","last_undegraded":"2024-03-16T19:32:32.108534+0000","last_fullsized":"2024-03-16T19:32:32.108534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059643+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059643+0000","last_peered":"2024-03-16T19:32:31.059643+0000","last_clean":"2024-03-16T19:32:31.059643+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059643+0000","last_undegraded":"2024-03-16T19:32:31.059643+0000","last_fullsized":"2024-03-16T19:32:31.059643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568517+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568517+0000","last_peered":"2024-03-16T19:32:27.568517+0000","last_clean":"2024-03-16T19:32:27.568517+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568517+0000","last_undegraded":"2024-03-16T19:32:27.568517+0000","last_fullsized":"2024-03-16T19:32:27.568517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651622+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651622+0000","last_peered":"2024-03-16T19:32:37.651622+0000","last_clean":"2024-03-16T19:32:37.651622+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651622+0000","last_undegraded":"2024-03-16T19:32:37.651622+0000","last_fullsized":"2024-03-16T19:32:37.651622+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205810+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205810+0000","last_peered":"2024-03-16T19:32:28.205810+0000","last_clean":"2024-03-16T19:32:28.205810+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205810+0000","last_undegraded":"2024-03-16T19:32:28.205810+0000","last_fullsized":"2024-03-16T19:32:28.205810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":165,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166583+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166583+0000","last_peered":"2024-03-16T19:32:40.166583+0000","last_clean":"2024-03-16T19:32:40.166583+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166583+0000","last_undegraded":"2024-03-16T19:32:40.166583+0000","last_fullsized":"2024-03-16T19:32:40.166583+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":375,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:47.115779+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:32:47.115779+0000","last_peered":"2024-03-16T19:32:47.115779+0000","last_clean":"2024-03-16T19:32:47.115779+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:47.115779+0000","last_undegraded":"2024-03-16T19:32:47.115779+0000","last_fullsized":"2024-03-16T19:32:47.115779+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.532556+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:32:42.532556+0000","last_peered":"2024-03-16T19:32:42.532556+0000","last_clean":"2024-03-16T19:32:42.532556+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:42.532556+0000","last_undegraded":"2024-03-16T19:32:42.532556+0000","last_fullsized":"2024-03-16T19:32:42.532556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.219236+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:32:29.219236+0000","last_peered":"2024-03-16T19:32:29.219236+0000","last_clean":"2024-03-16T19:32:29.219236+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:29.219236+0000","last_undegraded":"2024-03-16T19:32:29.219236+0000","last_fullsized":"2024-03-16T19:32:29.219236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.162302+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:32:34.162302+0000","last_peered":"2024-03-16T19:32:34.162302+0000","last_clean":"2024-03-16T19:32:34.162302+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:34.162302+0000","last_undegraded":"2024-03-16T19:32:34.162302+0000","last_fullsized":"2024-03-16T19:32:34.162302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240482+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240482+0000","last_peered":"2024-03-16T19:32:34.240482+0000","last_clean":"2024-03-16T19:32:34.240482+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240482+0000","last_undegraded":"2024-03-16T19:32:34.240482+0000","last_fullsized":"2024-03-16T19:32:34.240482+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.144993+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.144993+0000","last_peered":"2024-03-16T19:32:33.144993+0000","last_clean":"2024-03-16T19:32:33.144993+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.144993+0000","last_undegraded":"2024-03-16T19:32:33.144993+0000","last_fullsized":"2024-03-16T19:32:33.144993+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":193,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:33:08.239070+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:33:08.239070+0000","last_peered":"2024-03-16T19:33:08.239070+0000","last_clean":"2024-03-16T19:33:08.239070+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:33:08.239070+0000","last_undegraded":"2024-03-16T19:33:08.239070+0000","last_fullsized":"2024-03-16T19:33:08.239070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 47s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.692376+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:32:30.692376+0000","last_peered":"2024-03-16T19:32:30.692376+0000","last_clean":"2024-03-16T19:32:30.692376+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:30.692376+0000","last_undegraded":"2024-03-16T19:32:30.692376+0000","last_fullsized":"2024-03-16T19:32:30.692376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.998665+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:32:30.998665+0000","last_peered":"2024-03-16T19:32:30.998665+0000","last_clean":"2024-03-16T19:32:30.998665+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:30.998665+0000","last_undegraded":"2024-03-16T19:32:30.998665+0000","last_fullsized":"2024-03-16T19:32:30.998665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.721499+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:32:37.721499+0000","last_peered":"2024-03-16T19:32:37.721499+0000","last_clean":"2024-03-16T19:32:37.721499+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:37.721499+0000","last_undegraded":"2024-03-16T19:32:37.721499+0000","last_fullsized":"2024-03-16T19:32:37.721499+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.589644+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:32:39.589644+0000","last_peered":"2024-03-16T19:32:39.589644+0000","last_clean":"2024-03-16T19:32:39.589644+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:39.589644+0000","last_undegraded":"2024-03-16T19:32:39.589644+0000","last_fullsized":"2024-03-16T19:32:39.589644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:46.142279+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:32:46.142279+0000","last_peered":"2024-03-16T19:32:46.142279+0000","last_clean":"2024-03-16T19:32:46.142279+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:46.142279+0000","last_undegraded":"2024-03-16T19:32:46.142279+0000","last_fullsized":"2024-03-16T19:32:46.142279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":193,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:33:08.239118+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:33:08.239118+0000","last_peered":"2024-03-16T19:33:08.239118+0000","last_clean":"2024-03-16T19:33:08.239118+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:33:08.239118+0000","last_undegraded":"2024-03-16T19:33:08.239118+0000","last_fullsized":"2024-03-16T19:33:08.239118+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 48s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586853+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586853+0000","last_peered":"2024-03-16T19:32:29.586853+0000","last_clean":"2024-03-16T19:32:29.586853+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586853+0000","last_undegraded":"2024-03-16T19:32:29.586853+0000","last_fullsized":"2024-03-16T19:32:29.586853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.125468+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:32:33.125468+0000","last_peered":"2024-03-16T19:32:33.125468+0000","last_clean":"2024-03-16T19:32:33.125468+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:33.125468+0000","last_undegraded":"2024-03-16T19:32:33.125468+0000","last_fullsized":"2024-03-16T19:32:33.125468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910633+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910633+0000","last_peered":"2024-03-16T19:32:31.910633+0000","last_clean":"2024-03-16T19:32:31.910633+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910633+0000","last_undegraded":"2024-03-16T19:32:31.910633+0000","last_fullsized":"2024-03-16T19:32:31.910633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188001+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188001+0000","last_peered":"2024-03-16T19:32:32.188001+0000","last_clean":"2024-03-16T19:32:32.188001+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188001+0000","last_undegraded":"2024-03-16T19:32:32.188001+0000","last_fullsized":"2024-03-16T19:32:32.188001+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190102+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190102+0000","last_peered":"2024-03-16T19:32:44.190102+0000","last_clean":"2024-03-16T19:32:44.190102+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190102+0000","last_undegraded":"2024-03-16T19:32:44.190102+0000","last_fullsized":"2024-03-16T19:32:44.190102+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026127+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026127+0000","last_peered":"2024-03-16T19:32:30.026127+0000","last_clean":"2024-03-16T19:32:30.026127+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026127+0000","last_undegraded":"2024-03-16T19:32:30.026127+0000","last_fullsized":"2024-03-16T19:32:30.026127+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604768+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604768+0000","last_peered":"2024-03-16T19:32:28.604768+0000","last_clean":"2024-03-16T19:32:28.604768+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604768+0000","last_undegraded":"2024-03-16T19:32:28.604768+0000","last_fullsized":"2024-03-16T19:32:28.604768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:33:07.985094+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:33:07.985094+0000","last_peered":"2024-03-16T19:33:07.985094+0000","last_clean":"2024-03-16T19:33:07.985094+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:33:07.985094+0000","last_undegraded":"2024-03-16T19:33:07.985094+0000","last_fullsized":"2024-03-16T19:33:07.985094+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 48s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699866+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699866+0000","last_peered":"2024-03-16T19:32:33.699866+0000","last_clean":"2024-03-16T19:32:33.699866+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699866+0000","last_undegraded":"2024-03-16T19:32:33.699866+0000","last_fullsized":"2024-03-16T19:32:33.699866+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030418+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030418+0000","last_peered":"2024-03-16T19:32:39.030418+0000","last_clean":"2024-03-16T19:32:39.030418+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030418+0000","last_undegraded":"2024-03-16T19:32:39.030418+0000","last_fullsized":"2024-03-16T19:32:39.030418+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152403+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152403+0000","last_peered":"2024-03-16T19:32:33.152403+0000","last_clean":"2024-03-16T19:32:33.152403+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152403+0000","last_undegraded":"2024-03-16T19:32:33.152403+0000","last_fullsized":"2024-03-16T19:32:33.152403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889893+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889893+0000","last_peered":"2024-03-16T19:32:33.889893+0000","last_clean":"2024-03-16T19:32:33.889893+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889893+0000","last_undegraded":"2024-03-16T19:32:33.889893+0000","last_fullsized":"2024-03-16T19:32:33.889893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695467+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695467+0000","last_peered":"2024-03-16T19:32:31.695467+0000","last_clean":"2024-03-16T19:32:31.695467+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695467+0000","last_undegraded":"2024-03-16T19:32:31.695467+0000","last_fullsized":"2024-03-16T19:32:31.695467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102602+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102602+0000","last_peered":"2024-03-16T19:32:32.102602+0000","last_clean":"2024-03-16T19:32:32.102602+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102602+0000","last_undegraded":"2024-03-16T19:32:32.102602+0000","last_fullsized":"2024-03-16T19:32:32.102602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696329+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696329+0000","last_peered":"2024-03-16T19:32:32.696329+0000","last_clean":"2024-03-16T19:32:32.696329+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696329+0000","last_undegraded":"2024-03-16T19:32:32.696329+0000","last_fullsized":"2024-03-16T19:32:32.696329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177533+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177533+0000","last_peered":"2024-03-16T19:32:35.177533+0000","last_clean":"2024-03-16T19:32:35.177533+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177533+0000","last_undegraded":"2024-03-16T19:32:35.177533+0000","last_fullsized":"2024-03-16T19:32:35.177533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230753+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230753+0000","last_peered":"2024-03-16T19:32:33.230753+0000","last_clean":"2024-03-16T19:32:33.230753+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230753+0000","last_undegraded":"2024-03-16T19:32:33.230753+0000","last_fullsized":"2024-03-16T19:32:33.230753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.046537+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:32:40.046537+0000","last_peered":"2024-03-16T19:32:40.046537+0000","last_clean":"2024-03-16T19:32:40.046537+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:40.046537+0000","last_undegraded":"2024-03-16T19:32:40.046537+0000","last_fullsized":"2024-03-16T19:32:40.046537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":175,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.063436+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:32:37.063436+0000","last_peered":"2024-03-16T19:32:37.063436+0000","last_clean":"2024-03-16T19:32:37.063436+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:37.063436+0000","last_undegraded":"2024-03-16T19:32:37.063436+0000","last_fullsized":"2024-03-16T19:32:37.063436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272623+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272623+0000","last_peered":"2024-03-16T19:32:29.272623+0000","last_clean":"2024-03-16T19:32:29.272623+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272623+0000","last_undegraded":"2024-03-16T19:32:29.272623+0000","last_fullsized":"2024-03-16T19:32:29.272623+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:33:31.451 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:31.451 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:31.451 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:31.451 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:31.452 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:33:51.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:33:51.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.595+0000 7f0f51e6b640 1 Processor -- start 2024-03-16T19:33:51.574 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.595+0000 7f0f51e6b640 1 -- start start 2024-03-16T19:33:51.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f51e6b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0f4c06bef0 0x7f0f4c154660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:51.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f51e6b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c0cc720 con 0x7f0f4c06bb10 2024-03-16T19:33:51.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f51e6b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c0cc9c0 con 0x7f0f4c0cb560 2024-03-16T19:33:51.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f51e6b640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f4c15de30 con 0x7f0f4c06bef0 2024-03-16T19:33:51.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f4b7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f0f4c06bb10 0x7f0f4c0cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:52512/0 (socket says 172.21.15.172:52512) 2024-03-16T19:33:51.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f4affd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0f4c06bef0 0x7f0f4c154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:51.575 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f4b7fe640 1 -- 172.21.15.172:0/354163127 learned_addr learned my addr 172.21.15.172:0/354163127 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:33:51.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f4affd640 1 -- 172.21.15.172:0/354163127 >> v1:172.21.15.174:6789/0 conn(0x7f0f4c0cb560 legacy=0x7f0f4c1651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:51.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f4affd640 1 -- 172.21.15.172:0/354163127 >> v1:172.21.15.172:6789/0 conn(0x7f0f4c06bb10 legacy=0x7f0f4c0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:51.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f4a7fc640 1 -- 172.21.15.172:0/354163127 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668263598 0 0) 0x7f0f4c0cc9c0 con 0x7f0f4c0cb560 2024-03-16T19:33:51.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.596+0000 7f0f4affd640 1 -- 172.21.15.172:0/354163127 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c15e110 con 0x7f0f4c06bef0 2024-03-16T19:33:51.576 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.597+0000 7f0f4affd640 1 --2- 172.21.15.172:0/354163127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bef0 0x7f0f4c154660 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0f400097a0 tx=0x7f0f4002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4507f90b08f8d99 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:33:51.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.597+0000 7f0f4a7fc640 1 -- 172.21.15.172:0/354163127 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4002f810 con 0x7f0f4c06bef0 2024-03-16T19:33:51.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.597+0000 7f0f4a7fc640 1 -- 172.21.15.172:0/354163127 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4002f970 con 0x7f0f4c06bef0 2024-03-16T19:33:51.577 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.597+0000 7f0f4a7fc640 1 -- 172.21.15.172:0/354163127 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4002fd00 con 0x7f0f4c06bef0 2024-03-16T19:33:51.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.598+0000 7f0f51e6b640 1 -- 172.21.15.172:0/354163127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bef0 msgr2=0x7f0f4c154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:51.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.598+0000 7f0f51e6b640 1 --2- 172.21.15.172:0/354163127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bef0 0x7f0f4c154660 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0f400097a0 tx=0x7f0f4002b9f0 comp rx=0 tx=0).stop 2024-03-16T19:33:51.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.598+0000 7f0f51e6b640 1 -- 172.21.15.172:0/354163127 shutdown_connections 2024-03-16T19:33:51.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.598+0000 7f0f51e6b640 1 --2- 172.21.15.172:0/354163127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bef0 0x7f0f4c154660 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:51.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.598+0000 7f0f51e6b640 1 -- 172.21.15.172:0/354163127 >> 172.21.15.172:0/354163127 conn(0x7f0f4c067250 msgr2=0x7f0f4c0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:33:51.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.598+0000 7f0f51e6b640 1 -- 172.21.15.172:0/354163127 shutdown_connections 2024-03-16T19:33:51.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.598+0000 7f0f51e6b640 1 -- 172.21.15.172:0/354163127 wait complete. 2024-03-16T19:33:51.578 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.600+0000 7f0f51e6b640 1 Processor -- start 2024-03-16T19:33:51.579 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.600+0000 7f0f51e6b640 1 -- start start 2024-03-16T19:33:51.581 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f51e6b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bb10 0x7f0f4c0c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:51.581 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f51e6b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f4c06bef0 0x7f0f4c157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:51.581 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f51e6b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0cb560 0x7f0f4c141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f51e6b640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c1f0a00 con 0x7f0f4c06bb10 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f51e6b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c157cd0 con 0x7f0f4c0cb560 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f51e6b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c155380 con 0x7f0f4c06bef0 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bb10 0x7f0f4c0c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4affd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f4c06bef0 0x7f0f4c157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4affd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f4c06bef0 0x7f0f4c157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:56104/0 (socket says 172.21.15.172:56104) 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4affd640 1 -- 172.21.15.172:0/2523776108 learned_addr learned my addr 172.21.15.172:0/2523776108 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4b7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bb10 0x7f0f4c0c6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59154/0 (socket says 172.21.15.172:59154) 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4bfff640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0cb560 0x7f0f4c141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4affd640 1 -- 172.21.15.172:0/2523776108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0cb560 msgr2=0x7f0f4c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4affd640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0cb560 0x7f0f4c141aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:51.582 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4affd640 1 -- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bb10 msgr2=0x7f0f4c0c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4affd640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bb10 0x7f0f4c0c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4affd640 1 -- 172.21.15.172:0/2523776108 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f40009400 con 0x7f0f4c06bef0 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.601+0000 7f0f4bfff640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0cb560 0x7f0f4c141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.602+0000 7f0f4b7fe640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bb10 0x7f0f4c0c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.602+0000 7f0f4affd640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f4c06bef0 0x7f0f4c157290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009890 tx=0x7f0f40004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.602+0000 7f0f33fff640 1 -- 172.21.15.172:0/2523776108 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4003c040 con 0x7f0f4c06bef0 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.602+0000 7f0f33fff640 1 -- 172.21.15.172:0/2523776108 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f400379f0 con 0x7f0f4c06bef0 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.602+0000 7f0f33fff640 1 -- 172.21.15.172:0/2523776108 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f40037da0 con 0x7f0f4c06bef0 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.602+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4c1580f0 con 0x7f0f4c06bef0 2024-03-16T19:33:51.583 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.602+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4c155900 con 0x7f0f4c06bef0 2024-03-16T19:33:51.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.603+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4c1f0b40 con 0x7f0f4c06bef0 2024-03-16T19:33:51.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.604+0000 7f0f33fff640 1 -- 172.21.15.172:0/2523776108 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0f400403e0 con 0x7f0f4c06bef0 2024-03-16T19:33:51.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.604+0000 7f0f33fff640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0f3806b3b0 0x7f0f3806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:33:51.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.604+0000 7f0f33fff640 1 -- 172.21.15.172:0/2523776108 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f0f400abbf0 con 0x7f0f4c06bef0 2024-03-16T19:33:51.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.604+0000 7f0f4b7fe640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0f3806b3b0 0x7f0f3806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:33:51.584 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.605+0000 7f0f4b7fe640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0f3806b3b0 0x7f0f3806d870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c005880 tx=0x7f0f3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:33:51.587 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.608+0000 7f0f33fff640 1 -- 172.21.15.172:0/2523776108 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f4c1f0b40 con 0x7f0f4c06bef0 2024-03-16T19:33:51.901 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.922+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f4c15e9b0 con 0x7f0f3806b3b0 2024-03-16T19:33:51.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.933+0000 7f0f33fff640 1 -- 172.21.15.172:0/2523776108 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252919 (secure 0 0 0) 0x7f0f3c001970 con 0x7f0f3806b3b0 2024-03-16T19:33:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:33:51.916 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:33:51.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0f3806b3b0 msgr2=0x7f0f3806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0f3806b3b0 0x7f0f3806d870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c005880 tx=0x7f0f3c009040 comp rx=0 tx=0).stop 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f4c06bef0 msgr2=0x7f0f4c157290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f4c06bef0 0x7f0f4c157290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009890 tx=0x7f0f40004730 comp rx=0 tx=0).stop 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 shutdown_connections 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0cb560 0x7f0f4c141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0f3806b3b0 0x7f0f3806d870 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f4c06bef0 0x7f0f4c157290 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 --2- 172.21.15.172:0/2523776108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f4c06bb10 0x7f0f4c0c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 >> 172.21.15.172:0/2523776108 conn(0x7f0f4c067250 msgr2=0x7f0f4c14a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 shutdown_connections 2024-03-16T19:33:51.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:33:51.937+0000 7f0f51e6b640 1 -- 172.21.15.172:0/2523776108 wait complete. 2024-03-16T19:33:51.931 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-03-16T19:33:50.251092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002502"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.221338+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:32:36.221338+0000","last_peered":"2024-03-16T19:32:36.221338+0000","last_clean":"2024-03-16T19:32:36.221338+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:36.221338+0000","last_undegraded":"2024-03-16T19:32:36.221338+0000","last_fullsized":"2024-03-16T19:32:36.221338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.681944+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:32:36.681944+0000","last_peered":"2024-03-16T19:32:36.681944+0000","last_clean":"2024-03-16T19:32:36.681944+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:36.681944+0000","last_undegraded":"2024-03-16T19:32:36.681944+0000","last_fullsized":"2024-03-16T19:32:36.681944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":113,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.236924+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:32:28.236924+0000","last_peered":"2024-03-16T19:32:28.236924+0000","last_clean":"2024-03-16T19:32:28.236924+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:28.236924+0000","last_undegraded":"2024-03-16T19:32:28.236924+0000","last_fullsized":"2024-03-16T19:32:28.236924+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.687123+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:32:35.687123+0000","last_peered":"2024-03-16T19:32:35.687123+0000","last_clean":"2024-03-16T19:32:35.687123+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:35.687123+0000","last_undegraded":"2024-03-16T19:32:35.687123+0000","last_fullsized":"2024-03-16T19:32:35.687123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193158+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193158+0000","last_peered":"2024-03-16T19:32:42.193158+0000","last_clean":"2024-03-16T19:32:42.193158+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193158+0000","last_undegraded":"2024-03-16T19:32:42.193158+0000","last_fullsized":"2024-03-16T19:32:42.193158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501727+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501727+0000","last_peered":"2024-03-16T19:32:43.501727+0000","last_clean":"2024-03-16T19:32:43.501727+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501727+0000","last_undegraded":"2024-03-16T19:32:43.501727+0000","last_fullsized":"2024-03-16T19:32:43.501727+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180894+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180894+0000","last_peered":"2024-03-16T19:32:45.180894+0000","last_clean":"2024-03-16T19:32:45.180894+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180894+0000","last_undegraded":"2024-03-16T19:32:45.180894+0000","last_fullsized":"2024-03-16T19:32:45.180894+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142579+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142579+0000","last_peered":"2024-03-16T19:32:31.142579+0000","last_clean":"2024-03-16T19:32:31.142579+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142579+0000","last_undegraded":"2024-03-16T19:32:31.142579+0000","last_fullsized":"2024-03-16T19:32:31.142579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050015+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050015+0000","last_peered":"2024-03-16T19:32:30.050015+0000","last_clean":"2024-03-16T19:32:30.050015+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050015+0000","last_undegraded":"2024-03-16T19:32:30.050015+0000","last_fullsized":"2024-03-16T19:32:30.050015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906468+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906468+0000","last_peered":"2024-03-16T19:32:36.906468+0000","last_clean":"2024-03-16T19:32:36.906468+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906468+0000","last_undegraded":"2024-03-16T19:32:36.906468+0000","last_fullsized":"2024-03-16T19:32:36.906468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946447+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946447+0000","last_peered":"2024-03-16T19:32:37.946447+0000","last_clean":"2024-03-16T19:32:37.946447+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946447+0000","last_undegraded":"2024-03-16T19:32:37.946447+0000","last_fullsized":"2024-03-16T19:32:37.946447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012343+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012343+0000","last_peered":"2024-03-16T19:32:29.012343+0000","last_clean":"2024-03-16T19:32:29.012343+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012343+0000","last_undegraded":"2024-03-16T19:32:29.012343+0000","last_fullsized":"2024-03-16T19:32:29.012343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085947+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085947+0000","last_peered":"2024-03-16T19:32:34.085947+0000","last_clean":"2024-03-16T19:32:34.085947+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085947+0000","last_undegraded":"2024-03-16T19:32:34.085947+0000","last_fullsized":"2024-03-16T19:32:34.085947+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194777+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194777+0000","last_peered":"2024-03-16T19:32:31.194777+0000","last_clean":"2024-03-16T19:32:31.194777+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194777+0000","last_undegraded":"2024-03-16T19:32:31.194777+0000","last_fullsized":"2024-03-16T19:32:31.194777+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622130+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622130+0000","last_peered":"2024-03-16T19:32:30.622130+0000","last_clean":"2024-03-16T19:32:30.622130+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622130+0000","last_undegraded":"2024-03-16T19:32:30.622130+0000","last_fullsized":"2024-03-16T19:32:30.622130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.527986+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.527986+0000","last_peered":"2024-03-16T19:32:41.527986+0000","last_clean":"2024-03-16T19:32:41.527986+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.527986+0000","last_undegraded":"2024-03-16T19:32:41.527986+0000","last_fullsized":"2024-03-16T19:32:41.527986+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668862+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668862+0000","last_peered":"2024-03-16T19:32:33.668862+0000","last_clean":"2024-03-16T19:32:33.668862+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668862+0000","last_undegraded":"2024-03-16T19:32:33.668862+0000","last_fullsized":"2024-03-16T19:32:33.668862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":70,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158073+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158073+0000","last_peered":"2024-03-16T19:32:30.158073+0000","last_clean":"2024-03-16T19:32:30.158073+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158073+0000","last_undegraded":"2024-03-16T19:32:30.158073+0000","last_fullsized":"2024-03-16T19:32:30.158073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198963+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198963+0000","last_peered":"2024-03-16T19:32:43.198963+0000","last_clean":"2024-03-16T19:32:43.198963+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198963+0000","last_undegraded":"2024-03-16T19:32:43.198963+0000","last_fullsized":"2024-03-16T19:32:43.198963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874908+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874908+0000","last_peered":"2024-03-16T19:32:34.874908+0000","last_clean":"2024-03-16T19:32:34.874908+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874908+0000","last_undegraded":"2024-03-16T19:32:34.874908+0000","last_fullsized":"2024-03-16T19:32:34.874908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202018+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202018+0000","last_peered":"2024-03-16T19:32:26.202018+0000","last_clean":"2024-03-16T19:32:26.202018+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202018+0000","last_undegraded":"2024-03-16T19:32:26.202018+0000","last_fullsized":"2024-03-16T19:32:26.202018+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113786+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113786+0000","last_peered":"2024-03-16T19:32:31.113786+0000","last_clean":"2024-03-16T19:32:31.113786+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113786+0000","last_undegraded":"2024-03-16T19:32:31.113786+0000","last_fullsized":"2024-03-16T19:32:31.113786+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189280+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189280+0000","last_peered":"2024-03-16T19:32:30.189280+0000","last_clean":"2024-03-16T19:32:30.189280+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189280+0000","last_undegraded":"2024-03-16T19:32:30.189280+0000","last_fullsized":"2024-03-16T19:32:30.189280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633708+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633708+0000","last_peered":"2024-03-16T19:32:38.633708+0000","last_clean":"2024-03-16T19:32:38.633708+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633708+0000","last_undegraded":"2024-03-16T19:32:38.633708+0000","last_fullsized":"2024-03-16T19:32:38.633708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":172,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030046+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030046+0000","last_peered":"2024-03-16T19:32:36.030046+0000","last_clean":"2024-03-16T19:32:36.030046+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030046+0000","last_undegraded":"2024-03-16T19:32:36.030046+0000","last_fullsized":"2024-03-16T19:32:36.030046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881976+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881976+0000","last_peered":"2024-03-16T19:32:35.881976+0000","last_clean":"2024-03-16T19:32:35.881976+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881976+0000","last_undegraded":"2024-03-16T19:32:35.881976+0000","last_fullsized":"2024-03-16T19:32:35.881976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651765+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651765+0000","last_peered":"2024-03-16T19:32:31.651765+0000","last_clean":"2024-03-16T19:32:31.651765+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651765+0000","last_undegraded":"2024-03-16T19:32:31.651765+0000","last_fullsized":"2024-03-16T19:32:31.651765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.051224+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:32:35.051224+0000","last_peered":"2024-03-16T19:32:35.051224+0000","last_clean":"2024-03-16T19:32:35.051224+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:35.051224+0000","last_undegraded":"2024-03-16T19:32:35.051224+0000","last_fullsized":"2024-03-16T19:32:35.051224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":97,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209531+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209531+0000","last_peered":"2024-03-16T19:32:37.209531+0000","last_clean":"2024-03-16T19:32:37.209531+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209531+0000","last_undegraded":"2024-03-16T19:32:37.209531+0000","last_fullsized":"2024-03-16T19:32:37.209531+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.540714+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:32:40.540714+0000","last_peered":"2024-03-16T19:32:40.540714+0000","last_clean":"2024-03-16T19:32:40.540714+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:40.540714+0000","last_undegraded":"2024-03-16T19:32:40.540714+0000","last_fullsized":"2024-03-16T19:32:40.540714+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216652+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216652+0000","last_peered":"2024-03-16T19:32:27.216652+0000","last_clean":"2024-03-16T19:32:27.216652+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216652+0000","last_undegraded":"2024-03-16T19:32:27.216652+0000","last_fullsized":"2024-03-16T19:32:27.216652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.897672+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:32:40.897672+0000","last_peered":"2024-03-16T19:32:40.897672+0000","last_clean":"2024-03-16T19:32:40.897672+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:40.897672+0000","last_undegraded":"2024-03-16T19:32:40.897672+0000","last_fullsized":"2024-03-16T19:32:40.897672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002739+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002739+0000","last_peered":"2024-03-16T19:32:28.002739+0000","last_clean":"2024-03-16T19:32:28.002739+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002739+0000","last_undegraded":"2024-03-16T19:32:28.002739+0000","last_fullsized":"2024-03-16T19:32:28.002739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207538+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207538+0000","last_peered":"2024-03-16T19:32:28.207538+0000","last_clean":"2024-03-16T19:32:28.207538+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207538+0000","last_undegraded":"2024-03-16T19:32:28.207538+0000","last_fullsized":"2024-03-16T19:32:28.207538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265363+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265363+0000","last_peered":"2024-03-16T19:32:35.265363+0000","last_clean":"2024-03-16T19:32:35.265363+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265363+0000","last_undegraded":"2024-03-16T19:32:35.265363+0000","last_fullsized":"2024-03-16T19:32:35.265363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913639+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913639+0000","last_peered":"2024-03-16T19:32:32.913639+0000","last_clean":"2024-03-16T19:32:32.913639+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913639+0000","last_undegraded":"2024-03-16T19:32:32.913639+0000","last_fullsized":"2024-03-16T19:32:32.913639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105259+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105259+0000","last_peered":"2024-03-16T19:32:32.105259+0000","last_clean":"2024-03-16T19:32:32.105259+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105259+0000","last_undegraded":"2024-03-16T19:32:32.105259+0000","last_fullsized":"2024-03-16T19:32:32.105259+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062743+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062743+0000","last_peered":"2024-03-16T19:32:38.062743+0000","last_clean":"2024-03-16T19:32:38.062743+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062743+0000","last_undegraded":"2024-03-16T19:32:38.062743+0000","last_fullsized":"2024-03-16T19:32:38.062743+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.685985+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.685985+0000","last_peered":"2024-03-16T19:32:28.685985+0000","last_clean":"2024-03-16T19:32:28.685985+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.685985+0000","last_undegraded":"2024-03-16T19:32:28.685985+0000","last_fullsized":"2024-03-16T19:32:28.685985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691219+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691219+0000","last_peered":"2024-03-16T19:32:32.691219+0000","last_clean":"2024-03-16T19:32:32.691219+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691219+0000","last_undegraded":"2024-03-16T19:32:32.691219+0000","last_fullsized":"2024-03-16T19:32:32.691219+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944568+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944568+0000","last_peered":"2024-03-16T19:32:30.944568+0000","last_clean":"2024-03-16T19:32:30.944568+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944568+0000","last_undegraded":"2024-03-16T19:32:30.944568+0000","last_fullsized":"2024-03-16T19:32:30.944568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706768+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706768+0000","last_peered":"2024-03-16T19:32:34.706768+0000","last_clean":"2024-03-16T19:32:34.706768+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706768+0000","last_undegraded":"2024-03-16T19:32:34.706768+0000","last_fullsized":"2024-03-16T19:32:34.706768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646626+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646626+0000","last_peered":"2024-03-16T19:32:29.646626+0000","last_clean":"2024-03-16T19:32:29.646626+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646626+0000","last_undegraded":"2024-03-16T19:32:29.646626+0000","last_fullsized":"2024-03-16T19:32:29.646626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223824+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223824+0000","last_peered":"2024-03-16T19:32:30.223824+0000","last_clean":"2024-03-16T19:32:30.223824+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223824+0000","last_undegraded":"2024-03-16T19:32:30.223824+0000","last_fullsized":"2024-03-16T19:32:30.223824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538373+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538373+0000","last_peered":"2024-03-16T19:32:26.538373+0000","last_clean":"2024-03-16T19:32:26.538373+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538373+0000","last_undegraded":"2024-03-16T19:32:26.538373+0000","last_fullsized":"2024-03-16T19:32:26.538373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":30,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076122+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076122+0000","last_peered":"2024-03-16T19:32:35.076122+0000","last_clean":"2024-03-16T19:32:35.076122+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076122+0000","last_undegraded":"2024-03-16T19:32:35.076122+0000","last_fullsized":"2024-03-16T19:32:35.076122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":246,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954015+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954015+0000","last_peered":"2024-03-16T19:32:38.954015+0000","last_clean":"2024-03-16T19:32:38.954015+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954015+0000","last_undegraded":"2024-03-16T19:32:38.954015+0000","last_fullsized":"2024-03-16T19:32:38.954015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179908+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179908+0000","last_peered":"2024-03-16T19:32:41.179908+0000","last_clean":"2024-03-16T19:32:41.179908+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179908+0000","last_undegraded":"2024-03-16T19:32:41.179908+0000","last_fullsized":"2024-03-16T19:32:41.179908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245654+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245654+0000","last_peered":"2024-03-16T19:32:36.245654+0000","last_clean":"2024-03-16T19:32:36.245654+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245654+0000","last_undegraded":"2024-03-16T19:32:36.245654+0000","last_fullsized":"2024-03-16T19:32:36.245654+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197866+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197866+0000","last_peered":"2024-03-16T19:32:27.197866+0000","last_clean":"2024-03-16T19:32:27.197866+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197866+0000","last_undegraded":"2024-03-16T19:32:27.197866+0000","last_fullsized":"2024-03-16T19:32:27.197866+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696935+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696935+0000","last_peered":"2024-03-16T19:32:36.696935+0000","last_clean":"2024-03-16T19:32:36.696935+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696935+0000","last_undegraded":"2024-03-16T19:32:36.696935+0000","last_fullsized":"2024-03-16T19:32:36.696935+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190715+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190715+0000","last_peered":"2024-03-16T19:32:37.190715+0000","last_clean":"2024-03-16T19:32:37.190715+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190715+0000","last_undegraded":"2024-03-16T19:32:37.190715+0000","last_fullsized":"2024-03-16T19:32:37.190715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":189,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938090+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938090+0000","last_peered":"2024-03-16T19:32:29.938090+0000","last_clean":"2024-03-16T19:32:29.938090+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938090+0000","last_undegraded":"2024-03-16T19:32:29.938090+0000","last_fullsized":"2024-03-16T19:32:29.938090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932702+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932702+0000","last_peered":"2024-03-16T19:32:39.932702+0000","last_clean":"2024-03-16T19:32:39.932702+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932702+0000","last_undegraded":"2024-03-16T19:32:39.932702+0000","last_fullsized":"2024-03-16T19:32:39.932702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":68,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:33:32.987301+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:33:32.987301+0000","last_peered":"2024-03-16T19:33:32.987301+0000","last_clean":"2024-03-16T19:33:32.987301+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:33:32.987301+0000","last_undegraded":"2024-03-16T19:33:32.987301+0000","last_fullsized":"2024-03-16T19:33:32.987301+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 78s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176411+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176411+0000","last_peered":"2024-03-16T19:32:38.176411+0000","last_clean":"2024-03-16T19:32:38.176411+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176411+0000","last_undegraded":"2024-03-16T19:32:38.176411+0000","last_fullsized":"2024-03-16T19:32:38.176411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":214,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982041+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982041+0000","last_peered":"2024-03-16T19:32:31.982041+0000","last_clean":"2024-03-16T19:32:31.982041+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982041+0000","last_undegraded":"2024-03-16T19:32:31.982041+0000","last_fullsized":"2024-03-16T19:32:31.982041+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149057+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149057+0000","last_peered":"2024-03-16T19:32:39.149057+0000","last_clean":"2024-03-16T19:32:39.149057+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149057+0000","last_undegraded":"2024-03-16T19:32:39.149057+0000","last_fullsized":"2024-03-16T19:32:39.149057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":65,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108534+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108534+0000","last_peered":"2024-03-16T19:32:32.108534+0000","last_clean":"2024-03-16T19:32:32.108534+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108534+0000","last_undegraded":"2024-03-16T19:32:32.108534+0000","last_fullsized":"2024-03-16T19:32:32.108534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059643+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059643+0000","last_peered":"2024-03-16T19:32:31.059643+0000","last_clean":"2024-03-16T19:32:31.059643+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059643+0000","last_undegraded":"2024-03-16T19:32:31.059643+0000","last_fullsized":"2024-03-16T19:32:31.059643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568517+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568517+0000","last_peered":"2024-03-16T19:32:27.568517+0000","last_clean":"2024-03-16T19:32:27.568517+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568517+0000","last_undegraded":"2024-03-16T19:32:27.568517+0000","last_fullsized":"2024-03-16T19:32:27.568517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651622+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651622+0000","last_peered":"2024-03-16T19:32:37.651622+0000","last_clean":"2024-03-16T19:32:37.651622+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651622+0000","last_undegraded":"2024-03-16T19:32:37.651622+0000","last_fullsized":"2024-03-16T19:32:37.651622+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205810+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205810+0000","last_peered":"2024-03-16T19:32:28.205810+0000","last_clean":"2024-03-16T19:32:28.205810+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205810+0000","last_undegraded":"2024-03-16T19:32:28.205810+0000","last_fullsized":"2024-03-16T19:32:28.205810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":165,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166583+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166583+0000","last_peered":"2024-03-16T19:32:40.166583+0000","last_clean":"2024-03-16T19:32:40.166583+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166583+0000","last_undegraded":"2024-03-16T19:32:40.166583+0000","last_fullsized":"2024-03-16T19:32:40.166583+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":375,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:47.115779+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:32:47.115779+0000","last_peered":"2024-03-16T19:32:47.115779+0000","last_clean":"2024-03-16T19:32:47.115779+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:47.115779+0000","last_undegraded":"2024-03-16T19:32:47.115779+0000","last_fullsized":"2024-03-16T19:32:47.115779+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.532575+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:32:42.532575+0000","last_peered":"2024-03-16T19:32:42.532575+0000","last_clean":"2024-03-16T19:32:42.532575+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:42.532575+0000","last_undegraded":"2024-03-16T19:32:42.532575+0000","last_fullsized":"2024-03-16T19:32:42.532575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.219294+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:32:29.219294+0000","last_peered":"2024-03-16T19:32:29.219294+0000","last_clean":"2024-03-16T19:32:29.219294+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:29.219294+0000","last_undegraded":"2024-03-16T19:32:29.219294+0000","last_fullsized":"2024-03-16T19:32:29.219294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.162302+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:32:34.162302+0000","last_peered":"2024-03-16T19:32:34.162302+0000","last_clean":"2024-03-16T19:32:34.162302+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:34.162302+0000","last_undegraded":"2024-03-16T19:32:34.162302+0000","last_fullsized":"2024-03-16T19:32:34.162302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240482+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240482+0000","last_peered":"2024-03-16T19:32:34.240482+0000","last_clean":"2024-03-16T19:32:34.240482+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240482+0000","last_undegraded":"2024-03-16T19:32:34.240482+0000","last_fullsized":"2024-03-16T19:32:34.240482+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.144993+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.144993+0000","last_peered":"2024-03-16T19:32:33.144993+0000","last_clean":"2024-03-16T19:32:33.144993+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.144993+0000","last_undegraded":"2024-03-16T19:32:33.144993+0000","last_fullsized":"2024-03-16T19:32:33.144993+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":194,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:33:23.240244+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:33:23.240244+0000","last_peered":"2024-03-16T19:33:23.240244+0000","last_clean":"2024-03-16T19:33:23.240244+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:33:23.240244+0000","last_undegraded":"2024-03-16T19:33:23.240244+0000","last_fullsized":"2024-03-16T19:33:23.240244+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 62s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.692412+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:32:30.692412+0000","last_peered":"2024-03-16T19:32:30.692412+0000","last_clean":"2024-03-16T19:32:30.692412+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:30.692412+0000","last_undegraded":"2024-03-16T19:32:30.692412+0000","last_fullsized":"2024-03-16T19:32:30.692412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.998707+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:32:30.998707+0000","last_peered":"2024-03-16T19:32:30.998707+0000","last_clean":"2024-03-16T19:32:30.998707+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:30.998707+0000","last_undegraded":"2024-03-16T19:32:30.998707+0000","last_fullsized":"2024-03-16T19:32:30.998707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.721534+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:32:37.721534+0000","last_peered":"2024-03-16T19:32:37.721534+0000","last_clean":"2024-03-16T19:32:37.721534+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:37.721534+0000","last_undegraded":"2024-03-16T19:32:37.721534+0000","last_fullsized":"2024-03-16T19:32:37.721534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.589670+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:32:39.589670+0000","last_peered":"2024-03-16T19:32:39.589670+0000","last_clean":"2024-03-16T19:32:39.589670+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:39.589670+0000","last_undegraded":"2024-03-16T19:32:39.589670+0000","last_fullsized":"2024-03-16T19:32:39.589670+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:46.142320+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:32:46.142320+0000","last_peered":"2024-03-16T19:32:46.142320+0000","last_clean":"2024-03-16T19:32:46.142320+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:46.142320+0000","last_undegraded":"2024-03-16T19:32:46.142320+0000","last_fullsized":"2024-03-16T19:32:46.142320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":194,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:33:23.240305+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:33:23.240305+0000","last_peered":"2024-03-16T19:33:23.240305+0000","last_clean":"2024-03-16T19:33:23.240305+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:33:23.240305+0000","last_undegraded":"2024-03-16T19:33:23.240305+0000","last_fullsized":"2024-03-16T19:33:23.240305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 63s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586903+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586903+0000","last_peered":"2024-03-16T19:32:29.586903+0000","last_clean":"2024-03-16T19:32:29.586903+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586903+0000","last_undegraded":"2024-03-16T19:32:29.586903+0000","last_fullsized":"2024-03-16T19:32:29.586903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.125506+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:32:33.125506+0000","last_peered":"2024-03-16T19:32:33.125506+0000","last_clean":"2024-03-16T19:32:33.125506+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:33.125506+0000","last_undegraded":"2024-03-16T19:32:33.125506+0000","last_fullsized":"2024-03-16T19:32:33.125506+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910678+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910678+0000","last_peered":"2024-03-16T19:32:31.910678+0000","last_clean":"2024-03-16T19:32:31.910678+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910678+0000","last_undegraded":"2024-03-16T19:32:31.910678+0000","last_fullsized":"2024-03-16T19:32:31.910678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188001+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188001+0000","last_peered":"2024-03-16T19:32:32.188001+0000","last_clean":"2024-03-16T19:32:32.188001+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188001+0000","last_undegraded":"2024-03-16T19:32:32.188001+0000","last_fullsized":"2024-03-16T19:32:32.188001+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190121+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190121+0000","last_peered":"2024-03-16T19:32:44.190121+0000","last_clean":"2024-03-16T19:32:44.190121+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190121+0000","last_undegraded":"2024-03-16T19:32:44.190121+0000","last_fullsized":"2024-03-16T19:32:44.190121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026167+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026167+0000","last_peered":"2024-03-16T19:32:30.026167+0000","last_clean":"2024-03-16T19:32:30.026167+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026167+0000","last_undegraded":"2024-03-16T19:32:30.026167+0000","last_fullsized":"2024-03-16T19:32:30.026167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604791+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604791+0000","last_peered":"2024-03-16T19:32:28.604791+0000","last_clean":"2024-03-16T19:32:28.604791+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604791+0000","last_undegraded":"2024-03-16T19:32:28.604791+0000","last_fullsized":"2024-03-16T19:32:28.604791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:33:22.986477+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:33:22.986477+0000","last_peered":"2024-03-16T19:33:22.986477+0000","last_clean":"2024-03-16T19:33:22.986477+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:33:22.986477+0000","last_undegraded":"2024-03-16T19:33:22.986477+0000","last_fullsized":"2024-03-16T19:33:22.986477+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 63s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699900+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699900+0000","last_peered":"2024-03-16T19:32:33.699900+0000","last_clean":"2024-03-16T19:32:33.699900+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699900+0000","last_undegraded":"2024-03-16T19:32:33.699900+0000","last_fullsized":"2024-03-16T19:32:33.699900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030458+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030458+0000","last_peered":"2024-03-16T19:32:39.030458+0000","last_clean":"2024-03-16T19:32:39.030458+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030458+0000","last_undegraded":"2024-03-16T19:32:39.030458+0000","last_fullsized":"2024-03-16T19:32:39.030458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152403+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152403+0000","last_peered":"2024-03-16T19:32:33.152403+0000","last_clean":"2024-03-16T19:32:33.152403+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152403+0000","last_undegraded":"2024-03-16T19:32:33.152403+0000","last_fullsized":"2024-03-16T19:32:33.152403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889937+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889937+0000","last_peered":"2024-03-16T19:32:33.889937+0000","last_clean":"2024-03-16T19:32:33.889937+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889937+0000","last_undegraded":"2024-03-16T19:32:33.889937+0000","last_fullsized":"2024-03-16T19:32:33.889937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695467+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695467+0000","last_peered":"2024-03-16T19:32:31.695467+0000","last_clean":"2024-03-16T19:32:31.695467+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695467+0000","last_undegraded":"2024-03-16T19:32:31.695467+0000","last_fullsized":"2024-03-16T19:32:31.695467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102602+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102602+0000","last_peered":"2024-03-16T19:32:32.102602+0000","last_clean":"2024-03-16T19:32:32.102602+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102602+0000","last_undegraded":"2024-03-16T19:32:32.102602+0000","last_fullsized":"2024-03-16T19:32:32.102602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696329+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696329+0000","last_peered":"2024-03-16T19:32:32.696329+0000","last_clean":"2024-03-16T19:32:32.696329+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696329+0000","last_undegraded":"2024-03-16T19:32:32.696329+0000","last_fullsized":"2024-03-16T19:32:32.696329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177533+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177533+0000","last_peered":"2024-03-16T19:32:35.177533+0000","last_clean":"2024-03-16T19:32:35.177533+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177533+0000","last_undegraded":"2024-03-16T19:32:35.177533+0000","last_fullsized":"2024-03-16T19:32:35.177533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230753+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230753+0000","last_peered":"2024-03-16T19:32:33.230753+0000","last_clean":"2024-03-16T19:32:33.230753+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230753+0000","last_undegraded":"2024-03-16T19:32:33.230753+0000","last_fullsized":"2024-03-16T19:32:33.230753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.046537+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:32:40.046537+0000","last_peered":"2024-03-16T19:32:40.046537+0000","last_clean":"2024-03-16T19:32:40.046537+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:40.046537+0000","last_undegraded":"2024-03-16T19:32:40.046537+0000","last_fullsized":"2024-03-16T19:32:40.046537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":175,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.063436+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:32:37.063436+0000","last_peered":"2024-03-16T19:32:37.063436+0000","last_clean":"2024-03-16T19:32:37.063436+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:37.063436+0000","last_undegraded":"2024-03-16T19:32:37.063436+0000","last_fullsized":"2024-03-16T19:32:37.063436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272623+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272623+0000","last_peered":"2024-03-16T19:32:29.272623+0000","last_clean":"2024-03-16T19:32:29.272623+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272623+0000","last_undegraded":"2024-03-16T19:32:29.272623+0000","last_fullsized":"2024-03-16T19:32:29.272623+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122074,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:33:51.935 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:51.935 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:51.936 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:51.936 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:33:51.936 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:33:57.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:33:57.027 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:34:11.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:34:12.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.061+0000 7f6b5636c640 1 Processor -- start 2024-03-16T19:34:12.041 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.061+0000 7f6b5636c640 1 -- start start 2024-03-16T19:34:12.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.062+0000 7f6b5636c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b5006bb10 0x7f6b500cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:12.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.062+0000 7f6b5636c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b500cc720 con 0x7f6b500cc260 2024-03-16T19:34:12.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.062+0000 7f6b5636c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b500cc9c0 con 0x7f6b5006bef0 2024-03-16T19:34:12.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.062+0000 7f6b5636c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5015de30 con 0x7f6b5006bb10 2024-03-16T19:34:12.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.062+0000 7f6b4f7fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f6b5006bef0 0x7f6b50154660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:41124/0 (socket says 172.21.15.172:41124) 2024-03-16T19:34:12.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.062+0000 7f6b54909640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f6b500cc260 0x7f6b501651a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:34832/0 (socket says 172.21.15.172:34832) 2024-03-16T19:34:12.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.062+0000 7f6b4f7fe640 1 -- 172.21.15.172:0/3168975923 learned_addr learned my addr 172.21.15.172:0/3168975923 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:34:12.043 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.062+0000 7f6b4ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b5006bb10 0x7f6b500cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.063+0000 7f6b4effd640 1 -- 172.21.15.172:0/3168975923 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878575235 0 0) 0x7f6b500cc9c0 con 0x7f6b5006bef0 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.063+0000 7f6b4effd640 1 -- 172.21.15.172:0/3168975923 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b38003520 con 0x7f6b5006bef0 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.063+0000 7f6b4effd640 1 -- 172.21.15.172:0/3168975923 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727087104 0 0) 0x7f6b500cc720 con 0x7f6b500cc260 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.063+0000 7f6b4effd640 1 -- 172.21.15.172:0/3168975923 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b500cc9c0 con 0x7f6b500cc260 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.063+0000 7f6b4ffff640 1 -- 172.21.15.172:0/3168975923 >> v1:172.21.15.174:6789/0 conn(0x7f6b5006bef0 legacy=0x7f6b50154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.063+0000 7f6b4ffff640 1 -- 172.21.15.172:0/3168975923 >> v1:172.21.15.172:6789/0 conn(0x7f6b500cc260 legacy=0x7f6b501651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.063+0000 7f6b4ffff640 1 -- 172.21.15.172:0/3168975923 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5015e110 con 0x7f6b5006bb10 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b4ffff640 1 --2- 172.21.15.172:0/3168975923 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b5006bb10 0x7f6b500cae50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6b400097d0 tx=0x7f6b4002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b56b25c5454fc2df server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b4effd640 1 -- 172.21.15.172:0/3168975923 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40030880 con 0x7f6b5006bb10 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b4effd640 1 -- 172.21.15.172:0/3168975923 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b400309e0 con 0x7f6b5006bb10 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b4effd640 1 -- 172.21.15.172:0/3168975923 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40030d70 con 0x7f6b5006bb10 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b5636c640 1 -- 172.21.15.172:0/3168975923 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b5006bb10 msgr2=0x7f6b500cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b5636c640 1 --2- 172.21.15.172:0/3168975923 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b5006bb10 0x7f6b500cae50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6b400097d0 tx=0x7f6b4002b9c0 comp rx=0 tx=0).stop 2024-03-16T19:34:12.045 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b5636c640 1 -- 172.21.15.172:0/3168975923 shutdown_connections 2024-03-16T19:34:12.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b5636c640 1 --2- 172.21.15.172:0/3168975923 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b5006bb10 0x7f6b500cae50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:12.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b5636c640 1 -- 172.21.15.172:0/3168975923 >> 172.21.15.172:0/3168975923 conn(0x7f6b50067250 msgr2=0x7f6b500d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:12.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b5636c640 1 -- 172.21.15.172:0/3168975923 shutdown_connections 2024-03-16T19:34:12.046 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.064+0000 7f6b5636c640 1 -- 172.21.15.172:0/3168975923 wait complete. 2024-03-16T19:34:12.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.066+0000 7f6b5636c640 1 Processor -- start 2024-03-16T19:34:12.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.066+0000 7f6b5636c640 1 -- start start 2024-03-16T19:34:12.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.066+0000 7f6b5636c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5006bb10 0x7f6b500b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:12.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.066+0000 7f6b5636c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5006bef0 0x7f6b501486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:12.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.066+0000 7f6b5636c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b500cc260 0x7f6b50148c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:12.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.066+0000 7f6b5636c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b500cfff0 con 0x7f6b5006bef0 2024-03-16T19:34:12.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.066+0000 7f6b5636c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b500cfac0 con 0x7f6b5006bb10 2024-03-16T19:34:12.048 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.066+0000 7f6b5636c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5014e440 con 0x7f6b500cc260 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5006bef0 0x7f6b501486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4f7fe640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5006bef0 0x7f6b501486c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:34262/0 (socket says 172.21.15.172:34262) 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4f7fe640 1 -- 172.21.15.172:0/4279157476 learned_addr learned my addr 172.21.15.172:0/4279157476 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4ffff640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5006bb10 0x7f6b500b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b54909640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b500cc260 0x7f6b50148c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4f7fe640 1 -- 172.21.15.172:0/4279157476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b500cc260 msgr2=0x7f6b50148c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4f7fe640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b500cc260 0x7f6b50148c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4f7fe640 1 -- 172.21.15.172:0/4279157476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5006bb10 msgr2=0x7f6b500b56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4f7fe640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5006bb10 0x7f6b500b56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4f7fe640 1 -- 172.21.15.172:0/4279157476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b40009400 con 0x7f6b5006bef0 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4f7fe640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5006bef0 0x7f6b501486c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400c450 tx=0x7f6b4400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4cff9640 1 -- 172.21.15.172:0/4279157476 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44005da0 con 0x7f6b5006bef0 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4cff9640 1 -- 172.21.15.172:0/4279157476 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b440070d0 con 0x7f6b5006bef0 2024-03-16T19:34:12.049 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b4cff9640 1 -- 172.21.15.172:0/4279157476 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44007460 con 0x7f6b5006bef0 2024-03-16T19:34:12.050 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b501e3d50 con 0x7f6b5006bef0 2024-03-16T19:34:12.050 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.067+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5015dab0 con 0x7f6b5006bef0 2024-03-16T19:34:12.050 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.068+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5014e210 con 0x7f6b5006bef0 2024-03-16T19:34:12.050 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.069+0000 7f6b4cff9640 1 -- 172.21.15.172:0/4279157476 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6b440075c0 con 0x7f6b5006bef0 2024-03-16T19:34:12.051 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.069+0000 7f6b4cff9640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6b3c06b3b0 0x7f6b3c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:12.051 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.069+0000 7f6b4cff9640 1 -- 172.21.15.172:0/4279157476 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f6b44089230 con 0x7f6b5006bef0 2024-03-16T19:34:12.051 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.069+0000 7f6b4ffff640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6b3c06b3b0 0x7f6b3c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:12.051 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.070+0000 7f6b4ffff640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6b3c06b3b0 0x7f6b3c06d870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6b40036000 tx=0x7f6b40008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:12.053 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.074+0000 7f6b4cff9640 1 -- 172.21.15.172:0/4279157476 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b5014e210 con 0x7f6b5006bef0 2024-03-16T19:34:12.307 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.327+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b5015e6d0 con 0x7f6b3c06b3b0 2024-03-16T19:34:12.313 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.334+0000 7f6b4cff9640 1 -- 172.21.15.172:0/4279157476 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252919 (secure 0 0 0) 0x7f6b40030b50 con 0x7f6b3c06b3b0 2024-03-16T19:34:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:34:12.317 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6b3c06b3b0 msgr2=0x7f6b3c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6b3c06b3b0 0x7f6b3c06d870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6b40036000 tx=0x7f6b40008030 comp rx=0 tx=0).stop 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5006bef0 msgr2=0x7f6b501486c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5006bef0 0x7f6b501486c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400c450 tx=0x7f6b4400c850 comp rx=0 tx=0).stop 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 shutdown_connections 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f6b3c06b3b0 0x7f6b3c06d870 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b500cc260 0x7f6b50148c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b5006bef0 0x7f6b501486c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 --2- 172.21.15.172:0/4279157476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5006bb10 0x7f6b500b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:12.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 >> 172.21.15.172:0/4279157476 conn(0x7f6b50067250 msgr2=0x7f6b500c5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:12.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 shutdown_connections 2024-03-16T19:34:12.320 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:12.337+0000 7f6b5636c640 1 -- 172.21.15.172:0/4279157476 wait complete. 2024-03-16T19:34:12.330 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-16T19:34:12.255411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002348"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.221338+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:32:36.221338+0000","last_peered":"2024-03-16T19:32:36.221338+0000","last_clean":"2024-03-16T19:32:36.221338+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:36.221338+0000","last_undegraded":"2024-03-16T19:32:36.221338+0000","last_fullsized":"2024-03-16T19:32:36.221338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.681944+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:32:36.681944+0000","last_peered":"2024-03-16T19:32:36.681944+0000","last_clean":"2024-03-16T19:32:36.681944+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:36.681944+0000","last_undegraded":"2024-03-16T19:32:36.681944+0000","last_fullsized":"2024-03-16T19:32:36.681944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":113,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.236924+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:32:28.236924+0000","last_peered":"2024-03-16T19:32:28.236924+0000","last_clean":"2024-03-16T19:32:28.236924+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:28.236924+0000","last_undegraded":"2024-03-16T19:32:28.236924+0000","last_fullsized":"2024-03-16T19:32:28.236924+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.687123+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:32:35.687123+0000","last_peered":"2024-03-16T19:32:35.687123+0000","last_clean":"2024-03-16T19:32:35.687123+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:35.687123+0000","last_undegraded":"2024-03-16T19:32:35.687123+0000","last_fullsized":"2024-03-16T19:32:35.687123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193158+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193158+0000","last_peered":"2024-03-16T19:32:42.193158+0000","last_clean":"2024-03-16T19:32:42.193158+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193158+0000","last_undegraded":"2024-03-16T19:32:42.193158+0000","last_fullsized":"2024-03-16T19:32:42.193158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501727+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501727+0000","last_peered":"2024-03-16T19:32:43.501727+0000","last_clean":"2024-03-16T19:32:43.501727+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501727+0000","last_undegraded":"2024-03-16T19:32:43.501727+0000","last_fullsized":"2024-03-16T19:32:43.501727+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180894+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180894+0000","last_peered":"2024-03-16T19:32:45.180894+0000","last_clean":"2024-03-16T19:32:45.180894+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180894+0000","last_undegraded":"2024-03-16T19:32:45.180894+0000","last_fullsized":"2024-03-16T19:32:45.180894+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142618+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142618+0000","last_peered":"2024-03-16T19:32:31.142618+0000","last_clean":"2024-03-16T19:32:31.142618+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142618+0000","last_undegraded":"2024-03-16T19:32:31.142618+0000","last_fullsized":"2024-03-16T19:32:31.142618+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050015+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050015+0000","last_peered":"2024-03-16T19:32:30.050015+0000","last_clean":"2024-03-16T19:32:30.050015+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050015+0000","last_undegraded":"2024-03-16T19:32:30.050015+0000","last_fullsized":"2024-03-16T19:32:30.050015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906468+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906468+0000","last_peered":"2024-03-16T19:32:36.906468+0000","last_clean":"2024-03-16T19:32:36.906468+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906468+0000","last_undegraded":"2024-03-16T19:32:36.906468+0000","last_fullsized":"2024-03-16T19:32:36.906468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946447+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946447+0000","last_peered":"2024-03-16T19:32:37.946447+0000","last_clean":"2024-03-16T19:32:37.946447+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946447+0000","last_undegraded":"2024-03-16T19:32:37.946447+0000","last_fullsized":"2024-03-16T19:32:37.946447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012343+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012343+0000","last_peered":"2024-03-16T19:32:29.012343+0000","last_clean":"2024-03-16T19:32:29.012343+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012343+0000","last_undegraded":"2024-03-16T19:32:29.012343+0000","last_fullsized":"2024-03-16T19:32:29.012343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085970+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085970+0000","last_peered":"2024-03-16T19:32:34.085970+0000","last_clean":"2024-03-16T19:32:34.085970+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085970+0000","last_undegraded":"2024-03-16T19:32:34.085970+0000","last_fullsized":"2024-03-16T19:32:34.085970+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194777+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194777+0000","last_peered":"2024-03-16T19:32:31.194777+0000","last_clean":"2024-03-16T19:32:31.194777+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194777+0000","last_undegraded":"2024-03-16T19:32:31.194777+0000","last_fullsized":"2024-03-16T19:32:31.194777+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622169+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622169+0000","last_peered":"2024-03-16T19:32:30.622169+0000","last_clean":"2024-03-16T19:32:30.622169+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622169+0000","last_undegraded":"2024-03-16T19:32:30.622169+0000","last_fullsized":"2024-03-16T19:32:30.622169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.528006+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.528006+0000","last_peered":"2024-03-16T19:32:41.528006+0000","last_clean":"2024-03-16T19:32:41.528006+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.528006+0000","last_undegraded":"2024-03-16T19:32:41.528006+0000","last_fullsized":"2024-03-16T19:32:41.528006+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668910+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668910+0000","last_peered":"2024-03-16T19:32:33.668910+0000","last_clean":"2024-03-16T19:32:33.668910+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668910+0000","last_undegraded":"2024-03-16T19:32:33.668910+0000","last_fullsized":"2024-03-16T19:32:33.668910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158111+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158111+0000","last_peered":"2024-03-16T19:32:30.158111+0000","last_clean":"2024-03-16T19:32:30.158111+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158111+0000","last_undegraded":"2024-03-16T19:32:30.158111+0000","last_fullsized":"2024-03-16T19:32:30.158111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198985+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198985+0000","last_peered":"2024-03-16T19:32:43.198985+0000","last_clean":"2024-03-16T19:32:43.198985+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198985+0000","last_undegraded":"2024-03-16T19:32:43.198985+0000","last_fullsized":"2024-03-16T19:32:43.198985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874927+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874927+0000","last_peered":"2024-03-16T19:32:34.874927+0000","last_clean":"2024-03-16T19:32:34.874927+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874927+0000","last_undegraded":"2024-03-16T19:32:34.874927+0000","last_fullsized":"2024-03-16T19:32:34.874927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202050+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202050+0000","last_peered":"2024-03-16T19:32:26.202050+0000","last_clean":"2024-03-16T19:32:26.202050+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202050+0000","last_undegraded":"2024-03-16T19:32:26.202050+0000","last_fullsized":"2024-03-16T19:32:26.202050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113826+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113826+0000","last_peered":"2024-03-16T19:32:31.113826+0000","last_clean":"2024-03-16T19:32:31.113826+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113826+0000","last_undegraded":"2024-03-16T19:32:31.113826+0000","last_fullsized":"2024-03-16T19:32:31.113826+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189280+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189280+0000","last_peered":"2024-03-16T19:32:30.189280+0000","last_clean":"2024-03-16T19:32:30.189280+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189280+0000","last_undegraded":"2024-03-16T19:32:30.189280+0000","last_fullsized":"2024-03-16T19:32:30.189280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633708+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633708+0000","last_peered":"2024-03-16T19:32:38.633708+0000","last_clean":"2024-03-16T19:32:38.633708+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633708+0000","last_undegraded":"2024-03-16T19:32:38.633708+0000","last_fullsized":"2024-03-16T19:32:38.633708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":172,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030046+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030046+0000","last_peered":"2024-03-16T19:32:36.030046+0000","last_clean":"2024-03-16T19:32:36.030046+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030046+0000","last_undegraded":"2024-03-16T19:32:36.030046+0000","last_fullsized":"2024-03-16T19:32:36.030046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881976+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881976+0000","last_peered":"2024-03-16T19:32:35.881976+0000","last_clean":"2024-03-16T19:32:35.881976+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881976+0000","last_undegraded":"2024-03-16T19:32:35.881976+0000","last_fullsized":"2024-03-16T19:32:35.881976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651765+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651765+0000","last_peered":"2024-03-16T19:32:31.651765+0000","last_clean":"2024-03-16T19:32:31.651765+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651765+0000","last_undegraded":"2024-03-16T19:32:31.651765+0000","last_fullsized":"2024-03-16T19:32:31.651765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.051224+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:32:35.051224+0000","last_peered":"2024-03-16T19:32:35.051224+0000","last_clean":"2024-03-16T19:32:35.051224+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:35.051224+0000","last_undegraded":"2024-03-16T19:32:35.051224+0000","last_fullsized":"2024-03-16T19:32:35.051224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209568+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209568+0000","last_peered":"2024-03-16T19:32:37.209568+0000","last_clean":"2024-03-16T19:32:37.209568+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209568+0000","last_undegraded":"2024-03-16T19:32:37.209568+0000","last_fullsized":"2024-03-16T19:32:37.209568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.540714+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:32:40.540714+0000","last_peered":"2024-03-16T19:32:40.540714+0000","last_clean":"2024-03-16T19:32:40.540714+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:40.540714+0000","last_undegraded":"2024-03-16T19:32:40.540714+0000","last_fullsized":"2024-03-16T19:32:40.540714+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216691+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216691+0000","last_peered":"2024-03-16T19:32:27.216691+0000","last_clean":"2024-03-16T19:32:27.216691+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216691+0000","last_undegraded":"2024-03-16T19:32:27.216691+0000","last_fullsized":"2024-03-16T19:32:27.216691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.897672+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:32:40.897672+0000","last_peered":"2024-03-16T19:32:40.897672+0000","last_clean":"2024-03-16T19:32:40.897672+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:40.897672+0000","last_undegraded":"2024-03-16T19:32:40.897672+0000","last_fullsized":"2024-03-16T19:32:40.897672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002739+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002739+0000","last_peered":"2024-03-16T19:32:28.002739+0000","last_clean":"2024-03-16T19:32:28.002739+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002739+0000","last_undegraded":"2024-03-16T19:32:28.002739+0000","last_fullsized":"2024-03-16T19:32:28.002739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207579+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207579+0000","last_peered":"2024-03-16T19:32:28.207579+0000","last_clean":"2024-03-16T19:32:28.207579+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207579+0000","last_undegraded":"2024-03-16T19:32:28.207579+0000","last_fullsized":"2024-03-16T19:32:28.207579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265363+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265363+0000","last_peered":"2024-03-16T19:32:35.265363+0000","last_clean":"2024-03-16T19:32:35.265363+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265363+0000","last_undegraded":"2024-03-16T19:32:35.265363+0000","last_fullsized":"2024-03-16T19:32:35.265363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913639+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913639+0000","last_peered":"2024-03-16T19:32:32.913639+0000","last_clean":"2024-03-16T19:32:32.913639+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913639+0000","last_undegraded":"2024-03-16T19:32:32.913639+0000","last_fullsized":"2024-03-16T19:32:32.913639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105281+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105281+0000","last_peered":"2024-03-16T19:32:32.105281+0000","last_clean":"2024-03-16T19:32:32.105281+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105281+0000","last_undegraded":"2024-03-16T19:32:32.105281+0000","last_fullsized":"2024-03-16T19:32:32.105281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062784+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062784+0000","last_peered":"2024-03-16T19:32:38.062784+0000","last_clean":"2024-03-16T19:32:38.062784+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062784+0000","last_undegraded":"2024-03-16T19:32:38.062784+0000","last_fullsized":"2024-03-16T19:32:38.062784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.685985+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.685985+0000","last_peered":"2024-03-16T19:32:28.685985+0000","last_clean":"2024-03-16T19:32:28.685985+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.685985+0000","last_undegraded":"2024-03-16T19:32:28.685985+0000","last_fullsized":"2024-03-16T19:32:28.685985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691219+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691219+0000","last_peered":"2024-03-16T19:32:32.691219+0000","last_clean":"2024-03-16T19:32:32.691219+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691219+0000","last_undegraded":"2024-03-16T19:32:32.691219+0000","last_fullsized":"2024-03-16T19:32:32.691219+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944568+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944568+0000","last_peered":"2024-03-16T19:32:30.944568+0000","last_clean":"2024-03-16T19:32:30.944568+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944568+0000","last_undegraded":"2024-03-16T19:32:30.944568+0000","last_fullsized":"2024-03-16T19:32:30.944568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706768+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706768+0000","last_peered":"2024-03-16T19:32:34.706768+0000","last_clean":"2024-03-16T19:32:34.706768+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706768+0000","last_undegraded":"2024-03-16T19:32:34.706768+0000","last_fullsized":"2024-03-16T19:32:34.706768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646626+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646626+0000","last_peered":"2024-03-16T19:32:29.646626+0000","last_clean":"2024-03-16T19:32:29.646626+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646626+0000","last_undegraded":"2024-03-16T19:32:29.646626+0000","last_fullsized":"2024-03-16T19:32:29.646626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223824+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223824+0000","last_peered":"2024-03-16T19:32:30.223824+0000","last_clean":"2024-03-16T19:32:30.223824+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223824+0000","last_undegraded":"2024-03-16T19:32:30.223824+0000","last_fullsized":"2024-03-16T19:32:30.223824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538373+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538373+0000","last_peered":"2024-03-16T19:32:26.538373+0000","last_clean":"2024-03-16T19:32:26.538373+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538373+0000","last_undegraded":"2024-03-16T19:32:26.538373+0000","last_fullsized":"2024-03-16T19:32:26.538373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":30,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076122+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076122+0000","last_peered":"2024-03-16T19:32:35.076122+0000","last_clean":"2024-03-16T19:32:35.076122+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076122+0000","last_undegraded":"2024-03-16T19:32:35.076122+0000","last_fullsized":"2024-03-16T19:32:35.076122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":247,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954039+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954039+0000","last_peered":"2024-03-16T19:32:38.954039+0000","last_clean":"2024-03-16T19:32:38.954039+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954039+0000","last_undegraded":"2024-03-16T19:32:38.954039+0000","last_fullsized":"2024-03-16T19:32:38.954039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179908+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179908+0000","last_peered":"2024-03-16T19:32:41.179908+0000","last_clean":"2024-03-16T19:32:41.179908+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179908+0000","last_undegraded":"2024-03-16T19:32:41.179908+0000","last_fullsized":"2024-03-16T19:32:41.179908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245654+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245654+0000","last_peered":"2024-03-16T19:32:36.245654+0000","last_clean":"2024-03-16T19:32:36.245654+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245654+0000","last_undegraded":"2024-03-16T19:32:36.245654+0000","last_fullsized":"2024-03-16T19:32:36.245654+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197866+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197866+0000","last_peered":"2024-03-16T19:32:27.197866+0000","last_clean":"2024-03-16T19:32:27.197866+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197866+0000","last_undegraded":"2024-03-16T19:32:27.197866+0000","last_fullsized":"2024-03-16T19:32:27.197866+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696958+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696958+0000","last_peered":"2024-03-16T19:32:36.696958+0000","last_clean":"2024-03-16T19:32:36.696958+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696958+0000","last_undegraded":"2024-03-16T19:32:36.696958+0000","last_fullsized":"2024-03-16T19:32:36.696958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190754+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190754+0000","last_peered":"2024-03-16T19:32:37.190754+0000","last_clean":"2024-03-16T19:32:37.190754+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190754+0000","last_undegraded":"2024-03-16T19:32:37.190754+0000","last_fullsized":"2024-03-16T19:32:37.190754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938129+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938129+0000","last_peered":"2024-03-16T19:32:29.938129+0000","last_clean":"2024-03-16T19:32:29.938129+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938129+0000","last_undegraded":"2024-03-16T19:32:29.938129+0000","last_fullsized":"2024-03-16T19:32:29.938129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932726+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932726+0000","last_peered":"2024-03-16T19:32:39.932726+0000","last_clean":"2024-03-16T19:32:39.932726+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932726+0000","last_undegraded":"2024-03-16T19:32:39.932726+0000","last_fullsized":"2024-03-16T19:32:39.932726+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":69,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:33:47.988829+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:33:47.988829+0000","last_peered":"2024-03-16T19:33:47.988829+0000","last_clean":"2024-03-16T19:33:47.988829+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:33:47.988829+0000","last_undegraded":"2024-03-16T19:33:47.988829+0000","last_fullsized":"2024-03-16T19:33:47.988829+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 93s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176466+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176466+0000","last_peered":"2024-03-16T19:32:38.176466+0000","last_clean":"2024-03-16T19:32:38.176466+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176466+0000","last_undegraded":"2024-03-16T19:32:38.176466+0000","last_fullsized":"2024-03-16T19:32:38.176466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":215,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982088+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982088+0000","last_peered":"2024-03-16T19:32:31.982088+0000","last_clean":"2024-03-16T19:32:31.982088+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982088+0000","last_undegraded":"2024-03-16T19:32:31.982088+0000","last_fullsized":"2024-03-16T19:32:31.982088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149110+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149110+0000","last_peered":"2024-03-16T19:32:39.149110+0000","last_clean":"2024-03-16T19:32:39.149110+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149110+0000","last_undegraded":"2024-03-16T19:32:39.149110+0000","last_fullsized":"2024-03-16T19:32:39.149110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108585+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108585+0000","last_peered":"2024-03-16T19:32:32.108585+0000","last_clean":"2024-03-16T19:32:32.108585+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108585+0000","last_undegraded":"2024-03-16T19:32:32.108585+0000","last_fullsized":"2024-03-16T19:32:32.108585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059668+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059668+0000","last_peered":"2024-03-16T19:32:31.059668+0000","last_clean":"2024-03-16T19:32:31.059668+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059668+0000","last_undegraded":"2024-03-16T19:32:31.059668+0000","last_fullsized":"2024-03-16T19:32:31.059668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568574+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568574+0000","last_peered":"2024-03-16T19:32:27.568574+0000","last_clean":"2024-03-16T19:32:27.568574+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568574+0000","last_undegraded":"2024-03-16T19:32:27.568574+0000","last_fullsized":"2024-03-16T19:32:27.568574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651659+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651659+0000","last_peered":"2024-03-16T19:32:37.651659+0000","last_clean":"2024-03-16T19:32:37.651659+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651659+0000","last_undegraded":"2024-03-16T19:32:37.651659+0000","last_fullsized":"2024-03-16T19:32:37.651659+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":61,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205835+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205835+0000","last_peered":"2024-03-16T19:32:28.205835+0000","last_clean":"2024-03-16T19:32:28.205835+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205835+0000","last_undegraded":"2024-03-16T19:32:28.205835+0000","last_fullsized":"2024-03-16T19:32:28.205835+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":165,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166583+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166583+0000","last_peered":"2024-03-16T19:32:40.166583+0000","last_clean":"2024-03-16T19:32:40.166583+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166583+0000","last_undegraded":"2024-03-16T19:32:40.166583+0000","last_fullsized":"2024-03-16T19:32:40.166583+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":375,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:47.115779+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:32:47.115779+0000","last_peered":"2024-03-16T19:32:47.115779+0000","last_clean":"2024-03-16T19:32:47.115779+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:47.115779+0000","last_undegraded":"2024-03-16T19:32:47.115779+0000","last_fullsized":"2024-03-16T19:32:47.115779+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.532575+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:32:42.532575+0000","last_peered":"2024-03-16T19:32:42.532575+0000","last_clean":"2024-03-16T19:32:42.532575+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:42.532575+0000","last_undegraded":"2024-03-16T19:32:42.532575+0000","last_fullsized":"2024-03-16T19:32:42.532575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.219294+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:32:29.219294+0000","last_peered":"2024-03-16T19:32:29.219294+0000","last_clean":"2024-03-16T19:32:29.219294+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:29.219294+0000","last_undegraded":"2024-03-16T19:32:29.219294+0000","last_fullsized":"2024-03-16T19:32:29.219294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.162302+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:32:34.162302+0000","last_peered":"2024-03-16T19:32:34.162302+0000","last_clean":"2024-03-16T19:32:34.162302+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:34.162302+0000","last_undegraded":"2024-03-16T19:32:34.162302+0000","last_fullsized":"2024-03-16T19:32:34.162302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240521+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240521+0000","last_peered":"2024-03-16T19:32:34.240521+0000","last_clean":"2024-03-16T19:32:34.240521+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240521+0000","last_undegraded":"2024-03-16T19:32:34.240521+0000","last_fullsized":"2024-03-16T19:32:34.240521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.145030+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.145030+0000","last_peered":"2024-03-16T19:32:33.145030+0000","last_clean":"2024-03-16T19:32:33.145030+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.145030+0000","last_undegraded":"2024-03-16T19:32:33.145030+0000","last_fullsized":"2024-03-16T19:32:33.145030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:33:53.242966+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:33:53.242966+0000","last_peered":"2024-03-16T19:33:53.242966+0000","last_clean":"2024-03-16T19:33:53.242966+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:33:53.242966+0000","last_undegraded":"2024-03-16T19:33:53.242966+0000","last_fullsized":"2024-03-16T19:33:53.242966+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 92s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.692412+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:32:30.692412+0000","last_peered":"2024-03-16T19:32:30.692412+0000","last_clean":"2024-03-16T19:32:30.692412+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:30.692412+0000","last_undegraded":"2024-03-16T19:32:30.692412+0000","last_fullsized":"2024-03-16T19:32:30.692412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.998707+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:32:30.998707+0000","last_peered":"2024-03-16T19:32:30.998707+0000","last_clean":"2024-03-16T19:32:30.998707+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:30.998707+0000","last_undegraded":"2024-03-16T19:32:30.998707+0000","last_fullsized":"2024-03-16T19:32:30.998707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.721534+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:32:37.721534+0000","last_peered":"2024-03-16T19:32:37.721534+0000","last_clean":"2024-03-16T19:32:37.721534+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:37.721534+0000","last_undegraded":"2024-03-16T19:32:37.721534+0000","last_fullsized":"2024-03-16T19:32:37.721534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.589670+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:32:39.589670+0000","last_peered":"2024-03-16T19:32:39.589670+0000","last_clean":"2024-03-16T19:32:39.589670+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:39.589670+0000","last_undegraded":"2024-03-16T19:32:39.589670+0000","last_fullsized":"2024-03-16T19:32:39.589670+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:46.142320+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:32:46.142320+0000","last_peered":"2024-03-16T19:32:46.142320+0000","last_clean":"2024-03-16T19:32:46.142320+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:46.142320+0000","last_undegraded":"2024-03-16T19:32:46.142320+0000","last_fullsized":"2024-03-16T19:32:46.142320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":196,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:33:53.243008+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:33:53.243008+0000","last_peered":"2024-03-16T19:33:53.243008+0000","last_clean":"2024-03-16T19:33:53.243008+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:33:53.243008+0000","last_undegraded":"2024-03-16T19:33:53.243008+0000","last_fullsized":"2024-03-16T19:33:53.243008+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 93s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586903+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586903+0000","last_peered":"2024-03-16T19:32:29.586903+0000","last_clean":"2024-03-16T19:32:29.586903+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586903+0000","last_undegraded":"2024-03-16T19:32:29.586903+0000","last_fullsized":"2024-03-16T19:32:29.586903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.125506+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:32:33.125506+0000","last_peered":"2024-03-16T19:32:33.125506+0000","last_clean":"2024-03-16T19:32:33.125506+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:33.125506+0000","last_undegraded":"2024-03-16T19:32:33.125506+0000","last_fullsized":"2024-03-16T19:32:33.125506+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910678+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910678+0000","last_peered":"2024-03-16T19:32:31.910678+0000","last_clean":"2024-03-16T19:32:31.910678+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910678+0000","last_undegraded":"2024-03-16T19:32:31.910678+0000","last_fullsized":"2024-03-16T19:32:31.910678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188030+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188030+0000","last_peered":"2024-03-16T19:32:32.188030+0000","last_clean":"2024-03-16T19:32:32.188030+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188030+0000","last_undegraded":"2024-03-16T19:32:32.188030+0000","last_fullsized":"2024-03-16T19:32:32.188030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190121+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190121+0000","last_peered":"2024-03-16T19:32:44.190121+0000","last_clean":"2024-03-16T19:32:44.190121+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190121+0000","last_undegraded":"2024-03-16T19:32:44.190121+0000","last_fullsized":"2024-03-16T19:32:44.190121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026167+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026167+0000","last_peered":"2024-03-16T19:32:30.026167+0000","last_clean":"2024-03-16T19:32:30.026167+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026167+0000","last_undegraded":"2024-03-16T19:32:30.026167+0000","last_fullsized":"2024-03-16T19:32:30.026167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604791+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604791+0000","last_peered":"2024-03-16T19:32:28.604791+0000","last_clean":"2024-03-16T19:32:28.604791+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604791+0000","last_undegraded":"2024-03-16T19:32:28.604791+0000","last_fullsized":"2024-03-16T19:32:28.604791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:33:52.989261+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:33:52.989261+0000","last_peered":"2024-03-16T19:33:52.989261+0000","last_clean":"2024-03-16T19:33:52.989261+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:33:52.989261+0000","last_undegraded":"2024-03-16T19:33:52.989261+0000","last_fullsized":"2024-03-16T19:33:52.989261+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 93s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699900+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699900+0000","last_peered":"2024-03-16T19:32:33.699900+0000","last_clean":"2024-03-16T19:32:33.699900+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699900+0000","last_undegraded":"2024-03-16T19:32:33.699900+0000","last_fullsized":"2024-03-16T19:32:33.699900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030458+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030458+0000","last_peered":"2024-03-16T19:32:39.030458+0000","last_clean":"2024-03-16T19:32:39.030458+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030458+0000","last_undegraded":"2024-03-16T19:32:39.030458+0000","last_fullsized":"2024-03-16T19:32:39.030458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152425+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152425+0000","last_peered":"2024-03-16T19:32:33.152425+0000","last_clean":"2024-03-16T19:32:33.152425+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152425+0000","last_undegraded":"2024-03-16T19:32:33.152425+0000","last_fullsized":"2024-03-16T19:32:33.152425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889937+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889937+0000","last_peered":"2024-03-16T19:32:33.889937+0000","last_clean":"2024-03-16T19:32:33.889937+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889937+0000","last_undegraded":"2024-03-16T19:32:33.889937+0000","last_fullsized":"2024-03-16T19:32:33.889937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695467+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695467+0000","last_peered":"2024-03-16T19:32:31.695467+0000","last_clean":"2024-03-16T19:32:31.695467+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695467+0000","last_undegraded":"2024-03-16T19:32:31.695467+0000","last_fullsized":"2024-03-16T19:32:31.695467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102602+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102602+0000","last_peered":"2024-03-16T19:32:32.102602+0000","last_clean":"2024-03-16T19:32:32.102602+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102602+0000","last_undegraded":"2024-03-16T19:32:32.102602+0000","last_fullsized":"2024-03-16T19:32:32.102602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696329+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696329+0000","last_peered":"2024-03-16T19:32:32.696329+0000","last_clean":"2024-03-16T19:32:32.696329+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696329+0000","last_undegraded":"2024-03-16T19:32:32.696329+0000","last_fullsized":"2024-03-16T19:32:32.696329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177533+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177533+0000","last_peered":"2024-03-16T19:32:35.177533+0000","last_clean":"2024-03-16T19:32:35.177533+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177533+0000","last_undegraded":"2024-03-16T19:32:35.177533+0000","last_fullsized":"2024-03-16T19:32:35.177533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230753+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230753+0000","last_peered":"2024-03-16T19:32:33.230753+0000","last_clean":"2024-03-16T19:32:33.230753+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230753+0000","last_undegraded":"2024-03-16T19:32:33.230753+0000","last_fullsized":"2024-03-16T19:32:33.230753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.046537+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:32:40.046537+0000","last_peered":"2024-03-16T19:32:40.046537+0000","last_clean":"2024-03-16T19:32:40.046537+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:40.046537+0000","last_undegraded":"2024-03-16T19:32:40.046537+0000","last_fullsized":"2024-03-16T19:32:40.046537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":175,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.063436+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:32:37.063436+0000","last_peered":"2024-03-16T19:32:37.063436+0000","last_clean":"2024-03-16T19:32:37.063436+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:37.063436+0000","last_undegraded":"2024-03-16T19:32:37.063436+0000","last_fullsized":"2024-03-16T19:32:37.063436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272673+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272673+0000","last_peered":"2024-03-16T19:32:29.272673+0000","last_clean":"2024-03-16T19:32:29.272673+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272673+0000","last_undegraded":"2024-03-16T19:32:29.272673+0000","last_fullsized":"2024-03-16T19:32:29.272673+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122078,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:34:12.335 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:12.335 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:12.336 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:12.336 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:12.336 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:34:27.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:34:27.441 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:34:27.567 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:34:27.587+0000 7f3900bdc640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 68534) UID: 0 2024-03-16T19:34:27.567 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:34:27.587+0000 7fda8712b640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 68534) UID: 0 2024-03-16T19:34:27.568 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:34:27.587+0000 7fbcd8ae2640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 68534) UID: 0 2024-03-16T19:34:27.568 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:34:27.588+0000 7f1aabbce640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 68534) UID: 0 2024-03-16T19:34:32.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:34:32.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.478+0000 7fa5f746b640 1 Processor -- start 2024-03-16T19:34:32.457 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.478+0000 7fa5f746b640 1 -- start start 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.479+0000 7fa5f746b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa5f00c7290 0x7fa5f00c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.479+0000 7fa5f746b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa5f0072090 0x7fa5f00724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.479+0000 7fa5f746b640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f00c6d10 con 0x7fa5f014f340 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.479+0000 7fa5f746b640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f00c91a0 con 0x7fa5f0072090 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.479+0000 7fa5f746b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f00718d0 con 0x7fa5f00c7290 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.479+0000 7fa5f5a08640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa5f0072090 0x7fa5f00724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.479+0000 7fa5f5207640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fa5f014f340 0x7fa5f00c5fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:51754/0 (socket says 172.21.15.172:51754) 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.479+0000 7fa5f5207640 1 -- 172.21.15.172:0/2660821384 learned_addr learned my addr 172.21.15.172:0/2660821384 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.479+0000 7fa5f4a06640 1 --2- 172.21.15.172:0/2660821384 >> v2:172.21.15.174:3300/0 conn(0x7fa5f00c7290 0x7fa5f00c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.480+0000 7fa5f5a08640 1 -- 172.21.15.172:0/2660821384 >> v2:172.21.15.174:3300/0 conn(0x7fa5f00c7290 msgr2=0x7fa5f00c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.480+0000 7fa5f5a08640 1 --2- 172.21.15.172:0/2660821384 >> v2:172.21.15.174:3300/0 conn(0x7fa5f00c7290 0x7fa5f00c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.480+0000 7fa5f5a08640 1 -- 172.21.15.172:0/2660821384 >> v1:172.21.15.172:6789/0 conn(0x7fa5f014f340 legacy=0x7fa5f00c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:32.459 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.480+0000 7fa5e7fff640 1 -- 172.21.15.172:0/2660821384 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537656254 0 0) 0x7fa5f00c6d10 con 0x7fa5f014f340 2024-03-16T19:34:32.460 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.480+0000 7fa5f5a08640 1 -- 172.21.15.172:0/2660821384 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f0159540 con 0x7fa5f0072090 2024-03-16T19:34:32.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.480+0000 7fa5f5a08640 1 --2- 172.21.15.172:0/2660821384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 0x7fa5f00724b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0072ef0 tx=0x7fa5e802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73ace0b6e722a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:32.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5e7fff640 1 -- 172.21.15.172:0/2660821384 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e802cbf0 con 0x7fa5f0072090 2024-03-16T19:34:32.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5e7fff640 1 -- 172.21.15.172:0/2660821384 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e802cd50 con 0x7fa5f0072090 2024-03-16T19:34:32.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5e7fff640 1 -- 172.21.15.172:0/2660821384 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8034660 con 0x7fa5f0072090 2024-03-16T19:34:32.461 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5f746b640 1 -- 172.21.15.172:0/2660821384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 msgr2=0x7fa5f00724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:32.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5f746b640 1 --2- 172.21.15.172:0/2660821384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 0x7fa5f00724b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0072ef0 tx=0x7fa5e802be40 comp rx=0 tx=0).stop 2024-03-16T19:34:32.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5f746b640 1 -- 172.21.15.172:0/2660821384 shutdown_connections 2024-03-16T19:34:32.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5f746b640 1 --2- 172.21.15.172:0/2660821384 >> v2:172.21.15.174:3300/0 conn(0x7fa5f00c7290 0x7fa5f00c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:32.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5f746b640 1 --2- 172.21.15.172:0/2660821384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 0x7fa5f00724b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:32.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5f746b640 1 -- 172.21.15.172:0/2660821384 >> 172.21.15.172:0/2660821384 conn(0x7fa5f00c9390 msgr2=0x7fa5f00c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:32.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5f746b640 1 -- 172.21.15.172:0/2660821384 shutdown_connections 2024-03-16T19:34:32.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.481+0000 7fa5f746b640 1 -- 172.21.15.172:0/2660821384 wait complete. 2024-03-16T19:34:32.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.483+0000 7fa5f746b640 1 Processor -- start 2024-03-16T19:34:32.462 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.483+0000 7fa5f746b640 1 -- start start 2024-03-16T19:34:32.464 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.483+0000 7fa5f746b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 0x7fa5f0152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:32.464 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.483+0000 7fa5f746b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5f00c7290 0x7fa5f01f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:32.464 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.483+0000 7fa5f746b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5f014f340 0x7fa5f00cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:32.464 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.483+0000 7fa5f746b640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f0154fb0 con 0x7fa5f0072090 2024-03-16T19:34:32.464 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.483+0000 7fa5f746b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f01ee300 con 0x7fa5f00c7290 2024-03-16T19:34:32.464 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.483+0000 7fa5f746b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f01f15e0 con 0x7fa5f014f340 2024-03-16T19:34:32.464 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.483+0000 7fa5f5207640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 0x7fa5f0152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f5207640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 0x7fa5f0152010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:43164/0 (socket says 172.21.15.172:43164) 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f5207640 1 -- 172.21.15.172:0/3494218530 learned_addr learned my addr 172.21.15.172:0/3494218530 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f4a06640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5f00c7290 0x7fa5f01f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f5a08640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5f014f340 0x7fa5f00cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f5207640 1 -- 172.21.15.172:0/3494218530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5f014f340 msgr2=0x7fa5f00cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f5207640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5f014f340 0x7fa5f00cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f5207640 1 -- 172.21.15.172:0/3494218530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5f00c7290 msgr2=0x7fa5f01f10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f5207640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5f00c7290 0x7fa5f01f10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f5207640 1 -- 172.21.15.172:0/3494218530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8009a70 con 0x7fa5f0072090 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f5207640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 0x7fa5f0152010 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c460 tx=0x7fa5ec00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5e5ffb640 1 -- 172.21.15.172:0/3494218530 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0074e0 con 0x7fa5f0072090 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5e5ffb640 1 -- 172.21.15.172:0/3494218530 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec007660 con 0x7fa5f0072090 2024-03-16T19:34:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f0152bf0 con 0x7fa5f0072090 2024-03-16T19:34:32.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.484+0000 7fa5e5ffb640 1 -- 172.21.15.172:0/3494218530 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec004a20 con 0x7fa5f0072090 2024-03-16T19:34:32.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.485+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f0152910 con 0x7fa5f0072090 2024-03-16T19:34:32.466 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.486+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f00758f0 con 0x7fa5f0072090 2024-03-16T19:34:32.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.486+0000 7fa5e5ffb640 1 -- 172.21.15.172:0/3494218530 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa5ec004b80 con 0x7fa5f0072090 2024-03-16T19:34:32.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.487+0000 7fa5e5ffb640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa5d406b3b0 0x7fa5d406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:32.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.487+0000 7fa5e5ffb640 1 -- 172.21.15.172:0/3494218530 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fa5ec088e80 con 0x7fa5f0072090 2024-03-16T19:34:32.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.487+0000 7fa5f4a06640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa5d406b3b0 0x7fa5d406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:32.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.488+0000 7fa5f4a06640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa5d406b3b0 0x7fa5d406d870 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0150f70 tx=0x7fa5e0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:32.471 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.493+0000 7fa5e5ffb640 1 -- 172.21.15.172:0/3494218530 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5ec053a60 con 0x7fa5f0072090 2024-03-16T19:34:32.746 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.767+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5f00c8ec0 con 0x7fa5d406b3b0 2024-03-16T19:34:32.756 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.777+0000 7fa5e5ffb640 1 -- 172.21.15.172:0/3494218530 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252923 (secure 0 0 0) 0x7fa5e0011a30 con 0x7fa5d406b3b0 2024-03-16T19:34:32.759 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:34:32.760 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:34:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa5d406b3b0 msgr2=0x7fa5d406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa5d406b3b0 0x7fa5d406d870 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0150f70 tx=0x7fa5e0009190 comp rx=0 tx=0).stop 2024-03-16T19:34:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 msgr2=0x7fa5f0152010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 0x7fa5f0152010 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c460 tx=0x7fa5ec00c920 comp rx=0 tx=0).stop 2024-03-16T19:34:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 shutdown_connections 2024-03-16T19:34:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa5d406b3b0 0x7fa5d406d870 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0150f70 tx=0x7fa5e0009190 comp rx=0 tx=0).stop 2024-03-16T19:34:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5f014f340 0x7fa5f00cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5f00c7290 0x7fa5f01f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:32.762 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 --2- 172.21.15.172:0/3494218530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f0072090 0x7fa5f0152010 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:32.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 >> 172.21.15.172:0/3494218530 conn(0x7fa5f00c9390 msgr2=0x7fa5f014d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:32.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.781+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 shutdown_connections 2024-03-16T19:34:32.763 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:32.782+0000 7fa5f746b640 1 -- 172.21.15.172:0/3494218530 wait complete. 2024-03-16T19:34:32.775 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-03-16T19:34:32.259280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002269"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.221338+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:32:36.221338+0000","last_peered":"2024-03-16T19:32:36.221338+0000","last_clean":"2024-03-16T19:32:36.221338+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:36.221338+0000","last_undegraded":"2024-03-16T19:32:36.221338+0000","last_fullsized":"2024-03-16T19:32:36.221338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.681944+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:32:36.681944+0000","last_peered":"2024-03-16T19:32:36.681944+0000","last_clean":"2024-03-16T19:32:36.681944+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:36.681944+0000","last_undegraded":"2024-03-16T19:32:36.681944+0000","last_fullsized":"2024-03-16T19:32:36.681944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":113,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.236924+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:32:28.236924+0000","last_peered":"2024-03-16T19:32:28.236924+0000","last_clean":"2024-03-16T19:32:28.236924+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:28.236924+0000","last_undegraded":"2024-03-16T19:32:28.236924+0000","last_fullsized":"2024-03-16T19:32:28.236924+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.687123+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:32:35.687123+0000","last_peered":"2024-03-16T19:32:35.687123+0000","last_clean":"2024-03-16T19:32:35.687123+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:35.687123+0000","last_undegraded":"2024-03-16T19:32:35.687123+0000","last_fullsized":"2024-03-16T19:32:35.687123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193204+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193204+0000","last_peered":"2024-03-16T19:32:42.193204+0000","last_clean":"2024-03-16T19:32:42.193204+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193204+0000","last_undegraded":"2024-03-16T19:32:42.193204+0000","last_fullsized":"2024-03-16T19:32:42.193204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501752+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501752+0000","last_peered":"2024-03-16T19:32:43.501752+0000","last_clean":"2024-03-16T19:32:43.501752+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501752+0000","last_undegraded":"2024-03-16T19:32:43.501752+0000","last_fullsized":"2024-03-16T19:32:43.501752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180916+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180916+0000","last_peered":"2024-03-16T19:32:45.180916+0000","last_clean":"2024-03-16T19:32:45.180916+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180916+0000","last_undegraded":"2024-03-16T19:32:45.180916+0000","last_fullsized":"2024-03-16T19:32:45.180916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142618+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142618+0000","last_peered":"2024-03-16T19:32:31.142618+0000","last_clean":"2024-03-16T19:32:31.142618+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142618+0000","last_undegraded":"2024-03-16T19:32:31.142618+0000","last_fullsized":"2024-03-16T19:32:31.142618+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050037+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050037+0000","last_peered":"2024-03-16T19:32:30.050037+0000","last_clean":"2024-03-16T19:32:30.050037+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050037+0000","last_undegraded":"2024-03-16T19:32:30.050037+0000","last_fullsized":"2024-03-16T19:32:30.050037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906526+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906526+0000","last_peered":"2024-03-16T19:32:36.906526+0000","last_clean":"2024-03-16T19:32:36.906526+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906526+0000","last_undegraded":"2024-03-16T19:32:36.906526+0000","last_fullsized":"2024-03-16T19:32:36.906526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946474+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946474+0000","last_peered":"2024-03-16T19:32:37.946474+0000","last_clean":"2024-03-16T19:32:37.946474+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946474+0000","last_undegraded":"2024-03-16T19:32:37.946474+0000","last_fullsized":"2024-03-16T19:32:37.946474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012365+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012365+0000","last_peered":"2024-03-16T19:32:29.012365+0000","last_clean":"2024-03-16T19:32:29.012365+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012365+0000","last_undegraded":"2024-03-16T19:32:29.012365+0000","last_fullsized":"2024-03-16T19:32:29.012365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085970+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085970+0000","last_peered":"2024-03-16T19:32:34.085970+0000","last_clean":"2024-03-16T19:32:34.085970+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085970+0000","last_undegraded":"2024-03-16T19:32:34.085970+0000","last_fullsized":"2024-03-16T19:32:34.085970+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194818+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194818+0000","last_peered":"2024-03-16T19:32:31.194818+0000","last_clean":"2024-03-16T19:32:31.194818+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194818+0000","last_undegraded":"2024-03-16T19:32:31.194818+0000","last_fullsized":"2024-03-16T19:32:31.194818+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622169+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622169+0000","last_peered":"2024-03-16T19:32:30.622169+0000","last_clean":"2024-03-16T19:32:30.622169+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622169+0000","last_undegraded":"2024-03-16T19:32:30.622169+0000","last_fullsized":"2024-03-16T19:32:30.622169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.528006+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.528006+0000","last_peered":"2024-03-16T19:32:41.528006+0000","last_clean":"2024-03-16T19:32:41.528006+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.528006+0000","last_undegraded":"2024-03-16T19:32:41.528006+0000","last_fullsized":"2024-03-16T19:32:41.528006+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668910+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668910+0000","last_peered":"2024-03-16T19:32:33.668910+0000","last_clean":"2024-03-16T19:32:33.668910+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668910+0000","last_undegraded":"2024-03-16T19:32:33.668910+0000","last_fullsized":"2024-03-16T19:32:33.668910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158111+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158111+0000","last_peered":"2024-03-16T19:32:30.158111+0000","last_clean":"2024-03-16T19:32:30.158111+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158111+0000","last_undegraded":"2024-03-16T19:32:30.158111+0000","last_fullsized":"2024-03-16T19:32:30.158111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198985+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198985+0000","last_peered":"2024-03-16T19:32:43.198985+0000","last_clean":"2024-03-16T19:32:43.198985+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198985+0000","last_undegraded":"2024-03-16T19:32:43.198985+0000","last_fullsized":"2024-03-16T19:32:43.198985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874927+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874927+0000","last_peered":"2024-03-16T19:32:34.874927+0000","last_clean":"2024-03-16T19:32:34.874927+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874927+0000","last_undegraded":"2024-03-16T19:32:34.874927+0000","last_fullsized":"2024-03-16T19:32:34.874927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202050+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202050+0000","last_peered":"2024-03-16T19:32:26.202050+0000","last_clean":"2024-03-16T19:32:26.202050+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202050+0000","last_undegraded":"2024-03-16T19:32:26.202050+0000","last_fullsized":"2024-03-16T19:32:26.202050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113826+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113826+0000","last_peered":"2024-03-16T19:32:31.113826+0000","last_clean":"2024-03-16T19:32:31.113826+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113826+0000","last_undegraded":"2024-03-16T19:32:31.113826+0000","last_fullsized":"2024-03-16T19:32:31.113826+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189280+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189280+0000","last_peered":"2024-03-16T19:32:30.189280+0000","last_clean":"2024-03-16T19:32:30.189280+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189280+0000","last_undegraded":"2024-03-16T19:32:30.189280+0000","last_fullsized":"2024-03-16T19:32:30.189280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633708+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633708+0000","last_peered":"2024-03-16T19:32:38.633708+0000","last_clean":"2024-03-16T19:32:38.633708+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633708+0000","last_undegraded":"2024-03-16T19:32:38.633708+0000","last_fullsized":"2024-03-16T19:32:38.633708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":172,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030046+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030046+0000","last_peered":"2024-03-16T19:32:36.030046+0000","last_clean":"2024-03-16T19:32:36.030046+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030046+0000","last_undegraded":"2024-03-16T19:32:36.030046+0000","last_fullsized":"2024-03-16T19:32:36.030046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881976+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881976+0000","last_peered":"2024-03-16T19:32:35.881976+0000","last_clean":"2024-03-16T19:32:35.881976+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881976+0000","last_undegraded":"2024-03-16T19:32:35.881976+0000","last_fullsized":"2024-03-16T19:32:35.881976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651765+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651765+0000","last_peered":"2024-03-16T19:32:31.651765+0000","last_clean":"2024-03-16T19:32:31.651765+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651765+0000","last_undegraded":"2024-03-16T19:32:31.651765+0000","last_fullsized":"2024-03-16T19:32:31.651765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.051224+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:32:35.051224+0000","last_peered":"2024-03-16T19:32:35.051224+0000","last_clean":"2024-03-16T19:32:35.051224+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:35.051224+0000","last_undegraded":"2024-03-16T19:32:35.051224+0000","last_fullsized":"2024-03-16T19:32:35.051224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209568+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209568+0000","last_peered":"2024-03-16T19:32:37.209568+0000","last_clean":"2024-03-16T19:32:37.209568+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209568+0000","last_undegraded":"2024-03-16T19:32:37.209568+0000","last_fullsized":"2024-03-16T19:32:37.209568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.540714+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:32:40.540714+0000","last_peered":"2024-03-16T19:32:40.540714+0000","last_clean":"2024-03-16T19:32:40.540714+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:40.540714+0000","last_undegraded":"2024-03-16T19:32:40.540714+0000","last_fullsized":"2024-03-16T19:32:40.540714+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216691+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216691+0000","last_peered":"2024-03-16T19:32:27.216691+0000","last_clean":"2024-03-16T19:32:27.216691+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216691+0000","last_undegraded":"2024-03-16T19:32:27.216691+0000","last_fullsized":"2024-03-16T19:32:27.216691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.897672+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:32:40.897672+0000","last_peered":"2024-03-16T19:32:40.897672+0000","last_clean":"2024-03-16T19:32:40.897672+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:40.897672+0000","last_undegraded":"2024-03-16T19:32:40.897672+0000","last_fullsized":"2024-03-16T19:32:40.897672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002778+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002778+0000","last_peered":"2024-03-16T19:32:28.002778+0000","last_clean":"2024-03-16T19:32:28.002778+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002778+0000","last_undegraded":"2024-03-16T19:32:28.002778+0000","last_fullsized":"2024-03-16T19:32:28.002778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207579+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207579+0000","last_peered":"2024-03-16T19:32:28.207579+0000","last_clean":"2024-03-16T19:32:28.207579+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207579+0000","last_undegraded":"2024-03-16T19:32:28.207579+0000","last_fullsized":"2024-03-16T19:32:28.207579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265410+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265410+0000","last_peered":"2024-03-16T19:32:35.265410+0000","last_clean":"2024-03-16T19:32:35.265410+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265410+0000","last_undegraded":"2024-03-16T19:32:35.265410+0000","last_fullsized":"2024-03-16T19:32:35.265410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913659+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913659+0000","last_peered":"2024-03-16T19:32:32.913659+0000","last_clean":"2024-03-16T19:32:32.913659+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913659+0000","last_undegraded":"2024-03-16T19:32:32.913659+0000","last_fullsized":"2024-03-16T19:32:32.913659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105281+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105281+0000","last_peered":"2024-03-16T19:32:32.105281+0000","last_clean":"2024-03-16T19:32:32.105281+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105281+0000","last_undegraded":"2024-03-16T19:32:32.105281+0000","last_fullsized":"2024-03-16T19:32:32.105281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062784+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062784+0000","last_peered":"2024-03-16T19:32:38.062784+0000","last_clean":"2024-03-16T19:32:38.062784+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062784+0000","last_undegraded":"2024-03-16T19:32:38.062784+0000","last_fullsized":"2024-03-16T19:32:38.062784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.686005+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.686005+0000","last_peered":"2024-03-16T19:32:28.686005+0000","last_clean":"2024-03-16T19:32:28.686005+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.686005+0000","last_undegraded":"2024-03-16T19:32:28.686005+0000","last_fullsized":"2024-03-16T19:32:28.686005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691258+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691258+0000","last_peered":"2024-03-16T19:32:32.691258+0000","last_clean":"2024-03-16T19:32:32.691258+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691258+0000","last_undegraded":"2024-03-16T19:32:32.691258+0000","last_fullsized":"2024-03-16T19:32:32.691258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944590+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944590+0000","last_peered":"2024-03-16T19:32:30.944590+0000","last_clean":"2024-03-16T19:32:30.944590+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944590+0000","last_undegraded":"2024-03-16T19:32:30.944590+0000","last_fullsized":"2024-03-16T19:32:30.944590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706805+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706805+0000","last_peered":"2024-03-16T19:32:34.706805+0000","last_clean":"2024-03-16T19:32:34.706805+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706805+0000","last_undegraded":"2024-03-16T19:32:34.706805+0000","last_fullsized":"2024-03-16T19:32:34.706805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646648+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646648+0000","last_peered":"2024-03-16T19:32:29.646648+0000","last_clean":"2024-03-16T19:32:29.646648+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646648+0000","last_undegraded":"2024-03-16T19:32:29.646648+0000","last_fullsized":"2024-03-16T19:32:29.646648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223858+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223858+0000","last_peered":"2024-03-16T19:32:30.223858+0000","last_clean":"2024-03-16T19:32:30.223858+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223858+0000","last_undegraded":"2024-03-16T19:32:30.223858+0000","last_fullsized":"2024-03-16T19:32:30.223858+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538424+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538424+0000","last_peered":"2024-03-16T19:32:26.538424+0000","last_clean":"2024-03-16T19:32:26.538424+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538424+0000","last_undegraded":"2024-03-16T19:32:26.538424+0000","last_fullsized":"2024-03-16T19:32:26.538424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":31,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076164+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076164+0000","last_peered":"2024-03-16T19:32:35.076164+0000","last_clean":"2024-03-16T19:32:35.076164+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076164+0000","last_undegraded":"2024-03-16T19:32:35.076164+0000","last_fullsized":"2024-03-16T19:32:35.076164+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":247,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954039+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954039+0000","last_peered":"2024-03-16T19:32:38.954039+0000","last_clean":"2024-03-16T19:32:38.954039+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954039+0000","last_undegraded":"2024-03-16T19:32:38.954039+0000","last_fullsized":"2024-03-16T19:32:38.954039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179941+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179941+0000","last_peered":"2024-03-16T19:32:41.179941+0000","last_clean":"2024-03-16T19:32:41.179941+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179941+0000","last_undegraded":"2024-03-16T19:32:41.179941+0000","last_fullsized":"2024-03-16T19:32:41.179941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245702+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245702+0000","last_peered":"2024-03-16T19:32:36.245702+0000","last_clean":"2024-03-16T19:32:36.245702+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245702+0000","last_undegraded":"2024-03-16T19:32:36.245702+0000","last_fullsized":"2024-03-16T19:32:36.245702+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197908+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197908+0000","last_peered":"2024-03-16T19:32:27.197908+0000","last_clean":"2024-03-16T19:32:27.197908+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197908+0000","last_undegraded":"2024-03-16T19:32:27.197908+0000","last_fullsized":"2024-03-16T19:32:27.197908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696958+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696958+0000","last_peered":"2024-03-16T19:32:36.696958+0000","last_clean":"2024-03-16T19:32:36.696958+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696958+0000","last_undegraded":"2024-03-16T19:32:36.696958+0000","last_fullsized":"2024-03-16T19:32:36.696958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190754+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190754+0000","last_peered":"2024-03-16T19:32:37.190754+0000","last_clean":"2024-03-16T19:32:37.190754+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190754+0000","last_undegraded":"2024-03-16T19:32:37.190754+0000","last_fullsized":"2024-03-16T19:32:37.190754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938129+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938129+0000","last_peered":"2024-03-16T19:32:29.938129+0000","last_clean":"2024-03-16T19:32:29.938129+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938129+0000","last_undegraded":"2024-03-16T19:32:29.938129+0000","last_fullsized":"2024-03-16T19:32:29.938129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932726+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932726+0000","last_peered":"2024-03-16T19:32:39.932726+0000","last_clean":"2024-03-16T19:32:39.932726+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932726+0000","last_undegraded":"2024-03-16T19:32:39.932726+0000","last_fullsized":"2024-03-16T19:32:39.932726+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":70,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:34:02.990147+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:34:02.990147+0000","last_peered":"2024-03-16T19:34:02.990147+0000","last_clean":"2024-03-16T19:34:02.990147+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:34:02.990147+0000","last_undegraded":"2024-03-16T19:34:02.990147+0000","last_fullsized":"2024-03-16T19:34:02.990147+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 108s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176466+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176466+0000","last_peered":"2024-03-16T19:32:38.176466+0000","last_clean":"2024-03-16T19:32:38.176466+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176466+0000","last_undegraded":"2024-03-16T19:32:38.176466+0000","last_fullsized":"2024-03-16T19:32:38.176466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":215,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982088+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982088+0000","last_peered":"2024-03-16T19:32:31.982088+0000","last_clean":"2024-03-16T19:32:31.982088+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982088+0000","last_undegraded":"2024-03-16T19:32:31.982088+0000","last_fullsized":"2024-03-16T19:32:31.982088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149110+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149110+0000","last_peered":"2024-03-16T19:32:39.149110+0000","last_clean":"2024-03-16T19:32:39.149110+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149110+0000","last_undegraded":"2024-03-16T19:32:39.149110+0000","last_fullsized":"2024-03-16T19:32:39.149110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108585+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108585+0000","last_peered":"2024-03-16T19:32:32.108585+0000","last_clean":"2024-03-16T19:32:32.108585+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108585+0000","last_undegraded":"2024-03-16T19:32:32.108585+0000","last_fullsized":"2024-03-16T19:32:32.108585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059668+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059668+0000","last_peered":"2024-03-16T19:32:31.059668+0000","last_clean":"2024-03-16T19:32:31.059668+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059668+0000","last_undegraded":"2024-03-16T19:32:31.059668+0000","last_fullsized":"2024-03-16T19:32:31.059668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568574+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568574+0000","last_peered":"2024-03-16T19:32:27.568574+0000","last_clean":"2024-03-16T19:32:27.568574+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568574+0000","last_undegraded":"2024-03-16T19:32:27.568574+0000","last_fullsized":"2024-03-16T19:32:27.568574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651659+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651659+0000","last_peered":"2024-03-16T19:32:37.651659+0000","last_clean":"2024-03-16T19:32:37.651659+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651659+0000","last_undegraded":"2024-03-16T19:32:37.651659+0000","last_fullsized":"2024-03-16T19:32:37.651659+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":61,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205835+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205835+0000","last_peered":"2024-03-16T19:32:28.205835+0000","last_clean":"2024-03-16T19:32:28.205835+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205835+0000","last_undegraded":"2024-03-16T19:32:28.205835+0000","last_fullsized":"2024-03-16T19:32:28.205835+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":165,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166583+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166583+0000","last_peered":"2024-03-16T19:32:40.166583+0000","last_clean":"2024-03-16T19:32:40.166583+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166583+0000","last_undegraded":"2024-03-16T19:32:40.166583+0000","last_fullsized":"2024-03-16T19:32:40.166583+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":375,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:47.115779+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:32:47.115779+0000","last_peered":"2024-03-16T19:32:47.115779+0000","last_clean":"2024-03-16T19:32:47.115779+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:47.115779+0000","last_undegraded":"2024-03-16T19:32:47.115779+0000","last_fullsized":"2024-03-16T19:32:47.115779+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.532575+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:32:42.532575+0000","last_peered":"2024-03-16T19:32:42.532575+0000","last_clean":"2024-03-16T19:32:42.532575+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:42.532575+0000","last_undegraded":"2024-03-16T19:32:42.532575+0000","last_fullsized":"2024-03-16T19:32:42.532575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.219294+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:32:29.219294+0000","last_peered":"2024-03-16T19:32:29.219294+0000","last_clean":"2024-03-16T19:32:29.219294+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:29.219294+0000","last_undegraded":"2024-03-16T19:32:29.219294+0000","last_fullsized":"2024-03-16T19:32:29.219294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.162302+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:32:34.162302+0000","last_peered":"2024-03-16T19:32:34.162302+0000","last_clean":"2024-03-16T19:32:34.162302+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:34.162302+0000","last_undegraded":"2024-03-16T19:32:34.162302+0000","last_fullsized":"2024-03-16T19:32:34.162302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240521+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240521+0000","last_peered":"2024-03-16T19:32:34.240521+0000","last_clean":"2024-03-16T19:32:34.240521+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240521+0000","last_undegraded":"2024-03-16T19:32:34.240521+0000","last_fullsized":"2024-03-16T19:32:34.240521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.145030+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.145030+0000","last_peered":"2024-03-16T19:32:33.145030+0000","last_clean":"2024-03-16T19:32:33.145030+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.145030+0000","last_undegraded":"2024-03-16T19:32:33.145030+0000","last_fullsized":"2024-03-16T19:32:33.145030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:34:08.244151+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:34:08.244151+0000","last_peered":"2024-03-16T19:34:08.244151+0000","last_clean":"2024-03-16T19:34:08.244151+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:34:08.244151+0000","last_undegraded":"2024-03-16T19:34:08.244151+0000","last_fullsized":"2024-03-16T19:34:08.244151+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 107s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.692412+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:32:30.692412+0000","last_peered":"2024-03-16T19:32:30.692412+0000","last_clean":"2024-03-16T19:32:30.692412+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:30.692412+0000","last_undegraded":"2024-03-16T19:32:30.692412+0000","last_fullsized":"2024-03-16T19:32:30.692412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.998707+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:32:30.998707+0000","last_peered":"2024-03-16T19:32:30.998707+0000","last_clean":"2024-03-16T19:32:30.998707+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:30.998707+0000","last_undegraded":"2024-03-16T19:32:30.998707+0000","last_fullsized":"2024-03-16T19:32:30.998707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.721534+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:32:37.721534+0000","last_peered":"2024-03-16T19:32:37.721534+0000","last_clean":"2024-03-16T19:32:37.721534+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:37.721534+0000","last_undegraded":"2024-03-16T19:32:37.721534+0000","last_fullsized":"2024-03-16T19:32:37.721534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.589670+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:32:39.589670+0000","last_peered":"2024-03-16T19:32:39.589670+0000","last_clean":"2024-03-16T19:32:39.589670+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:39.589670+0000","last_undegraded":"2024-03-16T19:32:39.589670+0000","last_fullsized":"2024-03-16T19:32:39.589670+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:46.142320+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:32:46.142320+0000","last_peered":"2024-03-16T19:32:46.142320+0000","last_clean":"2024-03-16T19:32:46.142320+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:46.142320+0000","last_undegraded":"2024-03-16T19:32:46.142320+0000","last_fullsized":"2024-03-16T19:32:46.142320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":197,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:34:08.244209+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:34:08.244209+0000","last_peered":"2024-03-16T19:34:08.244209+0000","last_clean":"2024-03-16T19:34:08.244209+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:34:08.244209+0000","last_undegraded":"2024-03-16T19:34:08.244209+0000","last_fullsized":"2024-03-16T19:34:08.244209+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 108s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586903+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586903+0000","last_peered":"2024-03-16T19:32:29.586903+0000","last_clean":"2024-03-16T19:32:29.586903+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586903+0000","last_undegraded":"2024-03-16T19:32:29.586903+0000","last_fullsized":"2024-03-16T19:32:29.586903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.125506+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:32:33.125506+0000","last_peered":"2024-03-16T19:32:33.125506+0000","last_clean":"2024-03-16T19:32:33.125506+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:33.125506+0000","last_undegraded":"2024-03-16T19:32:33.125506+0000","last_fullsized":"2024-03-16T19:32:33.125506+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910678+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910678+0000","last_peered":"2024-03-16T19:32:31.910678+0000","last_clean":"2024-03-16T19:32:31.910678+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910678+0000","last_undegraded":"2024-03-16T19:32:31.910678+0000","last_fullsized":"2024-03-16T19:32:31.910678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188030+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188030+0000","last_peered":"2024-03-16T19:32:32.188030+0000","last_clean":"2024-03-16T19:32:32.188030+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188030+0000","last_undegraded":"2024-03-16T19:32:32.188030+0000","last_fullsized":"2024-03-16T19:32:32.188030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190121+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190121+0000","last_peered":"2024-03-16T19:32:44.190121+0000","last_clean":"2024-03-16T19:32:44.190121+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190121+0000","last_undegraded":"2024-03-16T19:32:44.190121+0000","last_fullsized":"2024-03-16T19:32:44.190121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026167+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026167+0000","last_peered":"2024-03-16T19:32:30.026167+0000","last_clean":"2024-03-16T19:32:30.026167+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026167+0000","last_undegraded":"2024-03-16T19:32:30.026167+0000","last_fullsized":"2024-03-16T19:32:30.026167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604791+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604791+0000","last_peered":"2024-03-16T19:32:28.604791+0000","last_clean":"2024-03-16T19:32:28.604791+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604791+0000","last_undegraded":"2024-03-16T19:32:28.604791+0000","last_fullsized":"2024-03-16T19:32:28.604791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:34:07.990555+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:34:07.990555+0000","last_peered":"2024-03-16T19:34:07.990555+0000","last_clean":"2024-03-16T19:34:07.990555+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:34:07.990555+0000","last_undegraded":"2024-03-16T19:34:07.990555+0000","last_fullsized":"2024-03-16T19:34:07.990555+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 108s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699900+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699900+0000","last_peered":"2024-03-16T19:32:33.699900+0000","last_clean":"2024-03-16T19:32:33.699900+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699900+0000","last_undegraded":"2024-03-16T19:32:33.699900+0000","last_fullsized":"2024-03-16T19:32:33.699900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030458+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030458+0000","last_peered":"2024-03-16T19:32:39.030458+0000","last_clean":"2024-03-16T19:32:39.030458+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030458+0000","last_undegraded":"2024-03-16T19:32:39.030458+0000","last_fullsized":"2024-03-16T19:32:39.030458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152425+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152425+0000","last_peered":"2024-03-16T19:32:33.152425+0000","last_clean":"2024-03-16T19:32:33.152425+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152425+0000","last_undegraded":"2024-03-16T19:32:33.152425+0000","last_fullsized":"2024-03-16T19:32:33.152425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889937+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889937+0000","last_peered":"2024-03-16T19:32:33.889937+0000","last_clean":"2024-03-16T19:32:33.889937+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889937+0000","last_undegraded":"2024-03-16T19:32:33.889937+0000","last_fullsized":"2024-03-16T19:32:33.889937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695513+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695513+0000","last_peered":"2024-03-16T19:32:31.695513+0000","last_clean":"2024-03-16T19:32:31.695513+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695513+0000","last_undegraded":"2024-03-16T19:32:31.695513+0000","last_fullsized":"2024-03-16T19:32:31.695513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102649+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102649+0000","last_peered":"2024-03-16T19:32:32.102649+0000","last_clean":"2024-03-16T19:32:32.102649+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102649+0000","last_undegraded":"2024-03-16T19:32:32.102649+0000","last_fullsized":"2024-03-16T19:32:32.102649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696352+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696352+0000","last_peered":"2024-03-16T19:32:32.696352+0000","last_clean":"2024-03-16T19:32:32.696352+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696352+0000","last_undegraded":"2024-03-16T19:32:32.696352+0000","last_fullsized":"2024-03-16T19:32:32.696352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177567+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177567+0000","last_peered":"2024-03-16T19:32:35.177567+0000","last_clean":"2024-03-16T19:32:35.177567+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177567+0000","last_undegraded":"2024-03-16T19:32:35.177567+0000","last_fullsized":"2024-03-16T19:32:35.177567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":180,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230753+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230753+0000","last_peered":"2024-03-16T19:32:33.230753+0000","last_clean":"2024-03-16T19:32:33.230753+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230753+0000","last_undegraded":"2024-03-16T19:32:33.230753+0000","last_fullsized":"2024-03-16T19:32:33.230753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.046537+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:32:40.046537+0000","last_peered":"2024-03-16T19:32:40.046537+0000","last_clean":"2024-03-16T19:32:40.046537+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:40.046537+0000","last_undegraded":"2024-03-16T19:32:40.046537+0000","last_fullsized":"2024-03-16T19:32:40.046537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":175,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.063436+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:32:37.063436+0000","last_peered":"2024-03-16T19:32:37.063436+0000","last_clean":"2024-03-16T19:32:37.063436+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:37.063436+0000","last_undegraded":"2024-03-16T19:32:37.063436+0000","last_fullsized":"2024-03-16T19:32:37.063436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272673+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272673+0000","last_peered":"2024-03-16T19:32:29.272673+0000","last_clean":"2024-03-16T19:32:29.272673+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272673+0000","last_undegraded":"2024-03-16T19:32:29.272673+0000","last_fullsized":"2024-03-16T19:32:29.272673+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122082,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:34:32.780 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:32.781 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:32.781 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:32.781 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:32.781 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:34:52.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:34:52.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.928+0000 7f6908c6b640 1 Processor -- start 2024-03-16T19:34:52.908 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.929+0000 7f6908c6b640 1 -- start start 2024-03-16T19:34:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6908c6b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f69040cac30 0x7f69040cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6908c6b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040c8ba0 0x7f69040c3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:52.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6908c6b640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f690406fe10 0x7f69040c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6908c6b640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69040cb550 con 0x7f690406fe10 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6908c6b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6904162c10 con 0x7f69040cac30 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6908c6b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69040caa30 con 0x7f69040c8ba0 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6902d9d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f690406fe10 0x7f69040c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6901d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040c8ba0 0x7f69040c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f690259c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f69040cac30 0x7f69040cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6901d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040c8ba0 0x7f69040c3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:60222/0 (socket says 172.21.15.172:60222) 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6901d9b640 1 -- 172.21.15.172:0/3793848582 learned_addr learned my addr 172.21.15.172:0/3793848582 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f690259c640 1 -- 172.21.15.172:0/3793848582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040c8ba0 msgr2=0x7f69040c3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f690259c640 1 --2- 172.21.15.172:0/3793848582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040c8ba0 0x7f69040c3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f690259c640 1 -- 172.21.15.172:0/3793848582 >> v2:172.21.15.172:3300/0 conn(0x7f690406fe10 msgr2=0x7f69040c3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f690259c640 1 --2- 172.21.15.172:0/3793848582 >> v2:172.21.15.172:3300/0 conn(0x7f690406fe10 0x7f69040c3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:52.912 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f690259c640 1 -- 172.21.15.172:0/3793848582 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6904163130 con 0x7f69040cac30 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.930+0000 7f6902d9d640 1 --2- 172.21.15.172:0/3793848582 >> v2:172.21.15.172:3300/0 conn(0x7f690406fe10 0x7f69040c3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.931+0000 7f690259c640 1 --2- 172.21.15.172:0/3793848582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69040cac30 0x7f69040cb010 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f69040c8860 tx=0x7f68f402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25a39ca91a766dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.931+0000 7f690159a640 1 -- 172.21.15.172:0/3793848582 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f402e070 con 0x7f69040cac30 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.931+0000 7f690159a640 1 -- 172.21.15.172:0/3793848582 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f40315a0 con 0x7f69040cac30 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.931+0000 7f690159a640 1 -- 172.21.15.172:0/3793848582 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4031930 con 0x7f69040cac30 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.932+0000 7f6908c6b640 1 -- 172.21.15.172:0/3793848582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69040cac30 msgr2=0x7f69040cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.932+0000 7f6908c6b640 1 --2- 172.21.15.172:0/3793848582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69040cac30 0x7f69040cb010 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f69040c8860 tx=0x7f68f402b840 comp rx=0 tx=0).stop 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.932+0000 7f6908c6b640 1 -- 172.21.15.172:0/3793848582 shutdown_connections 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.932+0000 7f6908c6b640 1 --2- 172.21.15.172:0/3793848582 >> v2:172.21.15.172:3300/0 conn(0x7f690406fe10 0x7f69040c3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.932+0000 7f6908c6b640 1 --2- 172.21.15.172:0/3793848582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040c8ba0 0x7f69040c3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:52.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.932+0000 7f6908c6b640 1 --2- 172.21.15.172:0/3793848582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69040cac30 0x7f69040cb010 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.932+0000 7f6908c6b640 1 -- 172.21.15.172:0/3793848582 >> 172.21.15.172:0/3793848582 conn(0x7f6904070860 msgr2=0x7f6904070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.932+0000 7f6908c6b640 1 -- 172.21.15.172:0/3793848582 shutdown_connections 2024-03-16T19:34:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.932+0000 7f6908c6b640 1 -- 172.21.15.172:0/3793848582 wait complete. 2024-03-16T19:34:52.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.934+0000 7f6908c6b640 1 Processor -- start 2024-03-16T19:34:52.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.934+0000 7f6908c6b640 1 -- start start 2024-03-16T19:34:52.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.935+0000 7f6908c6b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690406fe10 0x7f690415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.935+0000 7f6908c6b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69040c8ba0 0x7f69040ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.935+0000 7f6908c6b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040cac30 0x7f6904151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.935+0000 7f6908c6b640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69040cf030 con 0x7f690406fe10 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.935+0000 7f6908c6b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041f7960 con 0x7f69040c8ba0 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.935+0000 7f6908c6b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69041f7c40 con 0x7f69040cac30 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f690259c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690406fe10 0x7f690415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f6902d9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040cac30 0x7f6904151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f690259c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690406fe10 0x7f690415deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45914/0 (socket says 172.21.15.172:45914) 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f690259c640 1 -- 172.21.15.172:0/250047330 learned_addr learned my addr 172.21.15.172:0/250047330 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f6901d9b640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69040c8ba0 0x7f69040ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f690259c640 1 -- 172.21.15.172:0/250047330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040cac30 msgr2=0x7f6904151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f690259c640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040cac30 0x7f6904151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f690259c640 1 -- 172.21.15.172:0/250047330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69040c8ba0 msgr2=0x7f69040ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f690259c640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69040c8ba0 0x7f69040ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f690259c640 1 -- 172.21.15.172:0/250047330 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc009a70 con 0x7f690406fe10 2024-03-16T19:34:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f6902d9d640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040cac30 0x7f6904151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:34:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f690259c640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690406fe10 0x7f690415deb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009890 tx=0x7f68f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.936+0000 7f68f2ffd640 1 -- 172.21.15.172:0/250047330 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4032630 con 0x7f690406fe10 2024-03-16T19:34:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.937+0000 7f68f2ffd640 1 -- 172.21.15.172:0/250047330 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f4032790 con 0x7f690406fe10 2024-03-16T19:34:52.918 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.937+0000 7f68f2ffd640 1 -- 172.21.15.172:0/250047330 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4032b40 con 0x7f690406fe10 2024-03-16T19:34:52.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.937+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f4009400 con 0x7f690406fe10 2024-03-16T19:34:52.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.937+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69041f8200 con 0x7f690406fe10 2024-03-16T19:34:52.919 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.938+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69040681c0 con 0x7f690406fe10 2024-03-16T19:34:52.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.939+0000 7f68f2ffd640 1 -- 172.21.15.172:0/250047330 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f68f4032ca0 con 0x7f690406fe10 2024-03-16T19:34:52.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.939+0000 7f68f2ffd640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f68e406b400 0x7f68e406d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:52.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.940+0000 7f68f2ffd640 1 -- 172.21.15.172:0/250047330 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f68f40ab970 con 0x7f690406fe10 2024-03-16T19:34:52.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.940+0000 7f6901d9b640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f68e406b400 0x7f68e406d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:52.920 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.940+0000 7f6901d9b640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f68e406b400 0x7f68e406d8c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6904161850 tx=0x7f68f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:52.926 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:52.947+0000 7f68f2ffd640 1 -- 172.21.15.172:0/250047330 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68f4076610 con 0x7f690406fe10 2024-03-16T19:34:53.256 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.278+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6904162120 con 0x7f68e406b400 2024-03-16T19:34:53.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.285+0000 7f68f2ffd640 1 -- 172.21.15.172:0/250047330 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252929 (secure 0 0 0) 0x7f6904162120 con 0x7f68e406b400 2024-03-16T19:34:53.267 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:34:53.268 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:34:53.269 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f68e406b400 msgr2=0x7f68e406d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f68e406b400 0x7f68e406d8c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6904161850 tx=0x7f68f8008040 comp rx=0 tx=0).stop 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690406fe10 msgr2=0x7f690415deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690406fe10 0x7f690415deb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009890 tx=0x7f68f40046c0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 shutdown_connections 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f68e406b400 0x7f68e406d8c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69040cac30 0x7f6904151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69040c8ba0 0x7f69040ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 --2- 172.21.15.172:0/250047330 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f690406fe10 0x7f690415deb0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 >> 172.21.15.172:0/250047330 conn(0x7f6904070860 msgr2=0x7f690415f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 shutdown_connections 2024-03-16T19:34:53.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.290+0000 7f6908c6b640 1 -- 172.21.15.172:0/250047330 wait complete. 2024-03-16T19:34:53.284 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-03-16T19:34:52.263369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002324"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.221338+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:32:36.221338+0000","last_peered":"2024-03-16T19:32:36.221338+0000","last_clean":"2024-03-16T19:32:36.221338+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:36.221338+0000","last_undegraded":"2024-03-16T19:32:36.221338+0000","last_fullsized":"2024-03-16T19:32:36.221338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.681944+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:32:36.681944+0000","last_peered":"2024-03-16T19:32:36.681944+0000","last_clean":"2024-03-16T19:32:36.681944+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:36.681944+0000","last_undegraded":"2024-03-16T19:32:36.681944+0000","last_fullsized":"2024-03-16T19:32:36.681944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":113,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.236924+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:32:28.236924+0000","last_peered":"2024-03-16T19:32:28.236924+0000","last_clean":"2024-03-16T19:32:28.236924+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:28.236924+0000","last_undegraded":"2024-03-16T19:32:28.236924+0000","last_fullsized":"2024-03-16T19:32:28.236924+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.687123+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:32:35.687123+0000","last_peered":"2024-03-16T19:32:35.687123+0000","last_clean":"2024-03-16T19:32:35.687123+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:35.687123+0000","last_undegraded":"2024-03-16T19:32:35.687123+0000","last_fullsized":"2024-03-16T19:32:35.687123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193204+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193204+0000","last_peered":"2024-03-16T19:32:42.193204+0000","last_clean":"2024-03-16T19:32:42.193204+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193204+0000","last_undegraded":"2024-03-16T19:32:42.193204+0000","last_fullsized":"2024-03-16T19:32:42.193204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501752+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501752+0000","last_peered":"2024-03-16T19:32:43.501752+0000","last_clean":"2024-03-16T19:32:43.501752+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501752+0000","last_undegraded":"2024-03-16T19:32:43.501752+0000","last_fullsized":"2024-03-16T19:32:43.501752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180916+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180916+0000","last_peered":"2024-03-16T19:32:45.180916+0000","last_clean":"2024-03-16T19:32:45.180916+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180916+0000","last_undegraded":"2024-03-16T19:32:45.180916+0000","last_fullsized":"2024-03-16T19:32:45.180916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142618+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142618+0000","last_peered":"2024-03-16T19:32:31.142618+0000","last_clean":"2024-03-16T19:32:31.142618+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142618+0000","last_undegraded":"2024-03-16T19:32:31.142618+0000","last_fullsized":"2024-03-16T19:32:31.142618+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050037+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050037+0000","last_peered":"2024-03-16T19:32:30.050037+0000","last_clean":"2024-03-16T19:32:30.050037+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050037+0000","last_undegraded":"2024-03-16T19:32:30.050037+0000","last_fullsized":"2024-03-16T19:32:30.050037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906526+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906526+0000","last_peered":"2024-03-16T19:32:36.906526+0000","last_clean":"2024-03-16T19:32:36.906526+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906526+0000","last_undegraded":"2024-03-16T19:32:36.906526+0000","last_fullsized":"2024-03-16T19:32:36.906526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946474+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946474+0000","last_peered":"2024-03-16T19:32:37.946474+0000","last_clean":"2024-03-16T19:32:37.946474+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946474+0000","last_undegraded":"2024-03-16T19:32:37.946474+0000","last_fullsized":"2024-03-16T19:32:37.946474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012365+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012365+0000","last_peered":"2024-03-16T19:32:29.012365+0000","last_clean":"2024-03-16T19:32:29.012365+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012365+0000","last_undegraded":"2024-03-16T19:32:29.012365+0000","last_fullsized":"2024-03-16T19:32:29.012365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085970+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085970+0000","last_peered":"2024-03-16T19:32:34.085970+0000","last_clean":"2024-03-16T19:32:34.085970+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085970+0000","last_undegraded":"2024-03-16T19:32:34.085970+0000","last_fullsized":"2024-03-16T19:32:34.085970+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194818+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194818+0000","last_peered":"2024-03-16T19:32:31.194818+0000","last_clean":"2024-03-16T19:32:31.194818+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194818+0000","last_undegraded":"2024-03-16T19:32:31.194818+0000","last_fullsized":"2024-03-16T19:32:31.194818+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622169+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622169+0000","last_peered":"2024-03-16T19:32:30.622169+0000","last_clean":"2024-03-16T19:32:30.622169+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622169+0000","last_undegraded":"2024-03-16T19:32:30.622169+0000","last_fullsized":"2024-03-16T19:32:30.622169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.528006+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.528006+0000","last_peered":"2024-03-16T19:32:41.528006+0000","last_clean":"2024-03-16T19:32:41.528006+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.528006+0000","last_undegraded":"2024-03-16T19:32:41.528006+0000","last_fullsized":"2024-03-16T19:32:41.528006+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668910+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668910+0000","last_peered":"2024-03-16T19:32:33.668910+0000","last_clean":"2024-03-16T19:32:33.668910+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668910+0000","last_undegraded":"2024-03-16T19:32:33.668910+0000","last_fullsized":"2024-03-16T19:32:33.668910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158111+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158111+0000","last_peered":"2024-03-16T19:32:30.158111+0000","last_clean":"2024-03-16T19:32:30.158111+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158111+0000","last_undegraded":"2024-03-16T19:32:30.158111+0000","last_fullsized":"2024-03-16T19:32:30.158111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198985+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198985+0000","last_peered":"2024-03-16T19:32:43.198985+0000","last_clean":"2024-03-16T19:32:43.198985+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198985+0000","last_undegraded":"2024-03-16T19:32:43.198985+0000","last_fullsized":"2024-03-16T19:32:43.198985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874927+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874927+0000","last_peered":"2024-03-16T19:32:34.874927+0000","last_clean":"2024-03-16T19:32:34.874927+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874927+0000","last_undegraded":"2024-03-16T19:32:34.874927+0000","last_fullsized":"2024-03-16T19:32:34.874927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202050+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202050+0000","last_peered":"2024-03-16T19:32:26.202050+0000","last_clean":"2024-03-16T19:32:26.202050+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202050+0000","last_undegraded":"2024-03-16T19:32:26.202050+0000","last_fullsized":"2024-03-16T19:32:26.202050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113826+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113826+0000","last_peered":"2024-03-16T19:32:31.113826+0000","last_clean":"2024-03-16T19:32:31.113826+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113826+0000","last_undegraded":"2024-03-16T19:32:31.113826+0000","last_fullsized":"2024-03-16T19:32:31.113826+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189280+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189280+0000","last_peered":"2024-03-16T19:32:30.189280+0000","last_clean":"2024-03-16T19:32:30.189280+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189280+0000","last_undegraded":"2024-03-16T19:32:30.189280+0000","last_fullsized":"2024-03-16T19:32:30.189280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633708+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633708+0000","last_peered":"2024-03-16T19:32:38.633708+0000","last_clean":"2024-03-16T19:32:38.633708+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633708+0000","last_undegraded":"2024-03-16T19:32:38.633708+0000","last_fullsized":"2024-03-16T19:32:38.633708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":172,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030046+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030046+0000","last_peered":"2024-03-16T19:32:36.030046+0000","last_clean":"2024-03-16T19:32:36.030046+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030046+0000","last_undegraded":"2024-03-16T19:32:36.030046+0000","last_fullsized":"2024-03-16T19:32:36.030046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881976+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881976+0000","last_peered":"2024-03-16T19:32:35.881976+0000","last_clean":"2024-03-16T19:32:35.881976+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881976+0000","last_undegraded":"2024-03-16T19:32:35.881976+0000","last_fullsized":"2024-03-16T19:32:35.881976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651765+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651765+0000","last_peered":"2024-03-16T19:32:31.651765+0000","last_clean":"2024-03-16T19:32:31.651765+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651765+0000","last_undegraded":"2024-03-16T19:32:31.651765+0000","last_fullsized":"2024-03-16T19:32:31.651765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.051224+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:32:35.051224+0000","last_peered":"2024-03-16T19:32:35.051224+0000","last_clean":"2024-03-16T19:32:35.051224+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:35.051224+0000","last_undegraded":"2024-03-16T19:32:35.051224+0000","last_fullsized":"2024-03-16T19:32:35.051224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209568+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209568+0000","last_peered":"2024-03-16T19:32:37.209568+0000","last_clean":"2024-03-16T19:32:37.209568+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209568+0000","last_undegraded":"2024-03-16T19:32:37.209568+0000","last_fullsized":"2024-03-16T19:32:37.209568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.540714+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:32:40.540714+0000","last_peered":"2024-03-16T19:32:40.540714+0000","last_clean":"2024-03-16T19:32:40.540714+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:40.540714+0000","last_undegraded":"2024-03-16T19:32:40.540714+0000","last_fullsized":"2024-03-16T19:32:40.540714+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216691+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216691+0000","last_peered":"2024-03-16T19:32:27.216691+0000","last_clean":"2024-03-16T19:32:27.216691+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216691+0000","last_undegraded":"2024-03-16T19:32:27.216691+0000","last_fullsized":"2024-03-16T19:32:27.216691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.897672+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:32:40.897672+0000","last_peered":"2024-03-16T19:32:40.897672+0000","last_clean":"2024-03-16T19:32:40.897672+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:40.897672+0000","last_undegraded":"2024-03-16T19:32:40.897672+0000","last_fullsized":"2024-03-16T19:32:40.897672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002778+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002778+0000","last_peered":"2024-03-16T19:32:28.002778+0000","last_clean":"2024-03-16T19:32:28.002778+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002778+0000","last_undegraded":"2024-03-16T19:32:28.002778+0000","last_fullsized":"2024-03-16T19:32:28.002778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207579+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207579+0000","last_peered":"2024-03-16T19:32:28.207579+0000","last_clean":"2024-03-16T19:32:28.207579+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207579+0000","last_undegraded":"2024-03-16T19:32:28.207579+0000","last_fullsized":"2024-03-16T19:32:28.207579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265410+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265410+0000","last_peered":"2024-03-16T19:32:35.265410+0000","last_clean":"2024-03-16T19:32:35.265410+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265410+0000","last_undegraded":"2024-03-16T19:32:35.265410+0000","last_fullsized":"2024-03-16T19:32:35.265410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913659+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913659+0000","last_peered":"2024-03-16T19:32:32.913659+0000","last_clean":"2024-03-16T19:32:32.913659+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913659+0000","last_undegraded":"2024-03-16T19:32:32.913659+0000","last_fullsized":"2024-03-16T19:32:32.913659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105281+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105281+0000","last_peered":"2024-03-16T19:32:32.105281+0000","last_clean":"2024-03-16T19:32:32.105281+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105281+0000","last_undegraded":"2024-03-16T19:32:32.105281+0000","last_fullsized":"2024-03-16T19:32:32.105281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062784+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062784+0000","last_peered":"2024-03-16T19:32:38.062784+0000","last_clean":"2024-03-16T19:32:38.062784+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062784+0000","last_undegraded":"2024-03-16T19:32:38.062784+0000","last_fullsized":"2024-03-16T19:32:38.062784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.686005+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.686005+0000","last_peered":"2024-03-16T19:32:28.686005+0000","last_clean":"2024-03-16T19:32:28.686005+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.686005+0000","last_undegraded":"2024-03-16T19:32:28.686005+0000","last_fullsized":"2024-03-16T19:32:28.686005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691258+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691258+0000","last_peered":"2024-03-16T19:32:32.691258+0000","last_clean":"2024-03-16T19:32:32.691258+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691258+0000","last_undegraded":"2024-03-16T19:32:32.691258+0000","last_fullsized":"2024-03-16T19:32:32.691258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944590+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944590+0000","last_peered":"2024-03-16T19:32:30.944590+0000","last_clean":"2024-03-16T19:32:30.944590+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944590+0000","last_undegraded":"2024-03-16T19:32:30.944590+0000","last_fullsized":"2024-03-16T19:32:30.944590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706805+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706805+0000","last_peered":"2024-03-16T19:32:34.706805+0000","last_clean":"2024-03-16T19:32:34.706805+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706805+0000","last_undegraded":"2024-03-16T19:32:34.706805+0000","last_fullsized":"2024-03-16T19:32:34.706805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646648+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646648+0000","last_peered":"2024-03-16T19:32:29.646648+0000","last_clean":"2024-03-16T19:32:29.646648+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646648+0000","last_undegraded":"2024-03-16T19:32:29.646648+0000","last_fullsized":"2024-03-16T19:32:29.646648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223858+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223858+0000","last_peered":"2024-03-16T19:32:30.223858+0000","last_clean":"2024-03-16T19:32:30.223858+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223858+0000","last_undegraded":"2024-03-16T19:32:30.223858+0000","last_fullsized":"2024-03-16T19:32:30.223858+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538424+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538424+0000","last_peered":"2024-03-16T19:32:26.538424+0000","last_clean":"2024-03-16T19:32:26.538424+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538424+0000","last_undegraded":"2024-03-16T19:32:26.538424+0000","last_fullsized":"2024-03-16T19:32:26.538424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":31,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076164+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076164+0000","last_peered":"2024-03-16T19:32:35.076164+0000","last_clean":"2024-03-16T19:32:35.076164+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076164+0000","last_undegraded":"2024-03-16T19:32:35.076164+0000","last_fullsized":"2024-03-16T19:32:35.076164+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":247,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954039+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954039+0000","last_peered":"2024-03-16T19:32:38.954039+0000","last_clean":"2024-03-16T19:32:38.954039+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954039+0000","last_undegraded":"2024-03-16T19:32:38.954039+0000","last_fullsized":"2024-03-16T19:32:38.954039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179941+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179941+0000","last_peered":"2024-03-16T19:32:41.179941+0000","last_clean":"2024-03-16T19:32:41.179941+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179941+0000","last_undegraded":"2024-03-16T19:32:41.179941+0000","last_fullsized":"2024-03-16T19:32:41.179941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245702+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245702+0000","last_peered":"2024-03-16T19:32:36.245702+0000","last_clean":"2024-03-16T19:32:36.245702+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245702+0000","last_undegraded":"2024-03-16T19:32:36.245702+0000","last_fullsized":"2024-03-16T19:32:36.245702+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197908+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197908+0000","last_peered":"2024-03-16T19:32:27.197908+0000","last_clean":"2024-03-16T19:32:27.197908+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197908+0000","last_undegraded":"2024-03-16T19:32:27.197908+0000","last_fullsized":"2024-03-16T19:32:27.197908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696958+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696958+0000","last_peered":"2024-03-16T19:32:36.696958+0000","last_clean":"2024-03-16T19:32:36.696958+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696958+0000","last_undegraded":"2024-03-16T19:32:36.696958+0000","last_fullsized":"2024-03-16T19:32:36.696958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190754+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190754+0000","last_peered":"2024-03-16T19:32:37.190754+0000","last_clean":"2024-03-16T19:32:37.190754+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190754+0000","last_undegraded":"2024-03-16T19:32:37.190754+0000","last_fullsized":"2024-03-16T19:32:37.190754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938129+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938129+0000","last_peered":"2024-03-16T19:32:29.938129+0000","last_clean":"2024-03-16T19:32:29.938129+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938129+0000","last_undegraded":"2024-03-16T19:32:29.938129+0000","last_fullsized":"2024-03-16T19:32:29.938129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932726+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932726+0000","last_peered":"2024-03-16T19:32:39.932726+0000","last_clean":"2024-03-16T19:32:39.932726+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932726+0000","last_undegraded":"2024-03-16T19:32:39.932726+0000","last_fullsized":"2024-03-16T19:32:39.932726+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":72,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:34:32.992890+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:34:32.992890+0000","last_peered":"2024-03-16T19:34:32.992890+0000","last_clean":"2024-03-16T19:34:32.992890+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:34:32.992890+0000","last_undegraded":"2024-03-16T19:34:32.992890+0000","last_fullsized":"2024-03-16T19:34:32.992890+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 138s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176466+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176466+0000","last_peered":"2024-03-16T19:32:38.176466+0000","last_clean":"2024-03-16T19:32:38.176466+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176466+0000","last_undegraded":"2024-03-16T19:32:38.176466+0000","last_fullsized":"2024-03-16T19:32:38.176466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":215,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982088+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982088+0000","last_peered":"2024-03-16T19:32:31.982088+0000","last_clean":"2024-03-16T19:32:31.982088+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982088+0000","last_undegraded":"2024-03-16T19:32:31.982088+0000","last_fullsized":"2024-03-16T19:32:31.982088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149110+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149110+0000","last_peered":"2024-03-16T19:32:39.149110+0000","last_clean":"2024-03-16T19:32:39.149110+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149110+0000","last_undegraded":"2024-03-16T19:32:39.149110+0000","last_fullsized":"2024-03-16T19:32:39.149110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108585+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108585+0000","last_peered":"2024-03-16T19:32:32.108585+0000","last_clean":"2024-03-16T19:32:32.108585+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108585+0000","last_undegraded":"2024-03-16T19:32:32.108585+0000","last_fullsized":"2024-03-16T19:32:32.108585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059668+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059668+0000","last_peered":"2024-03-16T19:32:31.059668+0000","last_clean":"2024-03-16T19:32:31.059668+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059668+0000","last_undegraded":"2024-03-16T19:32:31.059668+0000","last_fullsized":"2024-03-16T19:32:31.059668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568574+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568574+0000","last_peered":"2024-03-16T19:32:27.568574+0000","last_clean":"2024-03-16T19:32:27.568574+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568574+0000","last_undegraded":"2024-03-16T19:32:27.568574+0000","last_fullsized":"2024-03-16T19:32:27.568574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651659+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651659+0000","last_peered":"2024-03-16T19:32:37.651659+0000","last_clean":"2024-03-16T19:32:37.651659+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651659+0000","last_undegraded":"2024-03-16T19:32:37.651659+0000","last_fullsized":"2024-03-16T19:32:37.651659+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":61,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205835+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205835+0000","last_peered":"2024-03-16T19:32:28.205835+0000","last_clean":"2024-03-16T19:32:28.205835+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205835+0000","last_undegraded":"2024-03-16T19:32:28.205835+0000","last_fullsized":"2024-03-16T19:32:28.205835+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":166,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166603+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166603+0000","last_peered":"2024-03-16T19:32:40.166603+0000","last_clean":"2024-03-16T19:32:40.166603+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166603+0000","last_undegraded":"2024-03-16T19:32:40.166603+0000","last_fullsized":"2024-03-16T19:32:40.166603+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":375,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:47.115779+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:32:47.115779+0000","last_peered":"2024-03-16T19:32:47.115779+0000","last_clean":"2024-03-16T19:32:47.115779+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:47.115779+0000","last_undegraded":"2024-03-16T19:32:47.115779+0000","last_fullsized":"2024-03-16T19:32:47.115779+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.532575+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:32:42.532575+0000","last_peered":"2024-03-16T19:32:42.532575+0000","last_clean":"2024-03-16T19:32:42.532575+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:42.532575+0000","last_undegraded":"2024-03-16T19:32:42.532575+0000","last_fullsized":"2024-03-16T19:32:42.532575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.219294+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:32:29.219294+0000","last_peered":"2024-03-16T19:32:29.219294+0000","last_clean":"2024-03-16T19:32:29.219294+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:29.219294+0000","last_undegraded":"2024-03-16T19:32:29.219294+0000","last_fullsized":"2024-03-16T19:32:29.219294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.162302+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:32:34.162302+0000","last_peered":"2024-03-16T19:32:34.162302+0000","last_clean":"2024-03-16T19:32:34.162302+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:34.162302+0000","last_undegraded":"2024-03-16T19:32:34.162302+0000","last_fullsized":"2024-03-16T19:32:34.162302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240521+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240521+0000","last_peered":"2024-03-16T19:32:34.240521+0000","last_clean":"2024-03-16T19:32:34.240521+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240521+0000","last_undegraded":"2024-03-16T19:32:34.240521+0000","last_fullsized":"2024-03-16T19:32:34.240521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.145030+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.145030+0000","last_peered":"2024-03-16T19:32:33.145030+0000","last_clean":"2024-03-16T19:32:33.145030+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.145030+0000","last_undegraded":"2024-03-16T19:32:33.145030+0000","last_fullsized":"2024-03-16T19:32:33.145030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":198,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:34:23.245394+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:34:23.245394+0000","last_peered":"2024-03-16T19:34:23.245394+0000","last_clean":"2024-03-16T19:34:23.245394+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:34:23.245394+0000","last_undegraded":"2024-03-16T19:34:23.245394+0000","last_fullsized":"2024-03-16T19:34:23.245394+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 122s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.692412+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:32:30.692412+0000","last_peered":"2024-03-16T19:32:30.692412+0000","last_clean":"2024-03-16T19:32:30.692412+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:30.692412+0000","last_undegraded":"2024-03-16T19:32:30.692412+0000","last_fullsized":"2024-03-16T19:32:30.692412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.998707+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:32:30.998707+0000","last_peered":"2024-03-16T19:32:30.998707+0000","last_clean":"2024-03-16T19:32:30.998707+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:30.998707+0000","last_undegraded":"2024-03-16T19:32:30.998707+0000","last_fullsized":"2024-03-16T19:32:30.998707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.721534+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:32:37.721534+0000","last_peered":"2024-03-16T19:32:37.721534+0000","last_clean":"2024-03-16T19:32:37.721534+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:37.721534+0000","last_undegraded":"2024-03-16T19:32:37.721534+0000","last_fullsized":"2024-03-16T19:32:37.721534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.589670+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:32:39.589670+0000","last_peered":"2024-03-16T19:32:39.589670+0000","last_clean":"2024-03-16T19:32:39.589670+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:39.589670+0000","last_undegraded":"2024-03-16T19:32:39.589670+0000","last_fullsized":"2024-03-16T19:32:39.589670+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:46.142320+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:32:46.142320+0000","last_peered":"2024-03-16T19:32:46.142320+0000","last_clean":"2024-03-16T19:32:46.142320+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:46.142320+0000","last_undegraded":"2024-03-16T19:32:46.142320+0000","last_fullsized":"2024-03-16T19:32:46.142320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":198,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:34:23.245430+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:34:23.245430+0000","last_peered":"2024-03-16T19:34:23.245430+0000","last_clean":"2024-03-16T19:34:23.245430+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:34:23.245430+0000","last_undegraded":"2024-03-16T19:34:23.245430+0000","last_fullsized":"2024-03-16T19:34:23.245430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 123s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586903+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586903+0000","last_peered":"2024-03-16T19:32:29.586903+0000","last_clean":"2024-03-16T19:32:29.586903+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586903+0000","last_undegraded":"2024-03-16T19:32:29.586903+0000","last_fullsized":"2024-03-16T19:32:29.586903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.125506+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:32:33.125506+0000","last_peered":"2024-03-16T19:32:33.125506+0000","last_clean":"2024-03-16T19:32:33.125506+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:33.125506+0000","last_undegraded":"2024-03-16T19:32:33.125506+0000","last_fullsized":"2024-03-16T19:32:33.125506+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910678+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910678+0000","last_peered":"2024-03-16T19:32:31.910678+0000","last_clean":"2024-03-16T19:32:31.910678+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910678+0000","last_undegraded":"2024-03-16T19:32:31.910678+0000","last_fullsized":"2024-03-16T19:32:31.910678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188030+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188030+0000","last_peered":"2024-03-16T19:32:32.188030+0000","last_clean":"2024-03-16T19:32:32.188030+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188030+0000","last_undegraded":"2024-03-16T19:32:32.188030+0000","last_fullsized":"2024-03-16T19:32:32.188030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190121+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190121+0000","last_peered":"2024-03-16T19:32:44.190121+0000","last_clean":"2024-03-16T19:32:44.190121+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190121+0000","last_undegraded":"2024-03-16T19:32:44.190121+0000","last_fullsized":"2024-03-16T19:32:44.190121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026167+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026167+0000","last_peered":"2024-03-16T19:32:30.026167+0000","last_clean":"2024-03-16T19:32:30.026167+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026167+0000","last_undegraded":"2024-03-16T19:32:30.026167+0000","last_fullsized":"2024-03-16T19:32:30.026167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604791+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604791+0000","last_peered":"2024-03-16T19:32:28.604791+0000","last_clean":"2024-03-16T19:32:28.604791+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604791+0000","last_undegraded":"2024-03-16T19:32:28.604791+0000","last_fullsized":"2024-03-16T19:32:28.604791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:34:22.991858+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:34:22.991858+0000","last_peered":"2024-03-16T19:34:22.991858+0000","last_clean":"2024-03-16T19:34:22.991858+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:34:22.991858+0000","last_undegraded":"2024-03-16T19:34:22.991858+0000","last_fullsized":"2024-03-16T19:34:22.991858+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 123s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699900+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699900+0000","last_peered":"2024-03-16T19:32:33.699900+0000","last_clean":"2024-03-16T19:32:33.699900+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699900+0000","last_undegraded":"2024-03-16T19:32:33.699900+0000","last_fullsized":"2024-03-16T19:32:33.699900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030458+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030458+0000","last_peered":"2024-03-16T19:32:39.030458+0000","last_clean":"2024-03-16T19:32:39.030458+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030458+0000","last_undegraded":"2024-03-16T19:32:39.030458+0000","last_fullsized":"2024-03-16T19:32:39.030458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152425+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152425+0000","last_peered":"2024-03-16T19:32:33.152425+0000","last_clean":"2024-03-16T19:32:33.152425+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152425+0000","last_undegraded":"2024-03-16T19:32:33.152425+0000","last_fullsized":"2024-03-16T19:32:33.152425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889937+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889937+0000","last_peered":"2024-03-16T19:32:33.889937+0000","last_clean":"2024-03-16T19:32:33.889937+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889937+0000","last_undegraded":"2024-03-16T19:32:33.889937+0000","last_fullsized":"2024-03-16T19:32:33.889937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695513+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695513+0000","last_peered":"2024-03-16T19:32:31.695513+0000","last_clean":"2024-03-16T19:32:31.695513+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695513+0000","last_undegraded":"2024-03-16T19:32:31.695513+0000","last_fullsized":"2024-03-16T19:32:31.695513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102649+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102649+0000","last_peered":"2024-03-16T19:32:32.102649+0000","last_clean":"2024-03-16T19:32:32.102649+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102649+0000","last_undegraded":"2024-03-16T19:32:32.102649+0000","last_fullsized":"2024-03-16T19:32:32.102649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696352+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696352+0000","last_peered":"2024-03-16T19:32:32.696352+0000","last_clean":"2024-03-16T19:32:32.696352+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696352+0000","last_undegraded":"2024-03-16T19:32:32.696352+0000","last_fullsized":"2024-03-16T19:32:32.696352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177567+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177567+0000","last_peered":"2024-03-16T19:32:35.177567+0000","last_clean":"2024-03-16T19:32:35.177567+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177567+0000","last_undegraded":"2024-03-16T19:32:35.177567+0000","last_fullsized":"2024-03-16T19:32:35.177567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230775+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230775+0000","last_peered":"2024-03-16T19:32:33.230775+0000","last_clean":"2024-03-16T19:32:33.230775+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230775+0000","last_undegraded":"2024-03-16T19:32:33.230775+0000","last_fullsized":"2024-03-16T19:32:33.230775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.046537+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:32:40.046537+0000","last_peered":"2024-03-16T19:32:40.046537+0000","last_clean":"2024-03-16T19:32:40.046537+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:40.046537+0000","last_undegraded":"2024-03-16T19:32:40.046537+0000","last_fullsized":"2024-03-16T19:32:40.046537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":175,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.063436+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:32:37.063436+0000","last_peered":"2024-03-16T19:32:37.063436+0000","last_clean":"2024-03-16T19:32:37.063436+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:37.063436+0000","last_undegraded":"2024-03-16T19:32:37.063436+0000","last_fullsized":"2024-03-16T19:32:37.063436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272673+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272673+0000","last_peered":"2024-03-16T19:32:29.272673+0000","last_clean":"2024-03-16T19:32:29.272673+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272673+0000","last_undegraded":"2024-03-16T19:32:29.272673+0000","last_fullsized":"2024-03-16T19:32:29.272673+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574919,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574920,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574919,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122086,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:34:53.290 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:53.290 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:53.291 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:53.291 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:34:53.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 2.10 deep-scrub 2024-03-16T19:34:53.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.494+0000 7f5fdeb77640 1 Processor -- start 2024-03-16T19:34:53.473 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.494+0000 7f5fdeb77640 1 -- start start 2024-03-16T19:34:53.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.495+0000 7f5fdeb77640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5fd8158fb0 0x7f5fd806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.495+0000 7f5fdeb77640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5fd80c3380 0x7f5fd80c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.495+0000 7f5fdeb77640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80c3d70 con 0x7f5fd80c4f60 2024-03-16T19:34:53.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.495+0000 7f5fdeb77640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd8070120 con 0x7f5fd8158fb0 2024-03-16T19:34:53.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.495+0000 7f5fdeb77640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd8162db0 con 0x7f5fd80c3380 2024-03-16T19:34:53.475 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.496+0000 7f5fdc913640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5fd8158fb0 0x7f5fd806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.496+0000 7f5fdc913640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f5fd8158fb0 0x7f5fd806dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45926/0 (socket says 172.21.15.172:45926) 2024-03-16T19:34:53.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.496+0000 7f5fdc913640 1 -- 172.21.15.172:0/3284825363 learned_addr learned my addr 172.21.15.172:0/3284825363 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:53.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.496+0000 7f5fdd114640 1 --2- 172.21.15.172:0/3284825363 >> v2:172.21.15.174:3300/0 conn(0x7f5fd80c3380 0x7f5fd80c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.496+0000 7f5fdd114640 1 -- 172.21.15.172:0/3284825363 >> v2:172.21.15.172:3300/0 conn(0x7f5fd8158fb0 msgr2=0x7f5fd806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:34:53.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.496+0000 7f5fdd114640 1 --2- 172.21.15.172:0/3284825363 >> v2:172.21.15.172:3300/0 conn(0x7f5fd8158fb0 0x7f5fd806dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.496+0000 7f5fdd114640 1 -- 172.21.15.172:0/3284825363 >> v1:172.21.15.174:6789/0 conn(0x7f5fd80c4f60 legacy=0x7f5fd8162280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.476 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.496+0000 7f5fdd114640 1 -- 172.21.15.172:0/3284825363 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd8163090 con 0x7f5fd80c3380 2024-03-16T19:34:53.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.497+0000 7f5fdd114640 1 --2- 172.21.15.172:0/3284825363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fd80c3380 0x7f5fd80c3830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80714e0 tx=0x7f5fd002bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fce0f4383603575c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.497+0000 7f5fd77fe640 1 -- 172.21.15.172:0/3284825363 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd002cac0 con 0x7f5fd80c3380 2024-03-16T19:34:53.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.497+0000 7f5fd77fe640 1 -- 172.21.15.172:0/3284825363 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd002cc20 con 0x7f5fd80c3380 2024-03-16T19:34:53.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.497+0000 7f5fd77fe640 1 -- 172.21.15.172:0/3284825363 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0034440 con 0x7f5fd80c3380 2024-03-16T19:34:53.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.498+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3284825363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fd80c3380 msgr2=0x7f5fd80c3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.478 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.498+0000 7f5fdeb77640 1 --2- 172.21.15.172:0/3284825363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fd80c3380 0x7f5fd80c3830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80714e0 tx=0x7f5fd002bd90 comp rx=0 tx=0).stop 2024-03-16T19:34:53.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.498+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3284825363 shutdown_connections 2024-03-16T19:34:53.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.498+0000 7f5fdeb77640 1 --2- 172.21.15.172:0/3284825363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fd80c3380 0x7f5fd80c3830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.498+0000 7f5fdeb77640 1 --2- 172.21.15.172:0/3284825363 >> v2:172.21.15.172:3300/0 conn(0x7f5fd8158fb0 0x7f5fd806dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.498+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3284825363 >> 172.21.15.172:0/3284825363 conn(0x7f5fd8070810 msgr2=0x7f5fd8070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:53.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.498+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3284825363 shutdown_connections 2024-03-16T19:34:53.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.498+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3284825363 wait complete. 2024-03-16T19:34:53.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.499+0000 7f5fdeb77640 1 Processor -- start 2024-03-16T19:34:53.479 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.499+0000 7f5fdeb77640 1 -- start start 2024-03-16T19:34:53.481 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdeb77640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fd80c3380 0x7f5fd814e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.481 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdeb77640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fd80c4f60 0x7f5fd814eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.481 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdeb77640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fd8158fb0 0x7f5fd8153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.481 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdeb77640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd80c5830 con 0x7f5fd80c3380 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdeb77640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd81f64c0 con 0x7f5fd80c4f60 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdeb77640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd81f67a0 con 0x7f5fd8158fb0 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdc913640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fd80c3380 0x7f5fd814e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdc913640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fd80c3380 0x7f5fd814e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45936/0 (socket says 172.21.15.172:45936) 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdc913640 1 -- 172.21.15.172:0/3215246914 learned_addr learned my addr 172.21.15.172:0/3215246914 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fd7fff640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fd80c4f60 0x7f5fd814eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.500+0000 7f5fdd114640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fd8158fb0 0x7f5fd8153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fdc913640 1 -- 172.21.15.172:0/3215246914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fd8158fb0 msgr2=0x7f5fd8153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fdc913640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5fd8158fb0 0x7f5fd8153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fdc913640 1 -- 172.21.15.172:0/3215246914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fd80c4f60 msgr2=0x7f5fd814eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fdc913640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fd80c4f60 0x7f5fd814eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fdc913640 1 -- 172.21.15.172:0/3215246914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd0009a70 con 0x7f5fd80c3380 2024-03-16T19:34:53.482 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fdc913640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fd80c3380 0x7f5fd814e5d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009fd0 tx=0x7f5fcc00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fd57fa640 1 -- 172.21.15.172:0/3215246914 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00c940 con 0x7f5fd80c3380 2024-03-16T19:34:53.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fd57fa640 1 -- 172.21.15.172:0/3215246914 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc00caa0 con 0x7f5fd80c3380 2024-03-16T19:34:53.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fd57fa640 1 -- 172.21.15.172:0/3215246914 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00ce30 con 0x7f5fd80c3380 2024-03-16T19:34:53.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd81f6a80 con 0x7f5fd80c3380 2024-03-16T19:34:53.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.501+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd81f7040 con 0x7f5fd80c3380 2024-03-16T19:34:53.483 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.503+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5fd806c2c0 con 0x7f5fd80c3380 2024-03-16T19:34:53.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.504+0000 7f5fd57fa640 1 -- 172.21.15.172:0/3215246914 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5fcc013470 con 0x7f5fd80c3380 2024-03-16T19:34:53.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.504+0000 7f5fd57fa640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fbc06b3b0 0x7f5fbc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.504+0000 7f5fd7fff640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fbc06b3b0 0x7f5fbc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.504+0000 7f5fd7fff640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fbc06b3b0 0x7f5fbc06d870 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80c40b0 tx=0x7f5fc4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.505+0000 7f5fd57fa640 1 -- 172.21.15.172:0/3215246914 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f5fcc057ef0 con 0x7f5fd80c3380 2024-03-16T19:34:53.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.505+0000 7f5fd57fa640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f5fbc0737f0 0x7f5fbc075cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.505+0000 7f5fd57fa640 1 -- 172.21.15.172:0/3215246914 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 1: {"prefix": "get_command_descriptions", "pgid": "2.10"}) v1 -- 0x7f5fbc076380 con 0x7f5fbc0737f0 2024-03-16T19:34:53.484 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.505+0000 7f5fd57fa640 1 -- 172.21.15.172:0/3215246914 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fcc054190 con 0x7f5fd80c3380 2024-03-16T19:34:53.486 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.508+0000 7f5fdd114640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f5fbc0737f0 0x7f5fbc075cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.508+0000 7f5fdd114640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f5fbc0737f0 0x7f5fbc075cb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.488 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.509+0000 7f5fd57fa640 1 -- 172.21.15.172:0/3215246914 <== osd.2 v2:172.21.15.172:6823/3572014952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5fbc076380 con 0x7f5fbc0737f0 2024-03-16T19:34:53.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.529+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 2: {"prefix": "deep-scrub", "pgid": "2.10"}) v1 -- 0x7f5fd80c8cf0 con 0x7f5fbc0737f0 2024-03-16T19:34:53.508 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.529+0000 7f5fd57fa640 1 -- 172.21.15.172:0/3215246914 <== osd.2 v2:172.21.15.172:6823/3572014952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+64 (crc 0 0 0) 0x7f5fd0009450 con 0x7f5fbc0737f0 2024-03-16T19:34:53.508 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:34:53.508 INFO:teuthology.orchestra.run.smithi172.stdout: "deep": true, 2024-03-16T19:34:53.509 INFO:teuthology.orchestra.run.smithi172.stdout: "must": true, 2024-03-16T19:34:53.509 INFO:teuthology.orchestra.run.smithi172.stdout: "stamp": "0.000000" 2024-03-16T19:34:53.509 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f5fbc0737f0 msgr2=0x7f5fbc075cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f5fbc0737f0 0x7f5fbc075cb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fbc06b3b0 msgr2=0x7f5fbc06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f5fbc06b3b0 0x7f5fbc06d870 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80c40b0 tx=0x7f5fc4009040 comp rx=0 tx=0).stop 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fd80c3380 msgr2=0x7f5fd814e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 --2- 172.21.15.172:0/3215246914 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fd80c3380 0x7f5fd814e5d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009fd0 tx=0x7f5fcc00ed40 comp rx=0 tx=0).stop 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdd114640 1 -- 172.21.15.172:0/3215246914 reap_dead start 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 shutdown_connections 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 >> 172.21.15.172:0/3215246914 conn(0x7f5fd8070810 msgr2=0x7f5fd80c4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 shutdown_connections 2024-03-16T19:34:53.510 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.530+0000 7f5fdeb77640 1 -- 172.21.15.172:0/3215246914 wait complete. 2024-03-16T19:34:53.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 3.0 deep-scrub 2024-03-16T19:34:53.630 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.652+0000 7f0ce9eff640 1 Processor -- start 2024-03-16T19:34:53.630 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.652+0000 7f0ce9eff640 1 -- start start 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.652+0000 7f0ce9eff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0ce4154e50 0x7f0ce40cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.652+0000 7f0ce9eff640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0ce40b5f10 0x7f0ce40cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.652+0000 7f0ce9eff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce406b110 con 0x7f0ce40caf40 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.652+0000 7f0ce9eff640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce415b8d0 con 0x7f0ce40b5f10 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.652+0000 7f0ce9eff640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce415ad50 con 0x7f0ce4154e50 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.653+0000 7f0ce2ffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0ce40b5f10 0x7f0ce40cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.653+0000 7f0ce2ffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f0ce40b5f10 0x7f0ce40cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45944/0 (socket says 172.21.15.172:45944) 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.653+0000 7f0ce2ffd640 1 -- 172.21.15.172:0/4203162025 learned_addr learned my addr 172.21.15.172:0/4203162025 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.653+0000 7f0ce37fe640 1 --2- 172.21.15.172:0/4203162025 >> v2:172.21.15.174:3300/0 conn(0x7f0ce4154e50 0x7f0ce40cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.653+0000 7f0ce2ffd640 1 -- 172.21.15.172:0/4203162025 >> v2:172.21.15.174:3300/0 conn(0x7f0ce4154e50 msgr2=0x7f0ce40cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:34:53.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.653+0000 7f0ce27fc640 1 -- 172.21.15.172:0/4203162025 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439625097 0 0) 0x7f0ce406b110 con 0x7f0ce40caf40 2024-03-16T19:34:53.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.653+0000 7f0ce2ffd640 1 --2- 172.21.15.172:0/4203162025 >> v2:172.21.15.174:3300/0 conn(0x7f0ce4154e50 0x7f0ce40cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.653+0000 7f0ce2ffd640 1 -- 172.21.15.172:0/4203162025 >> v1:172.21.15.174:6789/0 conn(0x7f0ce40caf40 legacy=0x7f0ce406b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.633 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.653+0000 7f0ce2ffd640 1 -- 172.21.15.172:0/4203162025 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce415b030 con 0x7f0ce40b5f10 2024-03-16T19:34:53.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.654+0000 7f0ce2ffd640 1 --2- 172.21.15.172:0/4203162025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce40b5f10 0x7f0ce40cc170 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0ce408e370 tx=0x7f0cd802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64df48136a6e3e34 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.654+0000 7f0ce27fc640 1 -- 172.21.15.172:0/4203162025 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd802f850 con 0x7f0ce40b5f10 2024-03-16T19:34:53.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.654+0000 7f0ce27fc640 1 -- 172.21.15.172:0/4203162025 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd802f9b0 con 0x7f0ce40b5f10 2024-03-16T19:34:53.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.654+0000 7f0ce27fc640 1 -- 172.21.15.172:0/4203162025 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd802fd40 con 0x7f0ce40b5f10 2024-03-16T19:34:53.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.654+0000 7f0ce9eff640 1 -- 172.21.15.172:0/4203162025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce40b5f10 msgr2=0x7f0ce40cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.654+0000 7f0ce9eff640 1 --2- 172.21.15.172:0/4203162025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce40b5f10 0x7f0ce40cc170 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0ce408e370 tx=0x7f0cd802ba30 comp rx=0 tx=0).stop 2024-03-16T19:34:53.634 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.655+0000 7f0ce9eff640 1 -- 172.21.15.172:0/4203162025 shutdown_connections 2024-03-16T19:34:53.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.655+0000 7f0ce9eff640 1 --2- 172.21.15.172:0/4203162025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce40b5f10 0x7f0ce40cc170 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.655+0000 7f0ce9eff640 1 --2- 172.21.15.172:0/4203162025 >> v2:172.21.15.174:3300/0 conn(0x7f0ce4154e50 0x7f0ce40cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.655+0000 7f0ce9eff640 1 -- 172.21.15.172:0/4203162025 >> 172.21.15.172:0/4203162025 conn(0x7f0ce4066c20 msgr2=0x7f0ce4067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:53.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.655+0000 7f0ce9eff640 1 -- 172.21.15.172:0/4203162025 shutdown_connections 2024-03-16T19:34:53.635 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.655+0000 7f0ce9eff640 1 -- 172.21.15.172:0/4203162025 wait complete. 2024-03-16T19:34:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.656+0000 7f0ce9eff640 1 Processor -- start 2024-03-16T19:34:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.656+0000 7f0ce9eff640 1 -- start start 2024-03-16T19:34:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce9eff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ce40b5f10 0x7f0ce4149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce9eff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ce40caf40 0x7f0ce414dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce9eff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce4154e50 0x7f0ce414e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce9eff640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce414d950 con 0x7f0ce4154e50 2024-03-16T19:34:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce9eff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce41f13e0 con 0x7f0ce40caf40 2024-03-16T19:34:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce9eff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce41f16c0 con 0x7f0ce40b5f10 2024-03-16T19:34:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce37fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ce40b5f10 0x7f0ce4149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce37fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ce40b5f10 0x7f0ce4149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:60244/0 (socket says 172.21.15.172:60244) 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce37fe640 1 -- 172.21.15.172:0/248317864 learned_addr learned my addr 172.21.15.172:0/248317864 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce2ffd640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ce40caf40 0x7f0ce414dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce3fff640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce4154e50 0x7f0ce414e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce37fe640 1 -- 172.21.15.172:0/248317864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ce40caf40 msgr2=0x7f0ce414dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce37fe640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ce40caf40 0x7f0ce414dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce37fe640 1 -- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce4154e50 msgr2=0x7f0ce414e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce37fe640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0ce4154e50 0x7f0ce414e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce37fe640 1 -- 172.21.15.172:0/248317864 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd80094d0 con 0x7f0ce40b5f10 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ce37fe640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ce40b5f10 0x7f0ce4149c00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400e870 tx=0x7f0cd400ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ccbfff640 1 -- 172.21.15.172:0/248317864 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4009510 con 0x7f0ce40b5f10 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.657+0000 7f0ccbfff640 1 -- 172.21.15.172:0/248317864 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd400eec0 con 0x7f0ce40b5f10 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.658+0000 7f0ccbfff640 1 -- 172.21.15.172:0/248317864 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd400cb80 con 0x7f0ce40b5f10 2024-03-16T19:34:53.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.658+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce415ab20 con 0x7f0ce40b5f10 2024-03-16T19:34:53.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.658+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce40c3fe0 con 0x7f0ce40b5f10 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.659+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0ce414eda0 con 0x7f0ce40b5f10 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.659+0000 7f0ccbfff640 1 -- 172.21.15.172:0/248317864 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0cd400cd20 con 0x7f0ce40b5f10 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.659+0000 7f0ccbfff640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0cd006b3b0 0x7f0cd006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.659+0000 7f0ccbfff640 1 -- 172.21.15.172:0/248317864 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f0cd408a130 con 0x7f0ce40b5f10 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.660+0000 7f0ccbfff640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0cd00737f0 0x7f0cd0075cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.660+0000 7f0ccbfff640 1 -- 172.21.15.172:0/248317864 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 1: {"prefix": "get_command_descriptions", "pgid": "3.0"}) v1 -- 0x7f0cd0076380 con 0x7f0cd00737f0 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.660+0000 7f0ccbfff640 1 -- 172.21.15.172:0/248317864 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cd408c7f0 con 0x7f0ce40b5f10 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.660+0000 7f0ce2ffd640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0cd006b3b0 0x7f0cd006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.660+0000 7f0ce3fff640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0cd00737f0 0x7f0cd0075cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.660+0000 7f0ce3fff640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0cd00737f0 0x7f0cd0075cb0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.660+0000 7f0ce2ffd640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0cd006b3b0 0x7f0cd006d870 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8036000 tx=0x7f0cd8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.661+0000 7f0ccbfff640 1 -- 172.21.15.172:0/248317864 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0cd0076380 con 0x7f0cd00737f0 2024-03-16T19:34:53.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.674+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 2: {"prefix": "deep-scrub", "pgid": "3.0"}) v1 -- 0x7f0ce415b5f0 con 0x7f0cd00737f0 2024-03-16T19:34:53.653 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ccbfff640 1 -- 172.21.15.172:0/248317864 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+64 (crc 0 0 0) 0x7f0cdc001ba0 con 0x7f0cd00737f0 2024-03-16T19:34:53.653 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:34:53.653 INFO:teuthology.orchestra.run.smithi172.stdout: "deep": true, 2024-03-16T19:34:53.654 INFO:teuthology.orchestra.run.smithi172.stdout: "must": true, 2024-03-16T19:34:53.654 INFO:teuthology.orchestra.run.smithi172.stdout: "stamp": "0.000000" 2024-03-16T19:34:53.654 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0cd00737f0 msgr2=0x7f0cd0075cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f0cd00737f0 0x7f0cd0075cb0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0cd006b3b0 msgr2=0x7f0cd006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f0cd006b3b0 0x7f0cd006d870 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8036000 tx=0x7f0cd8008030 comp rx=0 tx=0).stop 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ce40b5f10 msgr2=0x7f0ce4149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 --2- 172.21.15.172:0/248317864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ce40b5f10 0x7f0ce4149c00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400e870 tx=0x7f0cd400ec70 comp rx=0 tx=0).stop 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce3fff640 1 -- 172.21.15.172:0/248317864 reap_dead start 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 shutdown_connections 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 >> 172.21.15.172:0/248317864 conn(0x7f0ce4066c20 msgr2=0x7f0ce40c7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 shutdown_connections 2024-03-16T19:34:53.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.675+0000 7f0ce9eff640 1 -- 172.21.15.172:0/248317864 wait complete. 2024-03-16T19:34:53.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 3.4 deep-scrub 2024-03-16T19:34:53.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.816+0000 7fee93a8f640 1 Processor -- start 2024-03-16T19:34:53.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.817+0000 7fee93a8f640 1 -- start start 2024-03-16T19:34:53.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.817+0000 7fee93a8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c154d90 0x7fee8c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.817+0000 7fee93a8f640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fee8c073310 0x7fee8c06dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee93a8f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee8c06eb50 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee93a8f640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c06d0a0 con 0x7fee8c073310 2024-03-16T19:34:53.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee93a8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c06eef0 con 0x7fee8c154d90 2024-03-16T19:34:53.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee9182b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c154d90 0x7fee8c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee9202c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fee8c0b1fd0 0x7fee8c15e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40804/0 (socket says 172.21.15.172:40804) 2024-03-16T19:34:53.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee9202c640 1 -- 172.21.15.172:0/1506768944 learned_addr learned my addr 172.21.15.172:0/1506768944 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:34:53.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee9102a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fee8c073310 0x7fee8c06dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee9182b640 1 -- 172.21.15.172:0/1506768944 >> v2:172.21.15.172:3300/0 conn(0x7fee8c073310 msgr2=0x7fee8c06dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:34:53.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee90829640 1 -- 172.21.15.172:0/1506768944 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258737883 0 0) 0x7fee8c06eb50 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee9182b640 1 --2- 172.21.15.172:0/1506768944 >> v2:172.21.15.172:3300/0 conn(0x7fee8c073310 0x7fee8c06dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee9182b640 1 -- 172.21.15.172:0/1506768944 >> v1:172.21.15.172:6789/0 conn(0x7fee8c0b1fd0 legacy=0x7fee8c15e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.818+0000 7fee9182b640 1 -- 172.21.15.172:0/1506768944 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c06cb20 con 0x7fee8c154d90 2024-03-16T19:34:53.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.819+0000 7fee9182b640 1 --2- 172.21.15.172:0/1506768944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c154d90 0x7fee8c06afb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fee880097a0 tx=0x7fee8802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9863b496123d5d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.819+0000 7fee90829640 1 -- 172.21.15.172:0/1506768944 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8802d040 con 0x7fee8c154d90 2024-03-16T19:34:53.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.819+0000 7fee90829640 1 -- 172.21.15.172:0/1506768944 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee88031810 con 0x7fee8c154d90 2024-03-16T19:34:53.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.819+0000 7fee90829640 1 -- 172.21.15.172:0/1506768944 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88031ba0 con 0x7fee8c154d90 2024-03-16T19:34:53.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.820+0000 7fee93a8f640 1 -- 172.21.15.172:0/1506768944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c154d90 msgr2=0x7fee8c06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.820+0000 7fee93a8f640 1 --2- 172.21.15.172:0/1506768944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c154d90 0x7fee8c06afb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fee880097a0 tx=0x7fee8802b8b0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.820+0000 7fee93a8f640 1 -- 172.21.15.172:0/1506768944 shutdown_connections 2024-03-16T19:34:53.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.820+0000 7fee93a8f640 1 --2- 172.21.15.172:0/1506768944 >> v2:172.21.15.172:3300/0 conn(0x7fee8c073310 0x7fee8c06dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.820+0000 7fee93a8f640 1 --2- 172.21.15.172:0/1506768944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c154d90 0x7fee8c06afb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.820+0000 7fee93a8f640 1 -- 172.21.15.172:0/1506768944 >> 172.21.15.172:0/1506768944 conn(0x7fee8c0cd040 msgr2=0x7fee8c0cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:53.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.820+0000 7fee93a8f640 1 -- 172.21.15.172:0/1506768944 shutdown_connections 2024-03-16T19:34:53.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.821+0000 7fee93a8f640 1 -- 172.21.15.172:0/1506768944 wait complete. 2024-03-16T19:34:53.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.822+0000 7fee93a8f640 1 Processor -- start 2024-03-16T19:34:53.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.822+0000 7fee93a8f640 1 -- start start 2024-03-16T19:34:53.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.822+0000 7fee93a8f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee8c073310 0x7fee8c0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.822+0000 7fee93a8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c0b1fd0 0x7fee8c15d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.822+0000 7fee93a8f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee8c154d90 0x7fee8c146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.822+0000 7fee93a8f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c15cde0 con 0x7fee8c073310 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.822+0000 7fee93a8f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c15c950 con 0x7fee8c154d90 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.822+0000 7fee93a8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c14e640 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9102a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c0b1fd0 0x7fee8c15d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9102a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c0b1fd0 0x7fee8c15d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:60254/0 (socket says 172.21.15.172:60254) 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9102a640 1 -- 172.21.15.172:0/1297385235 learned_addr learned my addr 172.21.15.172:0/1297385235 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9202c640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee8c154d90 0x7fee8c146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9182b640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee8c073310 0x7fee8c0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9102a640 1 -- 172.21.15.172:0/1297385235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee8c154d90 msgr2=0x7fee8c146910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9102a640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee8c154d90 0x7fee8c146910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9102a640 1 -- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee8c073310 msgr2=0x7fee8c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9102a640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee8c073310 0x7fee8c0cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9102a640 1 -- 172.21.15.172:0/1297385235 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee88009400 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9182b640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee8c073310 0x7fee8c0cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:34:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee9102a640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c0b1fd0 0x7fee8c15d450 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00e8a0 tx=0x7fee7c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.823+0000 7fee827fc640 1 -- 172.21.15.172:0/1297385235 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c00ee90 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.824+0000 7fee827fc640 1 -- 172.21.15.172:0/1297385235 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7c00c940 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.824+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c15bd40 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.824+0000 7fee827fc640 1 -- 172.21.15.172:0/1297385235 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c00ccd0 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.824+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c15c300 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.825+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fee8c15b9d0 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.826+0000 7fee827fc640 1 -- 172.21.15.172:0/1297385235 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fee7c013470 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.826+0000 7fee827fc640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee7006b3b0 0x7fee7006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.826+0000 7fee827fc640 1 -- 172.21.15.172:0/1297385235 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fee7c08a620 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.826+0000 7fee827fc640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fee70073810 0x7fee70075cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.826+0000 7fee827fc640 1 -- 172.21.15.172:0/1297385235 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 1: {"prefix": "get_command_descriptions", "pgid": "3.4"}) v1 -- 0x7fee700763a0 con 0x7fee70073810 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.826+0000 7fee827fc640 1 -- 172.21.15.172:0/1297385235 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee7c08a9d0 con 0x7fee8c0b1fd0 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.827+0000 7fee9202c640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fee70073810 0x7fee70075cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.827+0000 7fee9182b640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee7006b3b0 0x7fee7006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.827+0000 7fee9202c640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fee70073810 0x7fee70075cd0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.827+0000 7fee9182b640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee7006b3b0 0x7fee7006d870 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fee88036000 tx=0x7fee8803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.807 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.828+0000 7fee827fc640 1 -- 172.21.15.172:0/1297385235 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fee700763a0 con 0x7fee70073810 2024-03-16T19:34:53.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.854+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 2: {"prefix": "deep-scrub", "pgid": "3.4"}) v1 -- 0x7fee8c15b6f0 con 0x7fee70073810 2024-03-16T19:34:53.833 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.855+0000 7fee827fc640 1 -- 172.21.15.172:0/1297385235 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+64 (crc 0 0 0) 0x7fee84004e60 con 0x7fee70073810 2024-03-16T19:34:53.834 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:34:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "deep": true, 2024-03-16T19:34:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "must": true, 2024-03-16T19:34:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "stamp": "0.000000" 2024-03-16T19:34:53.834 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:34:53.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.855+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fee70073810 msgr2=0x7fee70075cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.855+0000 7fee93a8f640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7fee70073810 0x7fee70075cd0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.855+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee7006b3b0 msgr2=0x7fee7006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.855+0000 7fee93a8f640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fee7006b3b0 0x7fee7006d870 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fee88036000 tx=0x7fee8803c040 comp rx=0 tx=0).stop 2024-03-16T19:34:53.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.855+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c0b1fd0 msgr2=0x7fee8c15d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.835 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.855+0000 7fee93a8f640 1 --2- 172.21.15.172:0/1297385235 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee8c0b1fd0 0x7fee8c15d450 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00e8a0 tx=0x7fee7c00ed60 comp rx=0 tx=0).stop 2024-03-16T19:34:53.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.855+0000 7fee9202c640 1 -- 172.21.15.172:0/1297385235 reap_dead start 2024-03-16T19:34:53.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.856+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 shutdown_connections 2024-03-16T19:34:53.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.856+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 >> 172.21.15.172:0/1297385235 conn(0x7fee8c0cd040 msgr2=0x7fee8c15a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:53.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.856+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 shutdown_connections 2024-03-16T19:34:53.836 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.856+0000 7fee93a8f640 1 -- 172.21.15.172:0/1297385235 wait complete. 2024-03-16T19:34:53.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 3.8 deep-scrub 2024-03-16T19:34:53.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.988+0000 7f4dc8108640 1 Processor -- start 2024-03-16T19:34:53.966 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.988+0000 7f4dc8108640 1 -- start start 2024-03-16T19:34:53.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc8108640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4dc00caf40 0x7f4dc0166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc8108640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc006b110 con 0x7f4dc00b5f10 2024-03-16T19:34:53.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc8108640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc015b8d0 con 0x7f4dc0154e50 2024-03-16T19:34:53.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc8108640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc015ad50 con 0x7f4dc00caf40 2024-03-16T19:34:53.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc56a3640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f4dc00b5f10 0x7f4dc006b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:40820/0 (socket says 172.21.15.172:40820) 2024-03-16T19:34:53.968 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc56a3640 1 -- 172.21.15.172:0/1421866180 learned_addr learned my addr 172.21.15.172:0/1421866180 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:34:53.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc66a5640 1 --2- 172.21.15.172:0/1421866180 >> v2:172.21.15.174:3300/0 conn(0x7f4dc00caf40 0x7f4dc0166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc4ea2640 1 -- 172.21.15.172:0/1421866180 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757524717 0 0) 0x7f4dc006b110 con 0x7f4dc00b5f10 2024-03-16T19:34:53.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc4ea2640 1 -- 172.21.15.172:0/1421866180 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dac003520 con 0x7f4dc00b5f10 2024-03-16T19:34:53.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc4ea2640 1 -- 172.21.15.172:0/1421866180 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574084866 0 0) 0x7f4dc015b8d0 con 0x7f4dc0154e50 2024-03-16T19:34:53.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc4ea2640 1 -- 172.21.15.172:0/1421866180 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc006b110 con 0x7f4dc0154e50 2024-03-16T19:34:53.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc66a5640 1 -- 172.21.15.172:0/1421866180 >> v1:172.21.15.174:6789/0 conn(0x7f4dc0154e50 legacy=0x7f4dc00cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc66a5640 1 -- 172.21.15.172:0/1421866180 >> v1:172.21.15.172:6789/0 conn(0x7f4dc00b5f10 legacy=0x7f4dc006b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.969 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.989+0000 7f4dc66a5640 1 -- 172.21.15.172:0/1421866180 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc015b030 con 0x7f4dc00caf40 2024-03-16T19:34:53.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.990+0000 7f4dc66a5640 1 --2- 172.21.15.172:0/1421866180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4dc00caf40 0x7f4dc0166270 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009d90 tx=0x7f4db802bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fedf0f78d1693b8c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.990+0000 7f4dc4ea2640 1 -- 172.21.15.172:0/1421866180 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db802cd80 con 0x7f4dc00caf40 2024-03-16T19:34:53.970 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.990+0000 7f4dc4ea2640 1 -- 172.21.15.172:0/1421866180 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db8034440 con 0x7f4dc00caf40 2024-03-16T19:34:53.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.990+0000 7f4dc4ea2640 1 -- 172.21.15.172:0/1421866180 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db80347d0 con 0x7f4dc00caf40 2024-03-16T19:34:53.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.991+0000 7f4dc8108640 1 -- 172.21.15.172:0/1421866180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4dc00caf40 msgr2=0x7f4dc0166270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.991+0000 7f4dc8108640 1 --2- 172.21.15.172:0/1421866180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4dc00caf40 0x7f4dc0166270 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009d90 tx=0x7f4db802bfd0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.991+0000 7f4dc8108640 1 -- 172.21.15.172:0/1421866180 shutdown_connections 2024-03-16T19:34:53.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.991+0000 7f4dc8108640 1 --2- 172.21.15.172:0/1421866180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4dc00caf40 0x7f4dc0166270 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.991+0000 7f4dc8108640 1 -- 172.21.15.172:0/1421866180 >> 172.21.15.172:0/1421866180 conn(0x7f4dc0066c20 msgr2=0x7f4dc0067030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:53.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.991+0000 7f4dc8108640 1 -- 172.21.15.172:0/1421866180 shutdown_connections 2024-03-16T19:34:53.971 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.991+0000 7f4dc8108640 1 -- 172.21.15.172:0/1421866180 wait complete. 2024-03-16T19:34:53.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.993+0000 7f4dc8108640 1 Processor -- start 2024-03-16T19:34:53.972 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.993+0000 7f4dc8108640 1 -- start start 2024-03-16T19:34:53.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc8108640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc00b5f10 0x7f4dc01497a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.973 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc8108640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4dc00caf40 0x7f4dc01f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc8108640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4dc0154e50 0x7f4dc01f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc8108640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc014ed20 con 0x7f4dc00b5f10 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc8108640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc014e760 con 0x7f4dc0154e50 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc8108640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc014ea40 con 0x7f4dc00caf40 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc5ea4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc00b5f10 0x7f4dc01497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc5ea4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc00b5f10 0x7f4dc01497a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45988/0 (socket says 172.21.15.172:45988) 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc5ea4640 1 -- 172.21.15.172:0/1076035102 learned_addr learned my addr 172.21.15.172:0/1076035102 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc56a3640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4dc00caf40 0x7f4dc01f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc66a5640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4dc0154e50 0x7f4dc01f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc5ea4640 1 -- 172.21.15.172:0/1076035102 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4dc00caf40 msgr2=0x7f4dc01f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc5ea4640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4dc00caf40 0x7f4dc01f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.974 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc5ea4640 1 -- 172.21.15.172:0/1076035102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4dc0154e50 msgr2=0x7f4dc01f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc5ea4640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4dc0154e50 0x7f4dc01f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc5ea4640 1 -- 172.21.15.172:0/1076035102 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db8009a70 con 0x7f4dc00b5f10 2024-03-16T19:34:53.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.994+0000 7f4dc5ea4640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc00b5f10 0x7f4dc01497a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00c430 tx=0x7f4dbc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.995+0000 7f4db67fc640 1 -- 172.21.15.172:0/1076035102 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0052d0 con 0x7f4dc00b5f10 2024-03-16T19:34:53.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.995+0000 7f4db67fc640 1 -- 172.21.15.172:0/1076035102 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc005450 con 0x7f4dc00b5f10 2024-03-16T19:34:53.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.995+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc015ab20 con 0x7f4dc00b5f10 2024-03-16T19:34:53.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.995+0000 7f4db67fc640 1 -- 172.21.15.172:0/1076035102 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc003830 con 0x7f4dc00b5f10 2024-03-16T19:34:53.975 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.995+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc00c4b10 con 0x7f4dc00b5f10 2024-03-16T19:34:53.976 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.996+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4dc0067000 con 0x7f4dc00b5f10 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.996+0000 7f4db67fc640 1 -- 172.21.15.172:0/1076035102 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4dbc003990 con 0x7f4dc00b5f10 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.997+0000 7f4db67fc640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4da406b3e0 0x7f4da406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.997+0000 7f4db67fc640 1 -- 172.21.15.172:0/1076035102 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f4dbc089210 con 0x7f4dc00b5f10 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.997+0000 7f4dc56a3640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4da406b3e0 0x7f4da406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.997+0000 7f4db67fc640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f4da4073820 0x7f4da4075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.997+0000 7f4db67fc640 1 -- 172.21.15.172:0/1076035102 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 1: {"prefix": "get_command_descriptions", "pgid": "3.8"}) v1 -- 0x7f4da40763b0 con 0x7f4da4073820 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.997+0000 7f4db67fc640 1 -- 172.21.15.172:0/1076035102 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4dbc08b8d0 con 0x7f4dc00b5f10 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.997+0000 7f4dc66a5640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f4da4073820 0x7f4da4075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.997+0000 7f4dc56a3640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4da406b3e0 0x7f4da406d8a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00c38d0 tx=0x7f4db0007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.997+0000 7f4dc66a5640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f4da4073820 0x7f4da4075ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:34:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:53.998+0000 7f4db67fc640 1 -- 172.21.15.172:0/1076035102 <== osd.2 v2:172.21.15.172:6823/3572014952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4da40763b0 con 0x7f4da4073820 2024-03-16T19:34:53.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.018+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 2: {"prefix": "deep-scrub", "pgid": "3.8"}) v1 -- 0x7f4dc014b9d0 con 0x7f4da4073820 2024-03-16T19:34:53.997 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4db67fc640 1 -- 172.21.15.172:0/1076035102 <== osd.2 v2:172.21.15.172:6823/3572014952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+64 (crc 0 0 0) 0x7f4dc014b9d0 con 0x7f4da4073820 2024-03-16T19:34:53.998 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:34:53.998 INFO:teuthology.orchestra.run.smithi172.stdout: "deep": true, 2024-03-16T19:34:53.998 INFO:teuthology.orchestra.run.smithi172.stdout: "must": true, 2024-03-16T19:34:53.998 INFO:teuthology.orchestra.run.smithi172.stdout: "stamp": "0.000000" 2024-03-16T19:34:53.998 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:34:53.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f4da4073820 msgr2=0x7f4da4075ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:53.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4dc8108640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f4da4073820 0x7f4da4075ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:34:53.999 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4da406b3e0 msgr2=0x7f4da406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:54.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4dc8108640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f4da406b3e0 0x7f4da406d8a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00c38d0 tx=0x7f4db0007d30 comp rx=0 tx=0).stop 2024-03-16T19:34:54.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc00b5f10 msgr2=0x7f4dc01497a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:34:54.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4dc8108640 1 --2- 172.21.15.172:0/1076035102 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc00b5f10 0x7f4dc01497a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00c430 tx=0x7f4dbc00c8f0 comp rx=0 tx=0).stop 2024-03-16T19:34:54.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4dc66a5640 1 -- 172.21.15.172:0/1076035102 reap_dead start 2024-03-16T19:34:54.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 shutdown_connections 2024-03-16T19:34:54.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.019+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 >> 172.21.15.172:0/1076035102 conn(0x7f4dc0066c20 msgr2=0x7f4dc00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:34:54.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.020+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 shutdown_connections 2024-03-16T19:34:54.000 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:34:54.020+0000 7f4dc8108640 1 -- 172.21.15.172:0/1076035102 wait complete. 2024-03-16T19:34:54.011 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:35:14.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:35:14.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.158+0000 7fbc52e6f640 1 Processor -- start 2024-03-16T19:35:14.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.158+0000 7fbc52e6f640 1 -- start start 2024-03-16T19:35:14.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc52e6f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbc4c0c7a10 0x7fbc4c071be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:14.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc52e6f640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c06b5e0 con 0x7fbc4c072120 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc52e6f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c154ff0 con 0x7fbc4c154c10 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc52e6f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4c15e010 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc4bfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbc4c0c7a10 0x7fbc4c071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc5140c640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7fbc4c072120 0x7fbc4c15e430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:39878/0 (socket says 172.21.15.172:39878) 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc5140c640 1 -- 172.21.15.172:0/4070440702 learned_addr learned my addr 172.21.15.172:0/4070440702 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc50c0b640 1 --1- 172.21.15.172:0/4070440702 >> v1:172.21.15.174:6789/0 conn(0x7fbc4c154c10 0x7fbc4c06aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:51652/0 (socket says 172.21.15.172:51652) 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc4b7fe640 1 -- 172.21.15.172:0/4070440702 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254871717 0 0) 0x7fbc4c06b5e0 con 0x7fbc4c072120 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc4b7fe640 1 -- 172.21.15.172:0/4070440702 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc3c003520 con 0x7fbc4c072120 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc4b7fe640 1 -- 172.21.15.172:0/4070440702 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384409654 0 0) 0x7fbc40001110 con 0x7fbc4c154c10 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.160+0000 7fbc4b7fe640 1 -- 172.21.15.172:0/4070440702 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4c06b5e0 con 0x7fbc4c154c10 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.161+0000 7fbc4b7fe640 1 -- 172.21.15.172:0/4070440702 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394314927 0 0) 0x7fbc4c06b5e0 con 0x7fbc4c154c10 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.161+0000 7fbc4bfff640 1 -- 172.21.15.172:0/4070440702 >> v1:172.21.15.174:6789/0 conn(0x7fbc4c154c10 legacy=0x7fbc4c06aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.161+0000 7fbc4bfff640 1 -- 172.21.15.172:0/4070440702 >> v1:172.21.15.172:6789/0 conn(0x7fbc4c072120 legacy=0x7fbc4c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:14.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.161+0000 7fbc4bfff640 1 -- 172.21.15.172:0/4070440702 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c06b980 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.162+0000 7fbc4bfff640 1 --2- 172.21.15.172:0/4070440702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7a10 0x7fbc4c071be0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbc380097a0 tx=0x7fbc3802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ada4e09590982b05 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:35:14.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.162+0000 7fbc4b7fe640 1 -- 172.21.15.172:0/4070440702 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc44001970 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.162+0000 7fbc4b7fe640 1 -- 172.21.15.172:0/4070440702 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc38033440 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.162+0000 7fbc4b7fe640 1 -- 172.21.15.172:0/4070440702 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40001800 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.163+0000 7fbc52e6f640 1 -- 172.21.15.172:0/4070440702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7a10 msgr2=0x7fbc4c071be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:14.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.163+0000 7fbc52e6f640 1 --2- 172.21.15.172:0/4070440702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7a10 0x7fbc4c071be0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbc380097a0 tx=0x7fbc3802b7a0 comp rx=0 tx=0).stop 2024-03-16T19:35:14.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.163+0000 7fbc52e6f640 1 -- 172.21.15.172:0/4070440702 shutdown_connections 2024-03-16T19:35:14.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.163+0000 7fbc52e6f640 1 --2- 172.21.15.172:0/4070440702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7a10 0x7fbc4c071be0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:14.143 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.163+0000 7fbc52e6f640 1 -- 172.21.15.172:0/4070440702 >> 172.21.15.172:0/4070440702 conn(0x7fbc4c072b40 msgr2=0x7fbc4c072f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:35:14.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.163+0000 7fbc52e6f640 1 -- 172.21.15.172:0/4070440702 shutdown_connections 2024-03-16T19:35:14.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.163+0000 7fbc52e6f640 1 -- 172.21.15.172:0/4070440702 wait complete. 2024-03-16T19:35:14.144 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.165+0000 7fbc52e6f640 1 Processor -- start 2024-03-16T19:35:14.145 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.165+0000 7fbc52e6f640 1 -- start start 2024-03-16T19:35:14.146 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.166+0000 7fbc52e6f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc4c072120 0x7fbc4c1f1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:14.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc52e6f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7a10 0x7fbc4c150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:14.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc52e6f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c154c10 0x7fbc4c151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:14.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc52e6f640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c147760 con 0x7fbc4c072120 2024-03-16T19:35:14.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc52e6f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c151ee0 con 0x7fbc4c154c10 2024-03-16T19:35:14.148 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc52e6f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c147bd0 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc50c0b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc4c072120 0x7fbc4c1f1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc50c0b640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc4c072120 0x7fbc4c1f1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:35224/0 (socket says 172.21.15.172:35224) 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc50c0b640 1 -- 172.21.15.172:0/1811770249 learned_addr learned my addr 172.21.15.172:0/1811770249 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc4bfff640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7a10 0x7fbc4c150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc5140c640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c154c10 0x7fbc4c151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc4bfff640 1 -- 172.21.15.172:0/1811770249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c154c10 msgr2=0x7fbc4c151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc4bfff640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c154c10 0x7fbc4c151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc4bfff640 1 -- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc4c072120 msgr2=0x7fbc4c1f1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc4bfff640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc4c072120 0x7fbc4c1f1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc4bfff640 1 -- 172.21.15.172:0/1811770249 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc38009400 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc50c0b640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc4c072120 0x7fbc4c1f1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.167+0000 7fbc4bfff640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7a10 0x7fbc4c150030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbc38032000 tx=0x7fbc380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:35:14.149 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.168+0000 7fbc497fa640 1 -- 172.21.15.172:0/1811770249 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3803d880 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.168+0000 7fbc497fa640 1 -- 172.21.15.172:0/1811770249 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3803d9e0 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.168+0000 7fbc497fa640 1 -- 172.21.15.172:0/1811770249 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3803dd70 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.168+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c1478f0 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.150 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.168+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c1f0c00 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.169+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4c06ce30 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.170+0000 7fbc497fa640 1 -- 172.21.15.172:0/1811770249 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbc3803a040 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.170+0000 7fbc497fa640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc3006b3e0 0x7fbc3006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:14.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.170+0000 7fbc497fa640 1 -- 172.21.15.172:0/1811770249 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fbc380ab510 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.170+0000 7fbc50c0b640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc3006b3e0 0x7fbc3006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:14.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.171+0000 7fbc50c0b640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc3006b3e0 0x7fbc3006d8a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbc40001930 tx=0x7fbc40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:35:14.154 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.174+0000 7fbc497fa640 1 -- 172.21.15.172:0/1811770249 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc380762a0 con 0x7fbc4c0c7a10 2024-03-16T19:35:14.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.507+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc4c06c1e0 con 0x7fbc3006b3e0 2024-03-16T19:35:14.495 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.516+0000 7fbc497fa640 1 -- 172.21.15.172:0/1811770249 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252929 (secure 0 0 0) 0x7fbc4000fcf0 con 0x7fbc3006b3e0 2024-03-16T19:35:14.498 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:35:14.500 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:35:14.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc3006b3e0 msgr2=0x7fbc3006d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:14.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc3006b3e0 0x7fbc3006d8a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbc40001930 tx=0x7fbc40009040 comp rx=0 tx=0).stop 2024-03-16T19:35:14.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7a10 msgr2=0x7fbc4c150030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:14.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7a10 0x7fbc4c150030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbc38032000 tx=0x7fbc380046c0 comp rx=0 tx=0).stop 2024-03-16T19:35:14.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 shutdown_connections 2024-03-16T19:35:14.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c154c10 0x7fbc4c151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:14.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbc3006b3e0 0x7fbc3006d8a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:14.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7a10 0x7fbc4c150030 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:14.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 --2- 172.21.15.172:0/1811770249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbc4c072120 0x7fbc4c1f1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:14.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.520+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 >> 172.21.15.172:0/1811770249 conn(0x7fbc4c072b40 msgr2=0x7fbc4c153a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:35:14.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.521+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 shutdown_connections 2024-03-16T19:35:14.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:14.521+0000 7fbc52e6f640 1 -- 172.21.15.172:0/1811770249 wait complete. 2024-03-16T19:35:14.515 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-03-16T19:35:14.267674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002503"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.221338+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:32:36.221338+0000","last_peered":"2024-03-16T19:32:36.221338+0000","last_clean":"2024-03-16T19:32:36.221338+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:36.221338+0000","last_undegraded":"2024-03-16T19:32:36.221338+0000","last_fullsized":"2024-03-16T19:32:36.221338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.681944+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:32:36.681944+0000","last_peered":"2024-03-16T19:32:36.681944+0000","last_clean":"2024-03-16T19:32:36.681944+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:36.681944+0000","last_undegraded":"2024-03-16T19:32:36.681944+0000","last_fullsized":"2024-03-16T19:32:36.681944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":113,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.236924+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:32:28.236924+0000","last_peered":"2024-03-16T19:32:28.236924+0000","last_clean":"2024-03-16T19:32:28.236924+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:28.236924+0000","last_undegraded":"2024-03-16T19:32:28.236924+0000","last_fullsized":"2024-03-16T19:32:28.236924+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.687123+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:32:35.687123+0000","last_peered":"2024-03-16T19:32:35.687123+0000","last_clean":"2024-03-16T19:32:35.687123+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:32:35.687123+0000","last_undegraded":"2024-03-16T19:32:35.687123+0000","last_fullsized":"2024-03-16T19:32:35.687123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193204+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193204+0000","last_peered":"2024-03-16T19:32:42.193204+0000","last_clean":"2024-03-16T19:32:42.193204+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193204+0000","last_undegraded":"2024-03-16T19:32:42.193204+0000","last_fullsized":"2024-03-16T19:32:42.193204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501752+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501752+0000","last_peered":"2024-03-16T19:32:43.501752+0000","last_clean":"2024-03-16T19:32:43.501752+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501752+0000","last_undegraded":"2024-03-16T19:32:43.501752+0000","last_fullsized":"2024-03-16T19:32:43.501752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180916+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180916+0000","last_peered":"2024-03-16T19:32:45.180916+0000","last_clean":"2024-03-16T19:32:45.180916+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180916+0000","last_undegraded":"2024-03-16T19:32:45.180916+0000","last_fullsized":"2024-03-16T19:32:45.180916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142618+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142618+0000","last_peered":"2024-03-16T19:32:31.142618+0000","last_clean":"2024-03-16T19:32:31.142618+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142618+0000","last_undegraded":"2024-03-16T19:32:31.142618+0000","last_fullsized":"2024-03-16T19:32:31.142618+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050037+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050037+0000","last_peered":"2024-03-16T19:32:30.050037+0000","last_clean":"2024-03-16T19:32:30.050037+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050037+0000","last_undegraded":"2024-03-16T19:32:30.050037+0000","last_fullsized":"2024-03-16T19:32:30.050037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906526+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906526+0000","last_peered":"2024-03-16T19:32:36.906526+0000","last_clean":"2024-03-16T19:32:36.906526+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906526+0000","last_undegraded":"2024-03-16T19:32:36.906526+0000","last_fullsized":"2024-03-16T19:32:36.906526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946474+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946474+0000","last_peered":"2024-03-16T19:32:37.946474+0000","last_clean":"2024-03-16T19:32:37.946474+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946474+0000","last_undegraded":"2024-03-16T19:32:37.946474+0000","last_fullsized":"2024-03-16T19:32:37.946474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012365+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012365+0000","last_peered":"2024-03-16T19:32:29.012365+0000","last_clean":"2024-03-16T19:32:29.012365+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012365+0000","last_undegraded":"2024-03-16T19:32:29.012365+0000","last_fullsized":"2024-03-16T19:32:29.012365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085970+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085970+0000","last_peered":"2024-03-16T19:32:34.085970+0000","last_clean":"2024-03-16T19:32:34.085970+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085970+0000","last_undegraded":"2024-03-16T19:32:34.085970+0000","last_fullsized":"2024-03-16T19:32:34.085970+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194818+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194818+0000","last_peered":"2024-03-16T19:32:31.194818+0000","last_clean":"2024-03-16T19:32:31.194818+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194818+0000","last_undegraded":"2024-03-16T19:32:31.194818+0000","last_fullsized":"2024-03-16T19:32:31.194818+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622169+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622169+0000","last_peered":"2024-03-16T19:32:30.622169+0000","last_clean":"2024-03-16T19:32:30.622169+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622169+0000","last_undegraded":"2024-03-16T19:32:30.622169+0000","last_fullsized":"2024-03-16T19:32:30.622169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.528006+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.528006+0000","last_peered":"2024-03-16T19:32:41.528006+0000","last_clean":"2024-03-16T19:32:41.528006+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.528006+0000","last_undegraded":"2024-03-16T19:32:41.528006+0000","last_fullsized":"2024-03-16T19:32:41.528006+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668910+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668910+0000","last_peered":"2024-03-16T19:32:33.668910+0000","last_clean":"2024-03-16T19:32:33.668910+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668910+0000","last_undegraded":"2024-03-16T19:32:33.668910+0000","last_fullsized":"2024-03-16T19:32:33.668910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158111+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158111+0000","last_peered":"2024-03-16T19:32:30.158111+0000","last_clean":"2024-03-16T19:32:30.158111+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158111+0000","last_undegraded":"2024-03-16T19:32:30.158111+0000","last_fullsized":"2024-03-16T19:32:30.158111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198985+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198985+0000","last_peered":"2024-03-16T19:32:43.198985+0000","last_clean":"2024-03-16T19:32:43.198985+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198985+0000","last_undegraded":"2024-03-16T19:32:43.198985+0000","last_fullsized":"2024-03-16T19:32:43.198985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874927+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874927+0000","last_peered":"2024-03-16T19:32:34.874927+0000","last_clean":"2024-03-16T19:32:34.874927+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874927+0000","last_undegraded":"2024-03-16T19:32:34.874927+0000","last_fullsized":"2024-03-16T19:32:34.874927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202050+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202050+0000","last_peered":"2024-03-16T19:32:26.202050+0000","last_clean":"2024-03-16T19:32:26.202050+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202050+0000","last_undegraded":"2024-03-16T19:32:26.202050+0000","last_fullsized":"2024-03-16T19:32:26.202050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113826+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113826+0000","last_peered":"2024-03-16T19:32:31.113826+0000","last_clean":"2024-03-16T19:32:31.113826+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113826+0000","last_undegraded":"2024-03-16T19:32:31.113826+0000","last_fullsized":"2024-03-16T19:32:31.113826+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189280+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189280+0000","last_peered":"2024-03-16T19:32:30.189280+0000","last_clean":"2024-03-16T19:32:30.189280+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189280+0000","last_undegraded":"2024-03-16T19:32:30.189280+0000","last_fullsized":"2024-03-16T19:32:30.189280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633708+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633708+0000","last_peered":"2024-03-16T19:32:38.633708+0000","last_clean":"2024-03-16T19:32:38.633708+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633708+0000","last_undegraded":"2024-03-16T19:32:38.633708+0000","last_fullsized":"2024-03-16T19:32:38.633708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":172,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030046+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030046+0000","last_peered":"2024-03-16T19:32:36.030046+0000","last_clean":"2024-03-16T19:32:36.030046+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030046+0000","last_undegraded":"2024-03-16T19:32:36.030046+0000","last_fullsized":"2024-03-16T19:32:36.030046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881976+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881976+0000","last_peered":"2024-03-16T19:32:35.881976+0000","last_clean":"2024-03-16T19:32:35.881976+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881976+0000","last_undegraded":"2024-03-16T19:32:35.881976+0000","last_fullsized":"2024-03-16T19:32:35.881976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651765+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651765+0000","last_peered":"2024-03-16T19:32:31.651765+0000","last_clean":"2024-03-16T19:32:31.651765+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651765+0000","last_undegraded":"2024-03-16T19:32:31.651765+0000","last_fullsized":"2024-03-16T19:32:31.651765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.051224+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:32:35.051224+0000","last_peered":"2024-03-16T19:32:35.051224+0000","last_clean":"2024-03-16T19:32:35.051224+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:35.051224+0000","last_undegraded":"2024-03-16T19:32:35.051224+0000","last_fullsized":"2024-03-16T19:32:35.051224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209568+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209568+0000","last_peered":"2024-03-16T19:32:37.209568+0000","last_clean":"2024-03-16T19:32:37.209568+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209568+0000","last_undegraded":"2024-03-16T19:32:37.209568+0000","last_fullsized":"2024-03-16T19:32:37.209568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.540714+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:32:40.540714+0000","last_peered":"2024-03-16T19:32:40.540714+0000","last_clean":"2024-03-16T19:32:40.540714+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:40.540714+0000","last_undegraded":"2024-03-16T19:32:40.540714+0000","last_fullsized":"2024-03-16T19:32:40.540714+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216691+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216691+0000","last_peered":"2024-03-16T19:32:27.216691+0000","last_clean":"2024-03-16T19:32:27.216691+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216691+0000","last_undegraded":"2024-03-16T19:32:27.216691+0000","last_fullsized":"2024-03-16T19:32:27.216691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.897672+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:32:40.897672+0000","last_peered":"2024-03-16T19:32:40.897672+0000","last_clean":"2024-03-16T19:32:40.897672+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:40.897672+0000","last_undegraded":"2024-03-16T19:32:40.897672+0000","last_fullsized":"2024-03-16T19:32:40.897672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002778+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002778+0000","last_peered":"2024-03-16T19:32:28.002778+0000","last_clean":"2024-03-16T19:32:28.002778+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002778+0000","last_undegraded":"2024-03-16T19:32:28.002778+0000","last_fullsized":"2024-03-16T19:32:28.002778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207579+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207579+0000","last_peered":"2024-03-16T19:32:28.207579+0000","last_clean":"2024-03-16T19:32:28.207579+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207579+0000","last_undegraded":"2024-03-16T19:32:28.207579+0000","last_fullsized":"2024-03-16T19:32:28.207579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265410+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265410+0000","last_peered":"2024-03-16T19:32:35.265410+0000","last_clean":"2024-03-16T19:32:35.265410+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265410+0000","last_undegraded":"2024-03-16T19:32:35.265410+0000","last_fullsized":"2024-03-16T19:32:35.265410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913659+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913659+0000","last_peered":"2024-03-16T19:32:32.913659+0000","last_clean":"2024-03-16T19:32:32.913659+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913659+0000","last_undegraded":"2024-03-16T19:32:32.913659+0000","last_fullsized":"2024-03-16T19:32:32.913659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105281+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105281+0000","last_peered":"2024-03-16T19:32:32.105281+0000","last_clean":"2024-03-16T19:32:32.105281+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105281+0000","last_undegraded":"2024-03-16T19:32:32.105281+0000","last_fullsized":"2024-03-16T19:32:32.105281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062784+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062784+0000","last_peered":"2024-03-16T19:32:38.062784+0000","last_clean":"2024-03-16T19:32:38.062784+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062784+0000","last_undegraded":"2024-03-16T19:32:38.062784+0000","last_fullsized":"2024-03-16T19:32:38.062784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.686005+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.686005+0000","last_peered":"2024-03-16T19:32:28.686005+0000","last_clean":"2024-03-16T19:32:28.686005+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.686005+0000","last_undegraded":"2024-03-16T19:32:28.686005+0000","last_fullsized":"2024-03-16T19:32:28.686005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691258+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691258+0000","last_peered":"2024-03-16T19:32:32.691258+0000","last_clean":"2024-03-16T19:32:32.691258+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691258+0000","last_undegraded":"2024-03-16T19:32:32.691258+0000","last_fullsized":"2024-03-16T19:32:32.691258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944590+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944590+0000","last_peered":"2024-03-16T19:32:30.944590+0000","last_clean":"2024-03-16T19:32:30.944590+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944590+0000","last_undegraded":"2024-03-16T19:32:30.944590+0000","last_fullsized":"2024-03-16T19:32:30.944590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706805+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706805+0000","last_peered":"2024-03-16T19:32:34.706805+0000","last_clean":"2024-03-16T19:32:34.706805+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706805+0000","last_undegraded":"2024-03-16T19:32:34.706805+0000","last_fullsized":"2024-03-16T19:32:34.706805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646648+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646648+0000","last_peered":"2024-03-16T19:32:29.646648+0000","last_clean":"2024-03-16T19:32:29.646648+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646648+0000","last_undegraded":"2024-03-16T19:32:29.646648+0000","last_fullsized":"2024-03-16T19:32:29.646648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223858+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223858+0000","last_peered":"2024-03-16T19:32:30.223858+0000","last_clean":"2024-03-16T19:32:30.223858+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223858+0000","last_undegraded":"2024-03-16T19:32:30.223858+0000","last_fullsized":"2024-03-16T19:32:30.223858+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538424+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538424+0000","last_peered":"2024-03-16T19:32:26.538424+0000","last_clean":"2024-03-16T19:32:26.538424+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538424+0000","last_undegraded":"2024-03-16T19:32:26.538424+0000","last_fullsized":"2024-03-16T19:32:26.538424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":31,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076164+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076164+0000","last_peered":"2024-03-16T19:32:35.076164+0000","last_clean":"2024-03-16T19:32:35.076164+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076164+0000","last_undegraded":"2024-03-16T19:32:35.076164+0000","last_fullsized":"2024-03-16T19:32:35.076164+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":247,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954039+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954039+0000","last_peered":"2024-03-16T19:32:38.954039+0000","last_clean":"2024-03-16T19:32:38.954039+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954039+0000","last_undegraded":"2024-03-16T19:32:38.954039+0000","last_fullsized":"2024-03-16T19:32:38.954039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179941+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179941+0000","last_peered":"2024-03-16T19:32:41.179941+0000","last_clean":"2024-03-16T19:32:41.179941+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179941+0000","last_undegraded":"2024-03-16T19:32:41.179941+0000","last_fullsized":"2024-03-16T19:32:41.179941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245702+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245702+0000","last_peered":"2024-03-16T19:32:36.245702+0000","last_clean":"2024-03-16T19:32:36.245702+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245702+0000","last_undegraded":"2024-03-16T19:32:36.245702+0000","last_fullsized":"2024-03-16T19:32:36.245702+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197908+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197908+0000","last_peered":"2024-03-16T19:32:27.197908+0000","last_clean":"2024-03-16T19:32:27.197908+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197908+0000","last_undegraded":"2024-03-16T19:32:27.197908+0000","last_fullsized":"2024-03-16T19:32:27.197908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696958+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696958+0000","last_peered":"2024-03-16T19:32:36.696958+0000","last_clean":"2024-03-16T19:32:36.696958+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696958+0000","last_undegraded":"2024-03-16T19:32:36.696958+0000","last_fullsized":"2024-03-16T19:32:36.696958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190754+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190754+0000","last_peered":"2024-03-16T19:32:37.190754+0000","last_clean":"2024-03-16T19:32:37.190754+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190754+0000","last_undegraded":"2024-03-16T19:32:37.190754+0000","last_fullsized":"2024-03-16T19:32:37.190754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938129+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938129+0000","last_peered":"2024-03-16T19:32:29.938129+0000","last_clean":"2024-03-16T19:32:29.938129+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938129+0000","last_undegraded":"2024-03-16T19:32:29.938129+0000","last_fullsized":"2024-03-16T19:32:29.938129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932726+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932726+0000","last_peered":"2024-03-16T19:32:39.932726+0000","last_clean":"2024-03-16T19:32:39.932726+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932726+0000","last_undegraded":"2024-03-16T19:32:39.932726+0000","last_fullsized":"2024-03-16T19:32:39.932726+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":74,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:34:53.530403+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:34:53.530403+0000","last_peered":"2024-03-16T19:34:53.530403+0000","last_clean":"2024-03-16T19:34:53.530403+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:34:53.530403+0000","last_undegraded":"2024-03-16T19:34:53.530403+0000","last_fullsized":"2024-03-16T19:34:53.530403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 153s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176466+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176466+0000","last_peered":"2024-03-16T19:32:38.176466+0000","last_clean":"2024-03-16T19:32:38.176466+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176466+0000","last_undegraded":"2024-03-16T19:32:38.176466+0000","last_fullsized":"2024-03-16T19:32:38.176466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":215,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982088+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982088+0000","last_peered":"2024-03-16T19:32:31.982088+0000","last_clean":"2024-03-16T19:32:31.982088+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982088+0000","last_undegraded":"2024-03-16T19:32:31.982088+0000","last_fullsized":"2024-03-16T19:32:31.982088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149110+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149110+0000","last_peered":"2024-03-16T19:32:39.149110+0000","last_clean":"2024-03-16T19:32:39.149110+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149110+0000","last_undegraded":"2024-03-16T19:32:39.149110+0000","last_fullsized":"2024-03-16T19:32:39.149110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108585+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108585+0000","last_peered":"2024-03-16T19:32:32.108585+0000","last_clean":"2024-03-16T19:32:32.108585+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108585+0000","last_undegraded":"2024-03-16T19:32:32.108585+0000","last_fullsized":"2024-03-16T19:32:32.108585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059668+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059668+0000","last_peered":"2024-03-16T19:32:31.059668+0000","last_clean":"2024-03-16T19:32:31.059668+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059668+0000","last_undegraded":"2024-03-16T19:32:31.059668+0000","last_fullsized":"2024-03-16T19:32:31.059668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568574+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568574+0000","last_peered":"2024-03-16T19:32:27.568574+0000","last_clean":"2024-03-16T19:32:27.568574+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568574+0000","last_undegraded":"2024-03-16T19:32:27.568574+0000","last_fullsized":"2024-03-16T19:32:27.568574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651659+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651659+0000","last_peered":"2024-03-16T19:32:37.651659+0000","last_clean":"2024-03-16T19:32:37.651659+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651659+0000","last_undegraded":"2024-03-16T19:32:37.651659+0000","last_fullsized":"2024-03-16T19:32:37.651659+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":61,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205835+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205835+0000","last_peered":"2024-03-16T19:32:28.205835+0000","last_clean":"2024-03-16T19:32:28.205835+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205835+0000","last_undegraded":"2024-03-16T19:32:28.205835+0000","last_fullsized":"2024-03-16T19:32:28.205835+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":166,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166603+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166603+0000","last_peered":"2024-03-16T19:32:40.166603+0000","last_clean":"2024-03-16T19:32:40.166603+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166603+0000","last_undegraded":"2024-03-16T19:32:40.166603+0000","last_fullsized":"2024-03-16T19:32:40.166603+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":375,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:47.115779+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:32:47.115779+0000","last_peered":"2024-03-16T19:32:47.115779+0000","last_clean":"2024-03-16T19:32:47.115779+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:32:47.115779+0000","last_undegraded":"2024-03-16T19:32:47.115779+0000","last_fullsized":"2024-03-16T19:32:47.115779+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.532575+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:32:42.532575+0000","last_peered":"2024-03-16T19:32:42.532575+0000","last_clean":"2024-03-16T19:32:42.532575+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:42.532575+0000","last_undegraded":"2024-03-16T19:32:42.532575+0000","last_fullsized":"2024-03-16T19:32:42.532575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.219294+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:32:29.219294+0000","last_peered":"2024-03-16T19:32:29.219294+0000","last_clean":"2024-03-16T19:32:29.219294+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:29.219294+0000","last_undegraded":"2024-03-16T19:32:29.219294+0000","last_fullsized":"2024-03-16T19:32:29.219294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.162302+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:32:34.162302+0000","last_peered":"2024-03-16T19:32:34.162302+0000","last_clean":"2024-03-16T19:32:34.162302+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:32:34.162302+0000","last_undegraded":"2024-03-16T19:32:34.162302+0000","last_fullsized":"2024-03-16T19:32:34.162302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240521+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240521+0000","last_peered":"2024-03-16T19:32:34.240521+0000","last_clean":"2024-03-16T19:32:34.240521+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240521+0000","last_undegraded":"2024-03-16T19:32:34.240521+0000","last_fullsized":"2024-03-16T19:32:34.240521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.145030+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.145030+0000","last_peered":"2024-03-16T19:32:33.145030+0000","last_clean":"2024-03-16T19:32:33.145030+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.145030+0000","last_undegraded":"2024-03-16T19:32:33.145030+0000","last_fullsized":"2024-03-16T19:32:33.145030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:34:53.675472+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:34:53.675472+0000","last_peered":"2024-03-16T19:34:53.675472+0000","last_clean":"2024-03-16T19:34:53.675472+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:34:53.675472+0000","last_undegraded":"2024-03-16T19:34:53.675472+0000","last_fullsized":"2024-03-16T19:34:53.675472+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 152s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.692412+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:32:30.692412+0000","last_peered":"2024-03-16T19:32:30.692412+0000","last_clean":"2024-03-16T19:32:30.692412+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:30.692412+0000","last_undegraded":"2024-03-16T19:32:30.692412+0000","last_fullsized":"2024-03-16T19:32:30.692412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.998707+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:32:30.998707+0000","last_peered":"2024-03-16T19:32:30.998707+0000","last_clean":"2024-03-16T19:32:30.998707+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:30.998707+0000","last_undegraded":"2024-03-16T19:32:30.998707+0000","last_fullsized":"2024-03-16T19:32:30.998707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.721534+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:32:37.721534+0000","last_peered":"2024-03-16T19:32:37.721534+0000","last_clean":"2024-03-16T19:32:37.721534+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:37.721534+0000","last_undegraded":"2024-03-16T19:32:37.721534+0000","last_fullsized":"2024-03-16T19:32:37.721534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.589670+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:32:39.589670+0000","last_peered":"2024-03-16T19:32:39.589670+0000","last_clean":"2024-03-16T19:32:39.589670+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:39.589670+0000","last_undegraded":"2024-03-16T19:32:39.589670+0000","last_fullsized":"2024-03-16T19:32:39.589670+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:46.142320+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:32:46.142320+0000","last_peered":"2024-03-16T19:32:46.142320+0000","last_clean":"2024-03-16T19:32:46.142320+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:46.142320+0000","last_undegraded":"2024-03-16T19:32:46.142320+0000","last_fullsized":"2024-03-16T19:32:46.142320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":201,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:34:53.855748+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:34:53.855748+0000","last_peered":"2024-03-16T19:34:53.855748+0000","last_clean":"2024-03-16T19:34:53.855748+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:34:53.855748+0000","last_undegraded":"2024-03-16T19:34:53.855748+0000","last_fullsized":"2024-03-16T19:34:53.855748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 153s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586903+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586903+0000","last_peered":"2024-03-16T19:32:29.586903+0000","last_clean":"2024-03-16T19:32:29.586903+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586903+0000","last_undegraded":"2024-03-16T19:32:29.586903+0000","last_fullsized":"2024-03-16T19:32:29.586903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.125506+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:32:33.125506+0000","last_peered":"2024-03-16T19:32:33.125506+0000","last_clean":"2024-03-16T19:32:33.125506+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:33.125506+0000","last_undegraded":"2024-03-16T19:32:33.125506+0000","last_fullsized":"2024-03-16T19:32:33.125506+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910678+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910678+0000","last_peered":"2024-03-16T19:32:31.910678+0000","last_clean":"2024-03-16T19:32:31.910678+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910678+0000","last_undegraded":"2024-03-16T19:32:31.910678+0000","last_fullsized":"2024-03-16T19:32:31.910678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188030+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188030+0000","last_peered":"2024-03-16T19:32:32.188030+0000","last_clean":"2024-03-16T19:32:32.188030+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188030+0000","last_undegraded":"2024-03-16T19:32:32.188030+0000","last_fullsized":"2024-03-16T19:32:32.188030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190121+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190121+0000","last_peered":"2024-03-16T19:32:44.190121+0000","last_clean":"2024-03-16T19:32:44.190121+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190121+0000","last_undegraded":"2024-03-16T19:32:44.190121+0000","last_fullsized":"2024-03-16T19:32:44.190121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026167+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026167+0000","last_peered":"2024-03-16T19:32:30.026167+0000","last_clean":"2024-03-16T19:32:30.026167+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026167+0000","last_undegraded":"2024-03-16T19:32:30.026167+0000","last_fullsized":"2024-03-16T19:32:30.026167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604791+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604791+0000","last_peered":"2024-03-16T19:32:28.604791+0000","last_clean":"2024-03-16T19:32:28.604791+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604791+0000","last_undegraded":"2024-03-16T19:32:28.604791+0000","last_fullsized":"2024-03-16T19:32:28.604791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:34:54.019679+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:34:54.019679+0000","last_peered":"2024-03-16T19:34:54.019679+0000","last_clean":"2024-03-16T19:34:54.019679+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:34:54.019679+0000","last_undegraded":"2024-03-16T19:34:54.019679+0000","last_fullsized":"2024-03-16T19:34:54.019679+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 153s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699900+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699900+0000","last_peered":"2024-03-16T19:32:33.699900+0000","last_clean":"2024-03-16T19:32:33.699900+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699900+0000","last_undegraded":"2024-03-16T19:32:33.699900+0000","last_fullsized":"2024-03-16T19:32:33.699900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030458+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030458+0000","last_peered":"2024-03-16T19:32:39.030458+0000","last_clean":"2024-03-16T19:32:39.030458+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030458+0000","last_undegraded":"2024-03-16T19:32:39.030458+0000","last_fullsized":"2024-03-16T19:32:39.030458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152425+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152425+0000","last_peered":"2024-03-16T19:32:33.152425+0000","last_clean":"2024-03-16T19:32:33.152425+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152425+0000","last_undegraded":"2024-03-16T19:32:33.152425+0000","last_fullsized":"2024-03-16T19:32:33.152425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889937+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889937+0000","last_peered":"2024-03-16T19:32:33.889937+0000","last_clean":"2024-03-16T19:32:33.889937+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889937+0000","last_undegraded":"2024-03-16T19:32:33.889937+0000","last_fullsized":"2024-03-16T19:32:33.889937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695513+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695513+0000","last_peered":"2024-03-16T19:32:31.695513+0000","last_clean":"2024-03-16T19:32:31.695513+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695513+0000","last_undegraded":"2024-03-16T19:32:31.695513+0000","last_fullsized":"2024-03-16T19:32:31.695513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102649+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102649+0000","last_peered":"2024-03-16T19:32:32.102649+0000","last_clean":"2024-03-16T19:32:32.102649+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102649+0000","last_undegraded":"2024-03-16T19:32:32.102649+0000","last_fullsized":"2024-03-16T19:32:32.102649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696352+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696352+0000","last_peered":"2024-03-16T19:32:32.696352+0000","last_clean":"2024-03-16T19:32:32.696352+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696352+0000","last_undegraded":"2024-03-16T19:32:32.696352+0000","last_fullsized":"2024-03-16T19:32:32.696352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177567+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177567+0000","last_peered":"2024-03-16T19:32:35.177567+0000","last_clean":"2024-03-16T19:32:35.177567+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177567+0000","last_undegraded":"2024-03-16T19:32:35.177567+0000","last_fullsized":"2024-03-16T19:32:35.177567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230775+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230775+0000","last_peered":"2024-03-16T19:32:33.230775+0000","last_clean":"2024-03-16T19:32:33.230775+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230775+0000","last_undegraded":"2024-03-16T19:32:33.230775+0000","last_fullsized":"2024-03-16T19:32:33.230775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.046537+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:32:40.046537+0000","last_peered":"2024-03-16T19:32:40.046537+0000","last_clean":"2024-03-16T19:32:40.046537+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:32:40.046537+0000","last_undegraded":"2024-03-16T19:32:40.046537+0000","last_fullsized":"2024-03-16T19:32:40.046537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":175,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.063436+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:32:37.063436+0000","last_peered":"2024-03-16T19:32:37.063436+0000","last_clean":"2024-03-16T19:32:37.063436+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:32:37.063436+0000","last_undegraded":"2024-03-16T19:32:37.063436+0000","last_fullsized":"2024-03-16T19:32:37.063436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272673+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272673+0000","last_peered":"2024-03-16T19:32:29.272673+0000","last_clean":"2024-03-16T19:32:29.272673+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272673+0000","last_undegraded":"2024-03-16T19:32:29.272673+0000","last_fullsized":"2024-03-16T19:32:29.272673+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574923,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574924,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574925,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122091,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574924,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:35:14.520 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:14.520 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:14.520 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:14.521 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:14.521 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:35:22.464 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:22.486+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:22.685 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:22.706+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:23.488 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:23.509+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:23.700 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:23.722+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:24.457 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:24.479+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:24.718 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:24.740+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:25.453 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:25.474+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:25.689 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:25.711+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:26.442 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:26.464+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:26.666 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:26.687+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:27.459 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:27.481+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:27.643 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:27.664+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:28.496 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:28.517+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:28.612 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:28.633+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:29.528 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:29.549+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:29.651 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:29.673+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:30.527 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:30.548+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:30.701 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:30.722+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:31.541 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:31.562+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:31.660 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:31.681+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:32.575 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:32.596+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:32.637 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:32.659+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:33.600 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:33.622+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:33.669 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:33.691+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:34.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:35:34.597 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:34.619+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:34.645 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:34.666+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:35.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da91d0640 1 Processor -- start 2024-03-16T19:35:35.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da91d0640 1 -- start start 2024-03-16T19:35:35.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da91d0640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8da40732d0 0x7f8da41648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:35.417 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da91d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40c9650 0x7f8da4164e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da91d0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da406b630 con 0x7f8da416aeb0 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da91d0640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da40c25b0 con 0x7f8da40732d0 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da91d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da40c1b70 con 0x7f8da40c9650 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da259c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8da40732d0 0x7f8da41648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da259c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8da40732d0 0x7f8da41648e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42292/0 (socket says 172.21.15.172:42292) 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.424+0000 7f8da259c640 1 -- 172.21.15.172:0/725065595 learned_addr learned my addr 172.21.15.172:0/725065595 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.425+0000 7f8da1d9b640 1 -- 172.21.15.172:0/725065595 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141655200 0 0) 0x7f8da406b630 con 0x7f8da416aeb0 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.425+0000 7f8da359e640 1 --2- 172.21.15.172:0/725065595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40c9650 0x7f8da4164e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.438+0000 7f8da1d9b640 1 -- 172.21.15.172:0/725065595 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d8c003500 con 0x7f8da416aeb0 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.438+0000 7f8da259c640 1 -- 172.21.15.172:0/725065595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40c9650 msgr2=0x7f8da4164e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.438+0000 7f8da259c640 1 --2- 172.21.15.172:0/725065595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40c9650 0x7f8da4164e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.438+0000 7f8da259c640 1 -- 172.21.15.172:0/725065595 >> v1:172.21.15.174:6789/0 conn(0x7f8da416aeb0 legacy=0x7f8da416b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:35.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.438+0000 7f8da259c640 1 -- 172.21.15.172:0/725065595 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da40c1e50 con 0x7f8da40732d0 2024-03-16T19:35:35.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.438+0000 7f8da1d9b640 1 -- 172.21.15.172:0/725065595 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1916048471 0 0) 0x7f8d8c003500 con 0x7f8da416aeb0 2024-03-16T19:35:35.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.438+0000 7f8da359e640 1 --2- 172.21.15.172:0/725065595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40c9650 0x7f8da4164e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.439+0000 7f8da259c640 1 --2- 172.21.15.172:0/725065595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40732d0 0x7f8da41648e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800b970 tx=0x7f8d9802c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f09bed9757ad8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.439+0000 7f8da1d9b640 1 -- 172.21.15.172:0/725065595 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94001670 con 0x7f8da40732d0 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.439+0000 7f8da1d9b640 1 -- 172.21.15.172:0/725065595 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d98030070 con 0x7f8da40732d0 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.439+0000 7f8da1d9b640 1 -- 172.21.15.172:0/725065595 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9802cd50 con 0x7f8da40732d0 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.440+0000 7f8da91d0640 1 -- 172.21.15.172:0/725065595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40732d0 msgr2=0x7f8da41648e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.440+0000 7f8da91d0640 1 --2- 172.21.15.172:0/725065595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40732d0 0x7f8da41648e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800b970 tx=0x7f8d9802c060 comp rx=0 tx=0).stop 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.440+0000 7f8da91d0640 1 -- 172.21.15.172:0/725065595 shutdown_connections 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.440+0000 7f8da91d0640 1 --2- 172.21.15.172:0/725065595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40c9650 0x7f8da4164e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.440+0000 7f8da91d0640 1 --2- 172.21.15.172:0/725065595 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40732d0 0x7f8da41648e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.440+0000 7f8da91d0640 1 -- 172.21.15.172:0/725065595 >> 172.21.15.172:0/725065595 conn(0x7f8da40c8b30 msgr2=0x7f8da40c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:35:35.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.440+0000 7f8da91d0640 1 -- 172.21.15.172:0/725065595 shutdown_connections 2024-03-16T19:35:35.421 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.440+0000 7f8da91d0640 1 -- 172.21.15.172:0/725065595 wait complete. 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.442+0000 7f8da91d0640 1 Processor -- start 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.442+0000 7f8da91d0640 1 -- start start 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da91d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40732d0 0x7f8da41fd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da91d0640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40c9650 0x7f8da40c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da91d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da416aeb0 0x7f8da415bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da91d0640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da415f210 con 0x7f8da40c9650 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da91d0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da4157330 con 0x7f8da416aeb0 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da91d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da415e2b0 con 0x7f8da40732d0 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da2d9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40732d0 0x7f8da41fd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da259c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40c9650 0x7f8da40c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:35.423 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da2d9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40732d0 0x7f8da41fd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:55150/0 (socket says 172.21.15.172:55150) 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da259c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40c9650 0x7f8da40c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42308/0 (socket says 172.21.15.172:42308) 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da2d9d640 1 -- 172.21.15.172:0/2990282986 learned_addr learned my addr 172.21.15.172:0/2990282986 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da359e640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da416aeb0 0x7f8da415bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da2d9d640 1 -- 172.21.15.172:0/2990282986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da416aeb0 msgr2=0x7f8da415bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da2d9d640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da416aeb0 0x7f8da415bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da2d9d640 1 -- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40c9650 msgr2=0x7f8da40c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da2d9d640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40c9650 0x7f8da40c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da2d9d640 1 -- 172.21.15.172:0/2990282986 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c007730 con 0x7f8da40732d0 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da259c640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40c9650 0x7f8da40c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8da2d9d640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40732d0 0x7f8da41fd180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400c530 tx=0x7f8d9400c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.443+0000 7f8d8b7fe640 1 -- 172.21.15.172:0/2990282986 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94012830 con 0x7f8da40732d0 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.444+0000 7f8d8b7fe640 1 -- 172.21.15.172:0/2990282986 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d940129b0 con 0x7f8da40732d0 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.444+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9800b5d0 con 0x7f8da40732d0 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.444+0000 7f8d8b7fe640 1 -- 172.21.15.172:0/2990282986 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94012da0 con 0x7f8da40732d0 2024-03-16T19:35:35.424 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.444+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da415e870 con 0x7f8da40732d0 2024-03-16T19:35:35.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.445+0000 7f8d8b7fe640 1 -- 172.21.15.172:0/2990282986 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8d9401c460 con 0x7f8da40732d0 2024-03-16T19:35:35.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.445+0000 7f8d8b7fe640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8d9006b3b0 0x7f8d9006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:35.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.445+0000 7f8d8b7fe640 1 -- 172.21.15.172:0/2990282986 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f8d94089bc0 con 0x7f8da40732d0 2024-03-16T19:35:35.425 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.445+0000 7f8da259c640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8d9006b3b0 0x7f8d9006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:35.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.446+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da40681c0 con 0x7f8da40732d0 2024-03-16T19:35:35.426 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.446+0000 7f8da259c640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8d9006b3b0 0x7f8d9006d870 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8d98037000 tx=0x7f8d98032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:35:35.429 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.451+0000 7f8d8b7fe640 1 -- 172.21.15.172:0/2990282986 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d94054700 con 0x7f8da40732d0 2024-03-16T19:35:35.611 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:35.633+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:35.612 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:35.633+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:35.724 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.746+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8da40c2130 con 0x7f8d9006b3b0 2024-03-16T19:35:35.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.755+0000 7f8d8b7fe640 1 -- 172.21.15.172:0/2990282986 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252929 (secure 0 0 0) 0x7f8d98036480 con 0x7f8d9006b3b0 2024-03-16T19:35:35.736 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:35:35.737 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:35:35.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8d9006b3b0 msgr2=0x7f8d9006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:35.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8d9006b3b0 0x7f8d9006d870 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8d98037000 tx=0x7f8d98032000 comp rx=0 tx=0).stop 2024-03-16T19:35:35.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40732d0 msgr2=0x7f8da41fd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:35.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40732d0 0x7f8da41fd180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400c530 tx=0x7f8d9400c9f0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 shutdown_connections 2024-03-16T19:35:35.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da416aeb0 0x7f8da415bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.739 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8da40c9650 0x7f8da40c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f8d9006b3b0 0x7f8d9006d870 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 --2- 172.21.15.172:0/2990282986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8da40732d0 0x7f8da41fd180 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:35.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.759+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 >> 172.21.15.172:0/2990282986 conn(0x7f8da40c8b30 msgr2=0x7f8da41639d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:35:35.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.760+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 shutdown_connections 2024-03-16T19:35:35.740 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:35.760+0000 7f8da91d0640 1 -- 172.21.15.172:0/2990282986 wait complete. 2024-03-16T19:35:35.775 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-16T19:35:34.271905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002513"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.221338+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:32:36.221338+0000","last_peered":"2024-03-16T19:32:36.221338+0000","last_clean":"2024-03-16T19:32:36.221338+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:32:36.221338+0000","last_undegraded":"2024-03-16T19:32:36.221338+0000","last_fullsized":"2024-03-16T19:32:36.221338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.681944+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:32:36.681944+0000","last_peered":"2024-03-16T19:32:36.681944+0000","last_clean":"2024-03-16T19:32:36.681944+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:32:36.681944+0000","last_undegraded":"2024-03-16T19:32:36.681944+0000","last_fullsized":"2024-03-16T19:32:36.681944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":113,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.236924+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:32:28.236924+0000","last_peered":"2024-03-16T19:32:28.236924+0000","last_clean":"2024-03-16T19:32:28.236924+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:32:28.236924+0000","last_undegraded":"2024-03-16T19:32:28.236924+0000","last_fullsized":"2024-03-16T19:32:28.236924+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230354+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:33:28.230354+0000","last_peered":"2024-03-16T19:33:28.230354+0000","last_clean":"2024-03-16T19:33:28.230354+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:33:28.230354+0000","last_undegraded":"2024-03-16T19:33:28.230354+0000","last_fullsized":"2024-03-16T19:33:28.230354+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193204+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193204+0000","last_peered":"2024-03-16T19:32:42.193204+0000","last_clean":"2024-03-16T19:32:42.193204+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193204+0000","last_undegraded":"2024-03-16T19:32:42.193204+0000","last_fullsized":"2024-03-16T19:32:42.193204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501752+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501752+0000","last_peered":"2024-03-16T19:32:43.501752+0000","last_clean":"2024-03-16T19:32:43.501752+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501752+0000","last_undegraded":"2024-03-16T19:32:43.501752+0000","last_fullsized":"2024-03-16T19:32:43.501752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180916+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180916+0000","last_peered":"2024-03-16T19:32:45.180916+0000","last_clean":"2024-03-16T19:32:45.180916+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180916+0000","last_undegraded":"2024-03-16T19:32:45.180916+0000","last_fullsized":"2024-03-16T19:32:45.180916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142618+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142618+0000","last_peered":"2024-03-16T19:32:31.142618+0000","last_clean":"2024-03-16T19:32:31.142618+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142618+0000","last_undegraded":"2024-03-16T19:32:31.142618+0000","last_fullsized":"2024-03-16T19:32:31.142618+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050037+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050037+0000","last_peered":"2024-03-16T19:32:30.050037+0000","last_clean":"2024-03-16T19:32:30.050037+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050037+0000","last_undegraded":"2024-03-16T19:32:30.050037+0000","last_fullsized":"2024-03-16T19:32:30.050037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906526+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906526+0000","last_peered":"2024-03-16T19:32:36.906526+0000","last_clean":"2024-03-16T19:32:36.906526+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906526+0000","last_undegraded":"2024-03-16T19:32:36.906526+0000","last_fullsized":"2024-03-16T19:32:36.906526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946474+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946474+0000","last_peered":"2024-03-16T19:32:37.946474+0000","last_clean":"2024-03-16T19:32:37.946474+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946474+0000","last_undegraded":"2024-03-16T19:32:37.946474+0000","last_fullsized":"2024-03-16T19:32:37.946474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012365+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012365+0000","last_peered":"2024-03-16T19:32:29.012365+0000","last_clean":"2024-03-16T19:32:29.012365+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012365+0000","last_undegraded":"2024-03-16T19:32:29.012365+0000","last_fullsized":"2024-03-16T19:32:29.012365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085970+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085970+0000","last_peered":"2024-03-16T19:32:34.085970+0000","last_clean":"2024-03-16T19:32:34.085970+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085970+0000","last_undegraded":"2024-03-16T19:32:34.085970+0000","last_fullsized":"2024-03-16T19:32:34.085970+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194818+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194818+0000","last_peered":"2024-03-16T19:32:31.194818+0000","last_clean":"2024-03-16T19:32:31.194818+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194818+0000","last_undegraded":"2024-03-16T19:32:31.194818+0000","last_fullsized":"2024-03-16T19:32:31.194818+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622169+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622169+0000","last_peered":"2024-03-16T19:32:30.622169+0000","last_clean":"2024-03-16T19:32:30.622169+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622169+0000","last_undegraded":"2024-03-16T19:32:30.622169+0000","last_fullsized":"2024-03-16T19:32:30.622169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.528006+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.528006+0000","last_peered":"2024-03-16T19:32:41.528006+0000","last_clean":"2024-03-16T19:32:41.528006+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.528006+0000","last_undegraded":"2024-03-16T19:32:41.528006+0000","last_fullsized":"2024-03-16T19:32:41.528006+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668910+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668910+0000","last_peered":"2024-03-16T19:32:33.668910+0000","last_clean":"2024-03-16T19:32:33.668910+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668910+0000","last_undegraded":"2024-03-16T19:32:33.668910+0000","last_fullsized":"2024-03-16T19:32:33.668910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158111+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158111+0000","last_peered":"2024-03-16T19:32:30.158111+0000","last_clean":"2024-03-16T19:32:30.158111+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158111+0000","last_undegraded":"2024-03-16T19:32:30.158111+0000","last_fullsized":"2024-03-16T19:32:30.158111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198985+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198985+0000","last_peered":"2024-03-16T19:32:43.198985+0000","last_clean":"2024-03-16T19:32:43.198985+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198985+0000","last_undegraded":"2024-03-16T19:32:43.198985+0000","last_fullsized":"2024-03-16T19:32:43.198985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874927+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874927+0000","last_peered":"2024-03-16T19:32:34.874927+0000","last_clean":"2024-03-16T19:32:34.874927+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874927+0000","last_undegraded":"2024-03-16T19:32:34.874927+0000","last_fullsized":"2024-03-16T19:32:34.874927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202050+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202050+0000","last_peered":"2024-03-16T19:32:26.202050+0000","last_clean":"2024-03-16T19:32:26.202050+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202050+0000","last_undegraded":"2024-03-16T19:32:26.202050+0000","last_fullsized":"2024-03-16T19:32:26.202050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113826+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113826+0000","last_peered":"2024-03-16T19:32:31.113826+0000","last_clean":"2024-03-16T19:32:31.113826+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113826+0000","last_undegraded":"2024-03-16T19:32:31.113826+0000","last_fullsized":"2024-03-16T19:32:31.113826+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189280+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189280+0000","last_peered":"2024-03-16T19:32:30.189280+0000","last_clean":"2024-03-16T19:32:30.189280+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189280+0000","last_undegraded":"2024-03-16T19:32:30.189280+0000","last_fullsized":"2024-03-16T19:32:30.189280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633708+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633708+0000","last_peered":"2024-03-16T19:32:38.633708+0000","last_clean":"2024-03-16T19:32:38.633708+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633708+0000","last_undegraded":"2024-03-16T19:32:38.633708+0000","last_fullsized":"2024-03-16T19:32:38.633708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":172,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030046+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030046+0000","last_peered":"2024-03-16T19:32:36.030046+0000","last_clean":"2024-03-16T19:32:36.030046+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030046+0000","last_undegraded":"2024-03-16T19:32:36.030046+0000","last_fullsized":"2024-03-16T19:32:36.030046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881976+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881976+0000","last_peered":"2024-03-16T19:32:35.881976+0000","last_clean":"2024-03-16T19:32:35.881976+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881976+0000","last_undegraded":"2024-03-16T19:32:35.881976+0000","last_fullsized":"2024-03-16T19:32:35.881976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651765+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651765+0000","last_peered":"2024-03-16T19:32:31.651765+0000","last_clean":"2024-03-16T19:32:31.651765+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651765+0000","last_undegraded":"2024-03-16T19:32:31.651765+0000","last_fullsized":"2024-03-16T19:32:31.651765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.051224+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:32:35.051224+0000","last_peered":"2024-03-16T19:32:35.051224+0000","last_clean":"2024-03-16T19:32:35.051224+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:32:35.051224+0000","last_undegraded":"2024-03-16T19:32:35.051224+0000","last_fullsized":"2024-03-16T19:32:35.051224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209568+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209568+0000","last_peered":"2024-03-16T19:32:37.209568+0000","last_clean":"2024-03-16T19:32:37.209568+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209568+0000","last_undegraded":"2024-03-16T19:32:37.209568+0000","last_fullsized":"2024-03-16T19:32:37.209568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.540714+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:32:40.540714+0000","last_peered":"2024-03-16T19:32:40.540714+0000","last_clean":"2024-03-16T19:32:40.540714+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:32:40.540714+0000","last_undegraded":"2024-03-16T19:32:40.540714+0000","last_fullsized":"2024-03-16T19:32:40.540714+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216691+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216691+0000","last_peered":"2024-03-16T19:32:27.216691+0000","last_clean":"2024-03-16T19:32:27.216691+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216691+0000","last_undegraded":"2024-03-16T19:32:27.216691+0000","last_fullsized":"2024-03-16T19:32:27.216691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.897672+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:32:40.897672+0000","last_peered":"2024-03-16T19:32:40.897672+0000","last_clean":"2024-03-16T19:32:40.897672+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:32:40.897672+0000","last_undegraded":"2024-03-16T19:32:40.897672+0000","last_fullsized":"2024-03-16T19:32:40.897672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002778+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002778+0000","last_peered":"2024-03-16T19:32:28.002778+0000","last_clean":"2024-03-16T19:32:28.002778+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002778+0000","last_undegraded":"2024-03-16T19:32:28.002778+0000","last_fullsized":"2024-03-16T19:32:28.002778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207579+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207579+0000","last_peered":"2024-03-16T19:32:28.207579+0000","last_clean":"2024-03-16T19:32:28.207579+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207579+0000","last_undegraded":"2024-03-16T19:32:28.207579+0000","last_fullsized":"2024-03-16T19:32:28.207579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265410+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265410+0000","last_peered":"2024-03-16T19:32:35.265410+0000","last_clean":"2024-03-16T19:32:35.265410+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265410+0000","last_undegraded":"2024-03-16T19:32:35.265410+0000","last_fullsized":"2024-03-16T19:32:35.265410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913659+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913659+0000","last_peered":"2024-03-16T19:32:32.913659+0000","last_clean":"2024-03-16T19:32:32.913659+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913659+0000","last_undegraded":"2024-03-16T19:32:32.913659+0000","last_fullsized":"2024-03-16T19:32:32.913659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105281+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105281+0000","last_peered":"2024-03-16T19:32:32.105281+0000","last_clean":"2024-03-16T19:32:32.105281+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105281+0000","last_undegraded":"2024-03-16T19:32:32.105281+0000","last_fullsized":"2024-03-16T19:32:32.105281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062784+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062784+0000","last_peered":"2024-03-16T19:32:38.062784+0000","last_clean":"2024-03-16T19:32:38.062784+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062784+0000","last_undegraded":"2024-03-16T19:32:38.062784+0000","last_fullsized":"2024-03-16T19:32:38.062784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.686005+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.686005+0000","last_peered":"2024-03-16T19:32:28.686005+0000","last_clean":"2024-03-16T19:32:28.686005+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.686005+0000","last_undegraded":"2024-03-16T19:32:28.686005+0000","last_fullsized":"2024-03-16T19:32:28.686005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691258+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691258+0000","last_peered":"2024-03-16T19:32:32.691258+0000","last_clean":"2024-03-16T19:32:32.691258+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691258+0000","last_undegraded":"2024-03-16T19:32:32.691258+0000","last_fullsized":"2024-03-16T19:32:32.691258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944590+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944590+0000","last_peered":"2024-03-16T19:32:30.944590+0000","last_clean":"2024-03-16T19:32:30.944590+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944590+0000","last_undegraded":"2024-03-16T19:32:30.944590+0000","last_fullsized":"2024-03-16T19:32:30.944590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706805+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706805+0000","last_peered":"2024-03-16T19:32:34.706805+0000","last_clean":"2024-03-16T19:32:34.706805+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706805+0000","last_undegraded":"2024-03-16T19:32:34.706805+0000","last_fullsized":"2024-03-16T19:32:34.706805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646648+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646648+0000","last_peered":"2024-03-16T19:32:29.646648+0000","last_clean":"2024-03-16T19:32:29.646648+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646648+0000","last_undegraded":"2024-03-16T19:32:29.646648+0000","last_fullsized":"2024-03-16T19:32:29.646648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223858+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223858+0000","last_peered":"2024-03-16T19:32:30.223858+0000","last_clean":"2024-03-16T19:32:30.223858+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223858+0000","last_undegraded":"2024-03-16T19:32:30.223858+0000","last_fullsized":"2024-03-16T19:32:30.223858+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538424+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538424+0000","last_peered":"2024-03-16T19:32:26.538424+0000","last_clean":"2024-03-16T19:32:26.538424+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538424+0000","last_undegraded":"2024-03-16T19:32:26.538424+0000","last_fullsized":"2024-03-16T19:32:26.538424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":31,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076164+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076164+0000","last_peered":"2024-03-16T19:32:35.076164+0000","last_clean":"2024-03-16T19:32:35.076164+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076164+0000","last_undegraded":"2024-03-16T19:32:35.076164+0000","last_fullsized":"2024-03-16T19:32:35.076164+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":247,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954039+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954039+0000","last_peered":"2024-03-16T19:32:38.954039+0000","last_clean":"2024-03-16T19:32:38.954039+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954039+0000","last_undegraded":"2024-03-16T19:32:38.954039+0000","last_fullsized":"2024-03-16T19:32:38.954039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179941+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179941+0000","last_peered":"2024-03-16T19:32:41.179941+0000","last_clean":"2024-03-16T19:32:41.179941+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179941+0000","last_undegraded":"2024-03-16T19:32:41.179941+0000","last_fullsized":"2024-03-16T19:32:41.179941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245702+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245702+0000","last_peered":"2024-03-16T19:32:36.245702+0000","last_clean":"2024-03-16T19:32:36.245702+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245702+0000","last_undegraded":"2024-03-16T19:32:36.245702+0000","last_fullsized":"2024-03-16T19:32:36.245702+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197908+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197908+0000","last_peered":"2024-03-16T19:32:27.197908+0000","last_clean":"2024-03-16T19:32:27.197908+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197908+0000","last_undegraded":"2024-03-16T19:32:27.197908+0000","last_fullsized":"2024-03-16T19:32:27.197908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696958+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696958+0000","last_peered":"2024-03-16T19:32:36.696958+0000","last_clean":"2024-03-16T19:32:36.696958+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696958+0000","last_undegraded":"2024-03-16T19:32:36.696958+0000","last_fullsized":"2024-03-16T19:32:36.696958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190754+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190754+0000","last_peered":"2024-03-16T19:32:37.190754+0000","last_clean":"2024-03-16T19:32:37.190754+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190754+0000","last_undegraded":"2024-03-16T19:32:37.190754+0000","last_fullsized":"2024-03-16T19:32:37.190754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938129+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938129+0000","last_peered":"2024-03-16T19:32:29.938129+0000","last_clean":"2024-03-16T19:32:29.938129+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938129+0000","last_undegraded":"2024-03-16T19:32:29.938129+0000","last_fullsized":"2024-03-16T19:32:29.938129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932726+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932726+0000","last_peered":"2024-03-16T19:32:39.932726+0000","last_clean":"2024-03-16T19:32:39.932726+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932726+0000","last_undegraded":"2024-03-16T19:32:39.932726+0000","last_fullsized":"2024-03-16T19:32:39.932726+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":76,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:35:17.996945+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:35:17.996945+0000","last_peered":"2024-03-16T19:35:17.996945+0000","last_clean":"2024-03-16T19:35:17.996945+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:35:17.996945+0000","last_undegraded":"2024-03-16T19:35:17.996945+0000","last_fullsized":"2024-03-16T19:35:17.996945+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 183s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176466+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176466+0000","last_peered":"2024-03-16T19:32:38.176466+0000","last_clean":"2024-03-16T19:32:38.176466+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176466+0000","last_undegraded":"2024-03-16T19:32:38.176466+0000","last_fullsized":"2024-03-16T19:32:38.176466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":215,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982088+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982088+0000","last_peered":"2024-03-16T19:32:31.982088+0000","last_clean":"2024-03-16T19:32:31.982088+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982088+0000","last_undegraded":"2024-03-16T19:32:31.982088+0000","last_fullsized":"2024-03-16T19:32:31.982088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149110+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149110+0000","last_peered":"2024-03-16T19:32:39.149110+0000","last_clean":"2024-03-16T19:32:39.149110+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149110+0000","last_undegraded":"2024-03-16T19:32:39.149110+0000","last_fullsized":"2024-03-16T19:32:39.149110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108585+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108585+0000","last_peered":"2024-03-16T19:32:32.108585+0000","last_clean":"2024-03-16T19:32:32.108585+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108585+0000","last_undegraded":"2024-03-16T19:32:32.108585+0000","last_fullsized":"2024-03-16T19:32:32.108585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059668+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059668+0000","last_peered":"2024-03-16T19:32:31.059668+0000","last_clean":"2024-03-16T19:32:31.059668+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059668+0000","last_undegraded":"2024-03-16T19:32:31.059668+0000","last_fullsized":"2024-03-16T19:32:31.059668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568574+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568574+0000","last_peered":"2024-03-16T19:32:27.568574+0000","last_clean":"2024-03-16T19:32:27.568574+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568574+0000","last_undegraded":"2024-03-16T19:32:27.568574+0000","last_fullsized":"2024-03-16T19:32:27.568574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651659+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651659+0000","last_peered":"2024-03-16T19:32:37.651659+0000","last_clean":"2024-03-16T19:32:37.651659+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651659+0000","last_undegraded":"2024-03-16T19:32:37.651659+0000","last_fullsized":"2024-03-16T19:32:37.651659+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":61,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205835+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205835+0000","last_peered":"2024-03-16T19:32:28.205835+0000","last_clean":"2024-03-16T19:32:28.205835+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205835+0000","last_undegraded":"2024-03-16T19:32:28.205835+0000","last_fullsized":"2024-03-16T19:32:28.205835+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":166,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166603+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166603+0000","last_peered":"2024-03-16T19:32:40.166603+0000","last_clean":"2024-03-16T19:32:40.166603+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166603+0000","last_undegraded":"2024-03-16T19:32:40.166603+0000","last_fullsized":"2024-03-16T19:32:40.166603+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":376,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:18.230159+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:33:18.230159+0000","last_peered":"2024-03-16T19:33:18.230159+0000","last_clean":"2024-03-16T19:33:18.230159+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:33:18.230159+0000","last_undegraded":"2024-03-16T19:33:18.230159+0000","last_fullsized":"2024-03-16T19:33:18.230159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.532575+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:32:42.532575+0000","last_peered":"2024-03-16T19:32:42.532575+0000","last_clean":"2024-03-16T19:32:42.532575+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:32:42.532575+0000","last_undegraded":"2024-03-16T19:32:42.532575+0000","last_fullsized":"2024-03-16T19:32:42.532575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.219294+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:32:29.219294+0000","last_peered":"2024-03-16T19:32:29.219294+0000","last_clean":"2024-03-16T19:32:29.219294+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:32:29.219294+0000","last_undegraded":"2024-03-16T19:32:29.219294+0000","last_fullsized":"2024-03-16T19:32:29.219294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230834+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:33:28.230834+0000","last_peered":"2024-03-16T19:33:28.230834+0000","last_clean":"2024-03-16T19:33:28.230834+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:33:28.230834+0000","last_undegraded":"2024-03-16T19:33:28.230834+0000","last_fullsized":"2024-03-16T19:33:28.230834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240521+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240521+0000","last_peered":"2024-03-16T19:32:34.240521+0000","last_clean":"2024-03-16T19:32:34.240521+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240521+0000","last_undegraded":"2024-03-16T19:32:34.240521+0000","last_fullsized":"2024-03-16T19:32:34.240521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.145030+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.145030+0000","last_peered":"2024-03-16T19:32:33.145030+0000","last_clean":"2024-03-16T19:32:33.145030+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.145030+0000","last_undegraded":"2024-03-16T19:32:33.145030+0000","last_fullsized":"2024-03-16T19:32:33.145030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":202,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:35:08.249227+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:35:08.249227+0000","last_peered":"2024-03-16T19:35:08.249227+0000","last_clean":"2024-03-16T19:35:08.249227+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:35:08.249227+0000","last_undegraded":"2024-03-16T19:35:08.249227+0000","last_fullsized":"2024-03-16T19:35:08.249227+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 167s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.692412+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:32:30.692412+0000","last_peered":"2024-03-16T19:32:30.692412+0000","last_clean":"2024-03-16T19:32:30.692412+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:32:30.692412+0000","last_undegraded":"2024-03-16T19:32:30.692412+0000","last_fullsized":"2024-03-16T19:32:30.692412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.998707+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:32:30.998707+0000","last_peered":"2024-03-16T19:32:30.998707+0000","last_clean":"2024-03-16T19:32:30.998707+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:32:30.998707+0000","last_undegraded":"2024-03-16T19:32:30.998707+0000","last_fullsized":"2024-03-16T19:32:30.998707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.721534+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:32:37.721534+0000","last_peered":"2024-03-16T19:32:37.721534+0000","last_clean":"2024-03-16T19:32:37.721534+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:32:37.721534+0000","last_undegraded":"2024-03-16T19:32:37.721534+0000","last_fullsized":"2024-03-16T19:32:37.721534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.589670+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:32:39.589670+0000","last_peered":"2024-03-16T19:32:39.589670+0000","last_clean":"2024-03-16T19:32:39.589670+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:32:39.589670+0000","last_undegraded":"2024-03-16T19:32:39.589670+0000","last_fullsized":"2024-03-16T19:32:39.589670+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:46.142320+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:32:46.142320+0000","last_peered":"2024-03-16T19:32:46.142320+0000","last_clean":"2024-03-16T19:32:46.142320+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:32:46.142320+0000","last_undegraded":"2024-03-16T19:32:46.142320+0000","last_fullsized":"2024-03-16T19:32:46.142320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:35:08.249275+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:35:08.249275+0000","last_peered":"2024-03-16T19:35:08.249275+0000","last_clean":"2024-03-16T19:35:08.249275+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:35:08.249275+0000","last_undegraded":"2024-03-16T19:35:08.249275+0000","last_fullsized":"2024-03-16T19:35:08.249275+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 168s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586903+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586903+0000","last_peered":"2024-03-16T19:32:29.586903+0000","last_clean":"2024-03-16T19:32:29.586903+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586903+0000","last_undegraded":"2024-03-16T19:32:29.586903+0000","last_fullsized":"2024-03-16T19:32:29.586903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.125506+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:32:33.125506+0000","last_peered":"2024-03-16T19:32:33.125506+0000","last_clean":"2024-03-16T19:32:33.125506+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:32:33.125506+0000","last_undegraded":"2024-03-16T19:32:33.125506+0000","last_fullsized":"2024-03-16T19:32:33.125506+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910678+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910678+0000","last_peered":"2024-03-16T19:32:31.910678+0000","last_clean":"2024-03-16T19:32:31.910678+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910678+0000","last_undegraded":"2024-03-16T19:32:31.910678+0000","last_fullsized":"2024-03-16T19:32:31.910678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188030+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188030+0000","last_peered":"2024-03-16T19:32:32.188030+0000","last_clean":"2024-03-16T19:32:32.188030+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188030+0000","last_undegraded":"2024-03-16T19:32:32.188030+0000","last_fullsized":"2024-03-16T19:32:32.188030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190121+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190121+0000","last_peered":"2024-03-16T19:32:44.190121+0000","last_clean":"2024-03-16T19:32:44.190121+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190121+0000","last_undegraded":"2024-03-16T19:32:44.190121+0000","last_fullsized":"2024-03-16T19:32:44.190121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026167+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026167+0000","last_peered":"2024-03-16T19:32:30.026167+0000","last_clean":"2024-03-16T19:32:30.026167+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026167+0000","last_undegraded":"2024-03-16T19:32:30.026167+0000","last_fullsized":"2024-03-16T19:32:30.026167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604791+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604791+0000","last_peered":"2024-03-16T19:32:28.604791+0000","last_clean":"2024-03-16T19:32:28.604791+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604791+0000","last_undegraded":"2024-03-16T19:32:28.604791+0000","last_fullsized":"2024-03-16T19:32:28.604791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":27,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:35:07.996078+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:35:07.996078+0000","last_peered":"2024-03-16T19:35:07.996078+0000","last_clean":"2024-03-16T19:35:07.996078+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:35:07.996078+0000","last_undegraded":"2024-03-16T19:35:07.996078+0000","last_fullsized":"2024-03-16T19:35:07.996078+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 168s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699900+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699900+0000","last_peered":"2024-03-16T19:32:33.699900+0000","last_clean":"2024-03-16T19:32:33.699900+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699900+0000","last_undegraded":"2024-03-16T19:32:33.699900+0000","last_fullsized":"2024-03-16T19:32:33.699900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030458+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030458+0000","last_peered":"2024-03-16T19:32:39.030458+0000","last_clean":"2024-03-16T19:32:39.030458+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030458+0000","last_undegraded":"2024-03-16T19:32:39.030458+0000","last_fullsized":"2024-03-16T19:32:39.030458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152425+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152425+0000","last_peered":"2024-03-16T19:32:33.152425+0000","last_clean":"2024-03-16T19:32:33.152425+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152425+0000","last_undegraded":"2024-03-16T19:32:33.152425+0000","last_fullsized":"2024-03-16T19:32:33.152425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889937+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889937+0000","last_peered":"2024-03-16T19:32:33.889937+0000","last_clean":"2024-03-16T19:32:33.889937+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889937+0000","last_undegraded":"2024-03-16T19:32:33.889937+0000","last_fullsized":"2024-03-16T19:32:33.889937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695513+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695513+0000","last_peered":"2024-03-16T19:32:31.695513+0000","last_clean":"2024-03-16T19:32:31.695513+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695513+0000","last_undegraded":"2024-03-16T19:32:31.695513+0000","last_fullsized":"2024-03-16T19:32:31.695513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102649+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102649+0000","last_peered":"2024-03-16T19:32:32.102649+0000","last_clean":"2024-03-16T19:32:32.102649+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102649+0000","last_undegraded":"2024-03-16T19:32:32.102649+0000","last_fullsized":"2024-03-16T19:32:32.102649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696352+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696352+0000","last_peered":"2024-03-16T19:32:32.696352+0000","last_clean":"2024-03-16T19:32:32.696352+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696352+0000","last_undegraded":"2024-03-16T19:32:32.696352+0000","last_fullsized":"2024-03-16T19:32:32.696352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177567+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177567+0000","last_peered":"2024-03-16T19:32:35.177567+0000","last_clean":"2024-03-16T19:32:35.177567+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177567+0000","last_undegraded":"2024-03-16T19:32:35.177567+0000","last_fullsized":"2024-03-16T19:32:35.177567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230775+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230775+0000","last_peered":"2024-03-16T19:32:33.230775+0000","last_clean":"2024-03-16T19:32:33.230775+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230775+0000","last_undegraded":"2024-03-16T19:32:33.230775+0000","last_fullsized":"2024-03-16T19:32:33.230775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236387+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:33:28.236387+0000","last_peered":"2024-03-16T19:33:28.236387+0000","last_clean":"2024-03-16T19:33:28.236387+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:33:28.236387+0000","last_undegraded":"2024-03-16T19:33:28.236387+0000","last_fullsized":"2024-03-16T19:33:28.236387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236413+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:33:28.236413+0000","last_peered":"2024-03-16T19:33:28.236413+0000","last_clean":"2024-03-16T19:33:28.236413+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:33:28.236413+0000","last_undegraded":"2024-03-16T19:33:28.236413+0000","last_fullsized":"2024-03-16T19:33:28.236413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272673+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272673+0000","last_peered":"2024-03-16T19:32:29.272673+0000","last_clean":"2024-03-16T19:32:29.272673+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272673+0000","last_undegraded":"2024-03-16T19:32:29.272673+0000","last_fullsized":"2024-03-16T19:32:29.272673+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574928,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574929,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574927,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122095,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574928,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:35:35.780 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:35.780 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:35.780 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:35.780 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:35.781 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:35:36.573 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:36.595+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:36.609 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:36.631+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:37.586 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:37.608+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:37.615 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:37.637+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:38.561 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:38.583+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:38.590 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:38.612+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:39.605 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:39.627+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:39.635 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:39.657+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:40.560 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:40.580+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:40.591 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:40.611+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:41.526 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:41.547+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:41.637 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:41.658+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:42.528 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:42.549+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:42.678 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:42.699+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:43.519 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:43.540+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:43.630 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:43.650+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:44.568 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:44.589+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:44.663 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:44.684+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:45.613 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:45.634+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:45.614 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:45.635+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:46.571 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:46.591+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:46.640 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:46.660+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:47.541 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:47.562+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:47.600 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:47.620+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:48.553 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:48.573+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:48.574 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:48.595+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:49.547 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:49.568+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:49.580 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:49.600+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:50.574 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:50.594+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:50.612 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:50.633+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:51.560 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:51.580+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:51.651 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:51.671+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:52.551 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:52.572+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:52.692 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:52.712+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:53.542 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:53.562+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:53.698 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:53.719+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:54.542 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:54.562+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:54.673 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:54.694+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:55.528 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:55.549+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:55.676 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:55.696+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:55.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:35:56.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.149+0000 7fa0ba015640 1 Processor -- start 2024-03-16T19:35:56.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.149+0000 7fa0ba015640 1 -- start start 2024-03-16T19:35:56.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0ba015640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa0b416aeb0 0x7fa0b416b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:56.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0ba015640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa0b40732d0 0x7fa0b41648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0ba015640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b4164e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0ba015640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b40c25b0 con 0x7fa0b40732d0 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0ba015640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b406b630 con 0x7fa0b416aeb0 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0ba015640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b40c1b70 con 0x7fa0b40c9650 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0b2ffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa0b40732d0 0x7fa0b41648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0b2ffd640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fa0b40732d0 0x7fa0b41648e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:33250/0 (socket says 172.21.15.172:33250) 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0b3fff640 1 --2- 172.21.15.172:0/2096177467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b4164e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0b2ffd640 1 -- 172.21.15.172:0/2096177467 learned_addr learned my addr 172.21.15.172:0/2096177467 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.150+0000 7fa0b37fe640 1 --2- 172.21.15.172:0/2096177467 >> v2:172.21.15.174:3300/0 conn(0x7fa0b416aeb0 0x7fa0b416b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.151+0000 7fa0b3fff640 1 -- 172.21.15.172:0/2096177467 >> v2:172.21.15.174:3300/0 conn(0x7fa0b416aeb0 msgr2=0x7fa0b416b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:35:56.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.151+0000 7fa0b3fff640 1 --2- 172.21.15.172:0/2096177467 >> v2:172.21.15.174:3300/0 conn(0x7fa0b416aeb0 0x7fa0b416b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.151+0000 7fa0b3fff640 1 -- 172.21.15.172:0/2096177467 >> v2:172.21.15.172:3300/0 conn(0x7fa0b40732d0 msgr2=0x7fa0b41648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:35:56.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.151+0000 7fa0b3fff640 1 --2- 172.21.15.172:0/2096177467 >> v2:172.21.15.172:3300/0 conn(0x7fa0b40732d0 0x7fa0b41648e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.151+0000 7fa0b3fff640 1 -- 172.21.15.172:0/2096177467 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b40c1e50 con 0x7fa0b40c9650 2024-03-16T19:35:56.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.151+0000 7fa0b37fe640 1 --2- 172.21.15.172:0/2096177467 >> v2:172.21.15.174:3300/0 conn(0x7fa0b416aeb0 0x7fa0b416b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:35:56.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.151+0000 7fa0b2ffd640 1 --2- 172.21.15.172:0/2096177467 >> v2:172.21.15.172:3300/0 conn(0x7fa0b40732d0 0x7fa0b41648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:35:56.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.151+0000 7fa0b3fff640 1 --2- 172.21.15.172:0/2096177467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b4164e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40cd0f0 tx=0x7fa0ac0301a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db18f7d01c0ac500 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:35:56.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.152+0000 7fa0b27fc640 1 -- 172.21.15.172:0/2096177467 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac00e0d0 con 0x7fa0b40c9650 2024-03-16T19:35:56.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.152+0000 7fa0b27fc640 1 -- 172.21.15.172:0/2096177467 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac00e230 con 0x7fa0b40c9650 2024-03-16T19:35:56.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.152+0000 7fa0b27fc640 1 -- 172.21.15.172:0/2096177467 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac003990 con 0x7fa0b40c9650 2024-03-16T19:35:56.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.152+0000 7fa0ba015640 1 -- 172.21.15.172:0/2096177467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 msgr2=0x7fa0b4164e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.152+0000 7fa0ba015640 1 --2- 172.21.15.172:0/2096177467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b4164e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40cd0f0 tx=0x7fa0ac0301a0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.153+0000 7fa0ba015640 1 -- 172.21.15.172:0/2096177467 shutdown_connections 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.153+0000 7fa0ba015640 1 --2- 172.21.15.172:0/2096177467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b4164e20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.153+0000 7fa0ba015640 1 --2- 172.21.15.172:0/2096177467 >> v2:172.21.15.172:3300/0 conn(0x7fa0b40732d0 0x7fa0b41648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.153+0000 7fa0ba015640 1 --2- 172.21.15.172:0/2096177467 >> v2:172.21.15.174:3300/0 conn(0x7fa0b416aeb0 0x7fa0b416b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.153+0000 7fa0ba015640 1 -- 172.21.15.172:0/2096177467 >> 172.21.15.172:0/2096177467 conn(0x7fa0b40c8b30 msgr2=0x7fa0b40c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.153+0000 7fa0ba015640 1 -- 172.21.15.172:0/2096177467 shutdown_connections 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.153+0000 7fa0ba015640 1 -- 172.21.15.172:0/2096177467 wait complete. 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.154+0000 7fa0ba015640 1 Processor -- start 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.154+0000 7fa0ba015640 1 -- start start 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0ba015640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b40732d0 0x7fa0b4168250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0ba015640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b415e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0ba015640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0b416aeb0 0x7fa0b4070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0ba015640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41676b0 con 0x7fa0b416aeb0 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0ba015640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41fd5f0 con 0x7fa0b40732d0 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0ba015640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b40cb100 con 0x7fa0b40c9650 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b3fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0b416aeb0 0x7fa0b4070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b40732d0 0x7fa0b4168250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b2ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b415e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b40732d0 0x7fa0b4168250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:38858/0 (socket says 172.21.15.172:38858) 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b37fe640 1 -- 172.21.15.172:0/3810603240 learned_addr learned my addr 172.21.15.172:0/3810603240 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b2ffd640 1 -- 172.21.15.172:0/3810603240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b40732d0 msgr2=0x7fa0b4168250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b2ffd640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b40732d0 0x7fa0b4168250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b2ffd640 1 -- 172.21.15.172:0/3810603240 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0b416aeb0 msgr2=0x7fa0b4070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:56.137 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b2ffd640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0b416aeb0 0x7fa0b4070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b2ffd640 1 -- 172.21.15.172:0/3810603240 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a800b580 con 0x7fa0b40c9650 2024-03-16T19:35:56.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.155+0000 7fa0b2ffd640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b415e4d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800fb70 tx=0x7fa0a800d720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:35:56.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.156+0000 7fa09bfff640 1 -- 172.21.15.172:0/3810603240 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8018640 con 0x7fa0b40c9650 2024-03-16T19:35:56.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.156+0000 7fa09bfff640 1 -- 172.21.15.172:0/3810603240 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a80187a0 con 0x7fa0b40c9650 2024-03-16T19:35:56.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.156+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a4009400 con 0x7fa0b40c9650 2024-03-16T19:35:56.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.156+0000 7fa09bfff640 1 -- 172.21.15.172:0/3810603240 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8018b10 con 0x7fa0b40c9650 2024-03-16T19:35:56.138 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.156+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ac00ec70 con 0x7fa0b40c9650 2024-03-16T19:35:56.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.157+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b41653a0 con 0x7fa0b40c9650 2024-03-16T19:35:56.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.157+0000 7fa09bfff640 1 -- 172.21.15.172:0/3810603240 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa0a8018c70 con 0x7fa0b40c9650 2024-03-16T19:35:56.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.158+0000 7fa09bfff640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa0a006b430 0x7fa0a006d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:35:56.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.158+0000 7fa09bfff640 1 -- 172.21.15.172:0/3810603240 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fa0a808a460 con 0x7fa0b40c9650 2024-03-16T19:35:56.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.158+0000 7fa0b37fe640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa0a006b430 0x7fa0a006d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:35:56.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.158+0000 7fa0b37fe640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa0a006b430 0x7fa0a006d8f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009d30 tx=0x7fa0a4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:35:56.141 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.162+0000 7fa09bfff640 1 -- 172.21.15.172:0/3810603240 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0a8055120 con 0x7fa0b40c9650 2024-03-16T19:35:56.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.423+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0b40c88c0 con 0x7fa0a006b430 2024-03-16T19:35:56.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.435+0000 7fa09bfff640 1 -- 172.21.15.172:0/3810603240 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252929 (secure 0 0 0) 0x7fa0b40c88c0 con 0x7fa0a006b430 2024-03-16T19:35:56.417 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:35:56.418 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:35:56.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.438+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa0a006b430 msgr2=0x7fa0a006d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:56.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.438+0000 7fa0ba015640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa0a006b430 0x7fa0a006d8f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009d30 tx=0x7fa0a4009190 comp rx=0 tx=0).stop 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.438+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 msgr2=0x7fa0b415e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.438+0000 7fa0ba015640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b415e4d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800fb70 tx=0x7fa0a800d720 comp rx=0 tx=0).stop 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.439+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 shutdown_connections 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.439+0000 7fa0ba015640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa0b416aeb0 0x7fa0b4070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.439+0000 7fa0ba015640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fa0a006b430 0x7fa0a006d8f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.439+0000 7fa0ba015640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0b40c9650 0x7fa0b415e4d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.439+0000 7fa0ba015640 1 --2- 172.21.15.172:0/3810603240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b40732d0 0x7fa0b4168250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.439+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 >> 172.21.15.172:0/3810603240 conn(0x7fa0b40c8b30 msgr2=0x7fa0b4161f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.439+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 shutdown_connections 2024-03-16T19:35:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:35:56.439+0000 7fa0ba015640 1 -- 172.21.15.172:0/3810603240 wait complete. 2024-03-16T19:35:56.432 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-03-16T19:35:56.276557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002516"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231608+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:33:38.231608+0000","last_peered":"2024-03-16T19:33:38.231608+0000","last_clean":"2024-03-16T19:33:38.231608+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:33:38.231608+0000","last_undegraded":"2024-03-16T19:33:38.231608+0000","last_fullsized":"2024-03-16T19:33:38.231608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241715+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:33:38.241715+0000","last_peered":"2024-03-16T19:33:38.241715+0000","last_clean":"2024-03-16T19:33:38.241715+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:33:38.241715+0000","last_undegraded":"2024-03-16T19:33:38.241715+0000","last_fullsized":"2024-03-16T19:33:38.241715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":114,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.235727+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:33:38.235727+0000","last_peered":"2024-03-16T19:33:38.235727+0000","last_clean":"2024-03-16T19:33:38.235727+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:33:38.235727+0000","last_undegraded":"2024-03-16T19:33:38.235727+0000","last_fullsized":"2024-03-16T19:33:38.235727+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230354+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:33:28.230354+0000","last_peered":"2024-03-16T19:33:28.230354+0000","last_clean":"2024-03-16T19:33:28.230354+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:33:28.230354+0000","last_undegraded":"2024-03-16T19:33:28.230354+0000","last_fullsized":"2024-03-16T19:33:28.230354+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193204+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193204+0000","last_peered":"2024-03-16T19:32:42.193204+0000","last_clean":"2024-03-16T19:32:42.193204+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193204+0000","last_undegraded":"2024-03-16T19:32:42.193204+0000","last_fullsized":"2024-03-16T19:32:42.193204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501752+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501752+0000","last_peered":"2024-03-16T19:32:43.501752+0000","last_clean":"2024-03-16T19:32:43.501752+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501752+0000","last_undegraded":"2024-03-16T19:32:43.501752+0000","last_fullsized":"2024-03-16T19:32:43.501752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180916+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180916+0000","last_peered":"2024-03-16T19:32:45.180916+0000","last_clean":"2024-03-16T19:32:45.180916+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180916+0000","last_undegraded":"2024-03-16T19:32:45.180916+0000","last_fullsized":"2024-03-16T19:32:45.180916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142618+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142618+0000","last_peered":"2024-03-16T19:32:31.142618+0000","last_clean":"2024-03-16T19:32:31.142618+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142618+0000","last_undegraded":"2024-03-16T19:32:31.142618+0000","last_fullsized":"2024-03-16T19:32:31.142618+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050037+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050037+0000","last_peered":"2024-03-16T19:32:30.050037+0000","last_clean":"2024-03-16T19:32:30.050037+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050037+0000","last_undegraded":"2024-03-16T19:32:30.050037+0000","last_fullsized":"2024-03-16T19:32:30.050037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906526+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906526+0000","last_peered":"2024-03-16T19:32:36.906526+0000","last_clean":"2024-03-16T19:32:36.906526+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906526+0000","last_undegraded":"2024-03-16T19:32:36.906526+0000","last_fullsized":"2024-03-16T19:32:36.906526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946474+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946474+0000","last_peered":"2024-03-16T19:32:37.946474+0000","last_clean":"2024-03-16T19:32:37.946474+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946474+0000","last_undegraded":"2024-03-16T19:32:37.946474+0000","last_fullsized":"2024-03-16T19:32:37.946474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012365+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012365+0000","last_peered":"2024-03-16T19:32:29.012365+0000","last_clean":"2024-03-16T19:32:29.012365+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012365+0000","last_undegraded":"2024-03-16T19:32:29.012365+0000","last_fullsized":"2024-03-16T19:32:29.012365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085970+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085970+0000","last_peered":"2024-03-16T19:32:34.085970+0000","last_clean":"2024-03-16T19:32:34.085970+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085970+0000","last_undegraded":"2024-03-16T19:32:34.085970+0000","last_fullsized":"2024-03-16T19:32:34.085970+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194818+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194818+0000","last_peered":"2024-03-16T19:32:31.194818+0000","last_clean":"2024-03-16T19:32:31.194818+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194818+0000","last_undegraded":"2024-03-16T19:32:31.194818+0000","last_fullsized":"2024-03-16T19:32:31.194818+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622169+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622169+0000","last_peered":"2024-03-16T19:32:30.622169+0000","last_clean":"2024-03-16T19:32:30.622169+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622169+0000","last_undegraded":"2024-03-16T19:32:30.622169+0000","last_fullsized":"2024-03-16T19:32:30.622169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.528006+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.528006+0000","last_peered":"2024-03-16T19:32:41.528006+0000","last_clean":"2024-03-16T19:32:41.528006+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.528006+0000","last_undegraded":"2024-03-16T19:32:41.528006+0000","last_fullsized":"2024-03-16T19:32:41.528006+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668910+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668910+0000","last_peered":"2024-03-16T19:32:33.668910+0000","last_clean":"2024-03-16T19:32:33.668910+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668910+0000","last_undegraded":"2024-03-16T19:32:33.668910+0000","last_fullsized":"2024-03-16T19:32:33.668910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158111+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158111+0000","last_peered":"2024-03-16T19:32:30.158111+0000","last_clean":"2024-03-16T19:32:30.158111+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158111+0000","last_undegraded":"2024-03-16T19:32:30.158111+0000","last_fullsized":"2024-03-16T19:32:30.158111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.198985+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:32:43.198985+0000","last_peered":"2024-03-16T19:32:43.198985+0000","last_clean":"2024-03-16T19:32:43.198985+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:32:43.198985+0000","last_undegraded":"2024-03-16T19:32:43.198985+0000","last_fullsized":"2024-03-16T19:32:43.198985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.874927+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:32:34.874927+0000","last_peered":"2024-03-16T19:32:34.874927+0000","last_clean":"2024-03-16T19:32:34.874927+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:32:34.874927+0000","last_undegraded":"2024-03-16T19:32:34.874927+0000","last_fullsized":"2024-03-16T19:32:34.874927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202050+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202050+0000","last_peered":"2024-03-16T19:32:26.202050+0000","last_clean":"2024-03-16T19:32:26.202050+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202050+0000","last_undegraded":"2024-03-16T19:32:26.202050+0000","last_fullsized":"2024-03-16T19:32:26.202050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113826+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113826+0000","last_peered":"2024-03-16T19:32:31.113826+0000","last_clean":"2024-03-16T19:32:31.113826+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113826+0000","last_undegraded":"2024-03-16T19:32:31.113826+0000","last_fullsized":"2024-03-16T19:32:31.113826+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.189280+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:32:30.189280+0000","last_peered":"2024-03-16T19:32:30.189280+0000","last_clean":"2024-03-16T19:32:30.189280+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:32:30.189280+0000","last_undegraded":"2024-03-16T19:32:30.189280+0000","last_fullsized":"2024-03-16T19:32:30.189280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.633708+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:32:38.633708+0000","last_peered":"2024-03-16T19:32:38.633708+0000","last_clean":"2024-03-16T19:32:38.633708+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:32:38.633708+0000","last_undegraded":"2024-03-16T19:32:38.633708+0000","last_fullsized":"2024-03-16T19:32:38.633708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":172,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.030046+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:32:36.030046+0000","last_peered":"2024-03-16T19:32:36.030046+0000","last_clean":"2024-03-16T19:32:36.030046+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:32:36.030046+0000","last_undegraded":"2024-03-16T19:32:36.030046+0000","last_fullsized":"2024-03-16T19:32:36.030046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.881976+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:32:35.881976+0000","last_peered":"2024-03-16T19:32:35.881976+0000","last_clean":"2024-03-16T19:32:35.881976+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:32:35.881976+0000","last_undegraded":"2024-03-16T19:32:35.881976+0000","last_fullsized":"2024-03-16T19:32:35.881976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.651765+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:32:31.651765+0000","last_peered":"2024-03-16T19:32:31.651765+0000","last_clean":"2024-03-16T19:32:31.651765+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:32:31.651765+0000","last_undegraded":"2024-03-16T19:32:31.651765+0000","last_fullsized":"2024-03-16T19:32:31.651765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237366+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:33:38.237366+0000","last_peered":"2024-03-16T19:33:38.237366+0000","last_clean":"2024-03-16T19:33:38.237366+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:33:38.237366+0000","last_undegraded":"2024-03-16T19:33:38.237366+0000","last_fullsized":"2024-03-16T19:33:38.237366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":98,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.209568+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:32:37.209568+0000","last_peered":"2024-03-16T19:32:37.209568+0000","last_clean":"2024-03-16T19:32:37.209568+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:32:37.209568+0000","last_undegraded":"2024-03-16T19:32:37.209568+0000","last_fullsized":"2024-03-16T19:32:37.209568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231143+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:33:38.231143+0000","last_peered":"2024-03-16T19:33:38.231143+0000","last_clean":"2024-03-16T19:33:38.231143+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:33:38.231143+0000","last_undegraded":"2024-03-16T19:33:38.231143+0000","last_fullsized":"2024-03-16T19:33:38.231143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.216691+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:32:27.216691+0000","last_peered":"2024-03-16T19:32:27.216691+0000","last_clean":"2024-03-16T19:32:27.216691+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:32:27.216691+0000","last_undegraded":"2024-03-16T19:32:27.216691+0000","last_fullsized":"2024-03-16T19:32:27.216691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.230545+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:33:38.230545+0000","last_peered":"2024-03-16T19:33:38.230545+0000","last_clean":"2024-03-16T19:33:38.230545+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:33:38.230545+0000","last_undegraded":"2024-03-16T19:33:38.230545+0000","last_fullsized":"2024-03-16T19:33:38.230545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002778+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002778+0000","last_peered":"2024-03-16T19:32:28.002778+0000","last_clean":"2024-03-16T19:32:28.002778+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002778+0000","last_undegraded":"2024-03-16T19:32:28.002778+0000","last_fullsized":"2024-03-16T19:32:28.002778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207579+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207579+0000","last_peered":"2024-03-16T19:32:28.207579+0000","last_clean":"2024-03-16T19:32:28.207579+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207579+0000","last_undegraded":"2024-03-16T19:32:28.207579+0000","last_fullsized":"2024-03-16T19:32:28.207579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265410+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265410+0000","last_peered":"2024-03-16T19:32:35.265410+0000","last_clean":"2024-03-16T19:32:35.265410+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265410+0000","last_undegraded":"2024-03-16T19:32:35.265410+0000","last_fullsized":"2024-03-16T19:32:35.265410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913659+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913659+0000","last_peered":"2024-03-16T19:32:32.913659+0000","last_clean":"2024-03-16T19:32:32.913659+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913659+0000","last_undegraded":"2024-03-16T19:32:32.913659+0000","last_fullsized":"2024-03-16T19:32:32.913659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105281+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105281+0000","last_peered":"2024-03-16T19:32:32.105281+0000","last_clean":"2024-03-16T19:32:32.105281+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105281+0000","last_undegraded":"2024-03-16T19:32:32.105281+0000","last_fullsized":"2024-03-16T19:32:32.105281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062784+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062784+0000","last_peered":"2024-03-16T19:32:38.062784+0000","last_clean":"2024-03-16T19:32:38.062784+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062784+0000","last_undegraded":"2024-03-16T19:32:38.062784+0000","last_fullsized":"2024-03-16T19:32:38.062784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.686005+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.686005+0000","last_peered":"2024-03-16T19:32:28.686005+0000","last_clean":"2024-03-16T19:32:28.686005+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.686005+0000","last_undegraded":"2024-03-16T19:32:28.686005+0000","last_fullsized":"2024-03-16T19:32:28.686005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691258+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691258+0000","last_peered":"2024-03-16T19:32:32.691258+0000","last_clean":"2024-03-16T19:32:32.691258+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691258+0000","last_undegraded":"2024-03-16T19:32:32.691258+0000","last_fullsized":"2024-03-16T19:32:32.691258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944590+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944590+0000","last_peered":"2024-03-16T19:32:30.944590+0000","last_clean":"2024-03-16T19:32:30.944590+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944590+0000","last_undegraded":"2024-03-16T19:32:30.944590+0000","last_fullsized":"2024-03-16T19:32:30.944590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706805+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706805+0000","last_peered":"2024-03-16T19:32:34.706805+0000","last_clean":"2024-03-16T19:32:34.706805+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706805+0000","last_undegraded":"2024-03-16T19:32:34.706805+0000","last_fullsized":"2024-03-16T19:32:34.706805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646648+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646648+0000","last_peered":"2024-03-16T19:32:29.646648+0000","last_clean":"2024-03-16T19:32:29.646648+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646648+0000","last_undegraded":"2024-03-16T19:32:29.646648+0000","last_fullsized":"2024-03-16T19:32:29.646648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223858+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223858+0000","last_peered":"2024-03-16T19:32:30.223858+0000","last_clean":"2024-03-16T19:32:30.223858+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223858+0000","last_undegraded":"2024-03-16T19:32:30.223858+0000","last_fullsized":"2024-03-16T19:32:30.223858+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538424+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538424+0000","last_peered":"2024-03-16T19:32:26.538424+0000","last_clean":"2024-03-16T19:32:26.538424+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538424+0000","last_undegraded":"2024-03-16T19:32:26.538424+0000","last_fullsized":"2024-03-16T19:32:26.538424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":31,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076164+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076164+0000","last_peered":"2024-03-16T19:32:35.076164+0000","last_clean":"2024-03-16T19:32:35.076164+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076164+0000","last_undegraded":"2024-03-16T19:32:35.076164+0000","last_fullsized":"2024-03-16T19:32:35.076164+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":247,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954039+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954039+0000","last_peered":"2024-03-16T19:32:38.954039+0000","last_clean":"2024-03-16T19:32:38.954039+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954039+0000","last_undegraded":"2024-03-16T19:32:38.954039+0000","last_fullsized":"2024-03-16T19:32:38.954039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179941+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179941+0000","last_peered":"2024-03-16T19:32:41.179941+0000","last_clean":"2024-03-16T19:32:41.179941+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179941+0000","last_undegraded":"2024-03-16T19:32:41.179941+0000","last_fullsized":"2024-03-16T19:32:41.179941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245702+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245702+0000","last_peered":"2024-03-16T19:32:36.245702+0000","last_clean":"2024-03-16T19:32:36.245702+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245702+0000","last_undegraded":"2024-03-16T19:32:36.245702+0000","last_fullsized":"2024-03-16T19:32:36.245702+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197908+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197908+0000","last_peered":"2024-03-16T19:32:27.197908+0000","last_clean":"2024-03-16T19:32:27.197908+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197908+0000","last_undegraded":"2024-03-16T19:32:27.197908+0000","last_fullsized":"2024-03-16T19:32:27.197908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696958+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696958+0000","last_peered":"2024-03-16T19:32:36.696958+0000","last_clean":"2024-03-16T19:32:36.696958+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696958+0000","last_undegraded":"2024-03-16T19:32:36.696958+0000","last_fullsized":"2024-03-16T19:32:36.696958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190754+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190754+0000","last_peered":"2024-03-16T19:32:37.190754+0000","last_clean":"2024-03-16T19:32:37.190754+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190754+0000","last_undegraded":"2024-03-16T19:32:37.190754+0000","last_fullsized":"2024-03-16T19:32:37.190754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938129+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938129+0000","last_peered":"2024-03-16T19:32:29.938129+0000","last_clean":"2024-03-16T19:32:29.938129+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938129+0000","last_undegraded":"2024-03-16T19:32:29.938129+0000","last_fullsized":"2024-03-16T19:32:29.938129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932726+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932726+0000","last_peered":"2024-03-16T19:32:39.932726+0000","last_clean":"2024-03-16T19:32:39.932726+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932726+0000","last_undegraded":"2024-03-16T19:32:39.932726+0000","last_fullsized":"2024-03-16T19:32:39.932726+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":77,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:35:32.998366+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:35:32.998366+0000","last_peered":"2024-03-16T19:35:32.998366+0000","last_clean":"2024-03-16T19:35:32.998366+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:35:32.998366+0000","last_undegraded":"2024-03-16T19:35:32.998366+0000","last_fullsized":"2024-03-16T19:35:32.998366+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 198s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176466+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176466+0000","last_peered":"2024-03-16T19:32:38.176466+0000","last_clean":"2024-03-16T19:32:38.176466+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176466+0000","last_undegraded":"2024-03-16T19:32:38.176466+0000","last_fullsized":"2024-03-16T19:32:38.176466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":215,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.982088+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:32:31.982088+0000","last_peered":"2024-03-16T19:32:31.982088+0000","last_clean":"2024-03-16T19:32:31.982088+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:32:31.982088+0000","last_undegraded":"2024-03-16T19:32:31.982088+0000","last_fullsized":"2024-03-16T19:32:31.982088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.149110+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:32:39.149110+0000","last_peered":"2024-03-16T19:32:39.149110+0000","last_clean":"2024-03-16T19:32:39.149110+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:32:39.149110+0000","last_undegraded":"2024-03-16T19:32:39.149110+0000","last_fullsized":"2024-03-16T19:32:39.149110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108585+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108585+0000","last_peered":"2024-03-16T19:32:32.108585+0000","last_clean":"2024-03-16T19:32:32.108585+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108585+0000","last_undegraded":"2024-03-16T19:32:32.108585+0000","last_fullsized":"2024-03-16T19:32:32.108585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.059668+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:32:31.059668+0000","last_peered":"2024-03-16T19:32:31.059668+0000","last_clean":"2024-03-16T19:32:31.059668+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:32:31.059668+0000","last_undegraded":"2024-03-16T19:32:31.059668+0000","last_fullsized":"2024-03-16T19:32:31.059668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.568574+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:32:27.568574+0000","last_peered":"2024-03-16T19:32:27.568574+0000","last_clean":"2024-03-16T19:32:27.568574+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:32:27.568574+0000","last_undegraded":"2024-03-16T19:32:27.568574+0000","last_fullsized":"2024-03-16T19:32:27.568574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.651659+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:32:37.651659+0000","last_peered":"2024-03-16T19:32:37.651659+0000","last_clean":"2024-03-16T19:32:37.651659+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:32:37.651659+0000","last_undegraded":"2024-03-16T19:32:37.651659+0000","last_fullsized":"2024-03-16T19:32:37.651659+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":61,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205835+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205835+0000","last_peered":"2024-03-16T19:32:28.205835+0000","last_clean":"2024-03-16T19:32:28.205835+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205835+0000","last_undegraded":"2024-03-16T19:32:28.205835+0000","last_fullsized":"2024-03-16T19:32:28.205835+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":166,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166603+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166603+0000","last_peered":"2024-03-16T19:32:40.166603+0000","last_clean":"2024-03-16T19:32:40.166603+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166603+0000","last_undegraded":"2024-03-16T19:32:40.166603+0000","last_fullsized":"2024-03-16T19:32:40.166603+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":376,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:18.230159+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:33:18.230159+0000","last_peered":"2024-03-16T19:33:18.230159+0000","last_clean":"2024-03-16T19:33:18.230159+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:33:18.230159+0000","last_undegraded":"2024-03-16T19:33:18.230159+0000","last_fullsized":"2024-03-16T19:33:18.230159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231177+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:33:38.231177+0000","last_peered":"2024-03-16T19:33:38.231177+0000","last_clean":"2024-03-16T19:33:38.231177+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:33:38.231177+0000","last_undegraded":"2024-03-16T19:33:38.231177+0000","last_fullsized":"2024-03-16T19:33:38.231177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:33.236883+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:33:33.236883+0000","last_peered":"2024-03-16T19:33:33.236883+0000","last_clean":"2024-03-16T19:33:33.236883+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:33:33.236883+0000","last_undegraded":"2024-03-16T19:33:33.236883+0000","last_fullsized":"2024-03-16T19:33:33.236883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230834+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:33:28.230834+0000","last_peered":"2024-03-16T19:33:28.230834+0000","last_clean":"2024-03-16T19:33:28.230834+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:33:28.230834+0000","last_undegraded":"2024-03-16T19:33:28.230834+0000","last_fullsized":"2024-03-16T19:33:28.230834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.240521+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:32:34.240521+0000","last_peered":"2024-03-16T19:32:34.240521+0000","last_clean":"2024-03-16T19:32:34.240521+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:32:34.240521+0000","last_undegraded":"2024-03-16T19:32:34.240521+0000","last_fullsized":"2024-03-16T19:32:34.240521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.145030+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:32:33.145030+0000","last_peered":"2024-03-16T19:32:33.145030+0000","last_clean":"2024-03-16T19:32:33.145030+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:32:33.145030+0000","last_undegraded":"2024-03-16T19:32:33.145030+0000","last_fullsized":"2024-03-16T19:32:33.145030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":204,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:35:38.251798+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:35:38.251798+0000","last_peered":"2024-03-16T19:35:38.251798+0000","last_clean":"2024-03-16T19:35:38.251798+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:35:38.251798+0000","last_undegraded":"2024-03-16T19:35:38.251798+0000","last_fullsized":"2024-03-16T19:35:38.251798+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 197s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231164+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:33:38.231164+0000","last_peered":"2024-03-16T19:33:38.231164+0000","last_clean":"2024-03-16T19:33:38.231164+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:33:38.231164+0000","last_undegraded":"2024-03-16T19:33:38.231164+0000","last_fullsized":"2024-03-16T19:33:38.231164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231343+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:33:38.231343+0000","last_peered":"2024-03-16T19:33:38.231343+0000","last_clean":"2024-03-16T19:33:38.231343+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:33:38.231343+0000","last_undegraded":"2024-03-16T19:33:38.231343+0000","last_fullsized":"2024-03-16T19:33:38.231343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241695+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:33:38.241695+0000","last_peered":"2024-03-16T19:33:38.241695+0000","last_clean":"2024-03-16T19:33:38.241695+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:33:38.241695+0000","last_undegraded":"2024-03-16T19:33:38.241695+0000","last_fullsized":"2024-03-16T19:33:38.241695+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231208+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:33:38.231208+0000","last_peered":"2024-03-16T19:33:38.231208+0000","last_clean":"2024-03-16T19:33:38.231208+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:33:38.231208+0000","last_undegraded":"2024-03-16T19:33:38.231208+0000","last_fullsized":"2024-03-16T19:33:38.231208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231584+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:33:38.231584+0000","last_peered":"2024-03-16T19:33:38.231584+0000","last_clean":"2024-03-16T19:33:38.231584+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:33:38.231584+0000","last_undegraded":"2024-03-16T19:33:38.231584+0000","last_fullsized":"2024-03-16T19:33:38.231584+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":204,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:35:38.251843+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:35:38.251843+0000","last_peered":"2024-03-16T19:35:38.251843+0000","last_clean":"2024-03-16T19:35:38.251843+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:35:38.251843+0000","last_undegraded":"2024-03-16T19:35:38.251843+0000","last_fullsized":"2024-03-16T19:35:38.251843+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 198s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.586903+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:32:29.586903+0000","last_peered":"2024-03-16T19:32:29.586903+0000","last_clean":"2024-03-16T19:32:29.586903+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:32:29.586903+0000","last_undegraded":"2024-03-16T19:32:29.586903+0000","last_fullsized":"2024-03-16T19:32:29.586903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237393+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:33:38.237393+0000","last_peered":"2024-03-16T19:33:38.237393+0000","last_clean":"2024-03-16T19:33:38.237393+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:33:38.237393+0000","last_undegraded":"2024-03-16T19:33:38.237393+0000","last_fullsized":"2024-03-16T19:33:38.237393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.910678+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:32:31.910678+0000","last_peered":"2024-03-16T19:32:31.910678+0000","last_clean":"2024-03-16T19:32:31.910678+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:32:31.910678+0000","last_undegraded":"2024-03-16T19:32:31.910678+0000","last_fullsized":"2024-03-16T19:32:31.910678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.188030+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:32:32.188030+0000","last_peered":"2024-03-16T19:32:32.188030+0000","last_clean":"2024-03-16T19:32:32.188030+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:32:32.188030+0000","last_undegraded":"2024-03-16T19:32:32.188030+0000","last_fullsized":"2024-03-16T19:32:32.188030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:44.190121+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:32:44.190121+0000","last_peered":"2024-03-16T19:32:44.190121+0000","last_clean":"2024-03-16T19:32:44.190121+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:32:44.190121+0000","last_undegraded":"2024-03-16T19:32:44.190121+0000","last_fullsized":"2024-03-16T19:32:44.190121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.026167+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:32:30.026167+0000","last_peered":"2024-03-16T19:32:30.026167+0000","last_clean":"2024-03-16T19:32:30.026167+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:32:30.026167+0000","last_undegraded":"2024-03-16T19:32:30.026167+0000","last_fullsized":"2024-03-16T19:32:30.026167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.604791+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:32:28.604791+0000","last_peered":"2024-03-16T19:32:28.604791+0000","last_clean":"2024-03-16T19:32:28.604791+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:32:28.604791+0000","last_undegraded":"2024-03-16T19:32:28.604791+0000","last_fullsized":"2024-03-16T19:32:28.604791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:35:37.998858+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:35:37.998858+0000","last_peered":"2024-03-16T19:35:37.998858+0000","last_clean":"2024-03-16T19:35:37.998858+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:35:37.998858+0000","last_undegraded":"2024-03-16T19:35:37.998858+0000","last_fullsized":"2024-03-16T19:35:37.998858+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 198s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.699900+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:32:33.699900+0000","last_peered":"2024-03-16T19:32:33.699900+0000","last_clean":"2024-03-16T19:32:33.699900+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:32:33.699900+0000","last_undegraded":"2024-03-16T19:32:33.699900+0000","last_fullsized":"2024-03-16T19:32:33.699900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.030458+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:32:39.030458+0000","last_peered":"2024-03-16T19:32:39.030458+0000","last_clean":"2024-03-16T19:32:39.030458+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:32:39.030458+0000","last_undegraded":"2024-03-16T19:32:39.030458+0000","last_fullsized":"2024-03-16T19:32:39.030458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.152425+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:32:33.152425+0000","last_peered":"2024-03-16T19:32:33.152425+0000","last_clean":"2024-03-16T19:32:33.152425+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:32:33.152425+0000","last_undegraded":"2024-03-16T19:32:33.152425+0000","last_fullsized":"2024-03-16T19:32:33.152425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.889937+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:32:33.889937+0000","last_peered":"2024-03-16T19:32:33.889937+0000","last_clean":"2024-03-16T19:32:33.889937+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:32:33.889937+0000","last_undegraded":"2024-03-16T19:32:33.889937+0000","last_fullsized":"2024-03-16T19:32:33.889937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695513+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695513+0000","last_peered":"2024-03-16T19:32:31.695513+0000","last_clean":"2024-03-16T19:32:31.695513+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695513+0000","last_undegraded":"2024-03-16T19:32:31.695513+0000","last_fullsized":"2024-03-16T19:32:31.695513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102649+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102649+0000","last_peered":"2024-03-16T19:32:32.102649+0000","last_clean":"2024-03-16T19:32:32.102649+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102649+0000","last_undegraded":"2024-03-16T19:32:32.102649+0000","last_fullsized":"2024-03-16T19:32:32.102649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696352+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696352+0000","last_peered":"2024-03-16T19:32:32.696352+0000","last_clean":"2024-03-16T19:32:32.696352+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696352+0000","last_undegraded":"2024-03-16T19:32:32.696352+0000","last_fullsized":"2024-03-16T19:32:32.696352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177567+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177567+0000","last_peered":"2024-03-16T19:32:35.177567+0000","last_clean":"2024-03-16T19:32:35.177567+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177567+0000","last_undegraded":"2024-03-16T19:32:35.177567+0000","last_fullsized":"2024-03-16T19:32:35.177567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230775+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230775+0000","last_peered":"2024-03-16T19:32:33.230775+0000","last_clean":"2024-03-16T19:32:33.230775+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230775+0000","last_undegraded":"2024-03-16T19:32:33.230775+0000","last_fullsized":"2024-03-16T19:32:33.230775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236387+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:33:28.236387+0000","last_peered":"2024-03-16T19:33:28.236387+0000","last_clean":"2024-03-16T19:33:28.236387+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:33:28.236387+0000","last_undegraded":"2024-03-16T19:33:28.236387+0000","last_fullsized":"2024-03-16T19:33:28.236387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236413+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:33:28.236413+0000","last_peered":"2024-03-16T19:33:28.236413+0000","last_clean":"2024-03-16T19:33:28.236413+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:33:28.236413+0000","last_undegraded":"2024-03-16T19:33:28.236413+0000","last_fullsized":"2024-03-16T19:33:28.236413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.272673+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:32:29.272673+0000","last_peered":"2024-03-16T19:32:29.272673+0000","last_clean":"2024-03-16T19:32:29.272673+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:32:29.272673+0000","last_undegraded":"2024-03-16T19:32:29.272673+0000","last_fullsized":"2024-03-16T19:32:29.272673+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574932,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574931,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574933,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122099,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574932,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:35:56.438 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:56.438 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:56.438 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:56.439 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:35:56.439 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:35:56.492 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:56.513+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:56.667 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:56.688+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:57.443 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:57.464+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:57.712 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:57.733+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:58.482 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:58.503+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:58.664 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:58.685+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:59.453 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:35:59.474+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:35:59.680 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:35:59.701+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:00.429 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:00.451+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:00.695 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:00.717+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:01.409 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:01.431+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:01.690 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:01.711+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:02.384 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:02.405+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:02.728 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:02.750+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:03.413 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:03.435+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:03.710 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:03.731+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:04.459 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:04.480+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:04.687 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:04.708+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:05.508 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:05.529+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:05.663 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:05.685+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:06.530 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:06.552+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:06.711 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:06.732+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:07.523 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:07.545+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:07.750 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:07.772+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:08.571 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:08.593+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:08.789 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:08.811+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:09.577 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:09.599+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:09.826 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:09.848+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:10.582 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:10.604+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:10.869 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:10.873+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:11.547 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:11.568+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:11.883 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:11.905+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:12.519 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:12.540+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:12.909 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:12.930+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:13.530 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:13.552+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:13.873 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:13.895+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:14.545 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:14.567+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:14.914 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:14.936+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:15.520 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:15.542+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:15.929 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:15.951+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:16.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:36:16.483 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:16.505+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:16.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.576+0000 7ffbc11f4640 1 Processor -- start 2024-03-16T19:36:16.554 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.576+0000 7ffbc11f4640 1 -- start start 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.577+0000 7ffbc11f4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffbbc152840 0x7ffbbc06e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.577+0000 7ffbc11f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc0c3610 0x7ffbbc0caba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.577+0000 7ffbc11f4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc1590a0 con 0x7ffbbc06fe10 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.577+0000 7ffbc11f4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbbc0c4260 con 0x7ffbbc152840 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.577+0000 7ffbc11f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc0c31c0 con 0x7ffbbc0c3610 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.577+0000 7ffbba59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc0c3610 0x7ffbbc0caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.577+0000 7ffbbad9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffbbc152840 0x7ffbbc06e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.577+0000 7ffbbb59e640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7ffbbc06fe10 0x7ffbbc15bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:50360/0 (socket says 172.21.15.172:50360) 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.577+0000 7ffbbb59e640 1 -- 172.21.15.172:0/2766874073 learned_addr learned my addr 172.21.15.172:0/2766874073 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.578+0000 7ffbbad9d640 1 -- 172.21.15.172:0/2766874073 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc0c3610 msgr2=0x7ffbbc0caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:36:16.558 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.578+0000 7ffbbad9d640 1 --2- 172.21.15.172:0/2766874073 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc0c3610 0x7ffbbc0caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.578+0000 7ffbbad9d640 1 -- 172.21.15.172:0/2766874073 >> v1:172.21.15.174:6789/0 conn(0x7ffbbc06fe10 legacy=0x7ffbbc15bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.578+0000 7ffbb9d9b640 1 -- 172.21.15.172:0/2766874073 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967692281 0 0) 0x7ffbbc1590a0 con 0x7ffbbc06fe10 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.578+0000 7ffbbad9d640 1 -- 172.21.15.172:0/2766874073 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc1595d0 con 0x7ffbbc152840 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.578+0000 7ffbba59c640 1 --2- 172.21.15.172:0/2766874073 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc0c3610 0x7ffbbc0caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbbad9d640 1 --2- 172.21.15.172:0/2766874073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbbc152840 0x7ffbbc06e330 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0097a0 tx=0x7ffbac02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da9ab72fdc79f499 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbb9d9b640 1 -- 172.21.15.172:0/2766874073 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac02d040 con 0x7ffbbc152840 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbb9d9b640 1 -- 172.21.15.172:0/2766874073 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac031810 con 0x7ffbbc152840 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbb9d9b640 1 -- 172.21.15.172:0/2766874073 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac031ba0 con 0x7ffbbc152840 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2766874073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbbc152840 msgr2=0x7ffbbc06e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbc11f4640 1 --2- 172.21.15.172:0/2766874073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbbc152840 0x7ffbbc06e330 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0097a0 tx=0x7ffbac02b8b0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2766874073 shutdown_connections 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbc11f4640 1 --2- 172.21.15.172:0/2766874073 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc0c3610 0x7ffbbc0caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.559 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbc11f4640 1 --2- 172.21.15.172:0/2766874073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbbc152840 0x7ffbbc06e330 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2766874073 >> 172.21.15.172:0/2766874073 conn(0x7ffbbc070860 msgr2=0x7ffbbc070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:16.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2766874073 shutdown_connections 2024-03-16T19:36:16.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.579+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2766874073 wait complete. 2024-03-16T19:36:16.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.581+0000 7ffbc11f4640 1 Processor -- start 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.581+0000 7ffbc11f4640 1 -- start start 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.581+0000 7ffbc11f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbbc06fe10 0x7ffbbc14de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.581+0000 7ffbc11f4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c3610 0x7ffbbc0b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbc11f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc152840 0x7ffbbc1f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbc11f4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc1f0150 con 0x7ffbbc0c3610 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbc11f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc14e440 con 0x7ffbbc06fe10 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbc11f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc14ca20 con 0x7ffbbc152840 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbba59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c3610 0x7ffbbc0b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbba59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c3610 0x7ffbbc0b2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:48708/0 (socket says 172.21.15.172:48708) 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbba59c640 1 -- 172.21.15.172:0/2293875579 learned_addr learned my addr 172.21.15.172:0/2293875579 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:36:16.562 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbbad9d640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbbc06fe10 0x7ffbbc14de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbbb59e640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc152840 0x7ffbbc1f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbba59c640 1 -- 172.21.15.172:0/2293875579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc152840 msgr2=0x7ffbbc1f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbba59c640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc152840 0x7ffbbc1f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbba59c640 1 -- 172.21.15.172:0/2293875579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbbc06fe10 msgr2=0x7ffbbc14de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbba59c640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbbc06fe10 0x7ffbbc14de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbba59c640 1 -- 172.21.15.172:0/2293875579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0009400 con 0x7ffbbc0c3610 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffbba59c640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c3610 0x7ffbbc0b2d70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000ec10 tx=0x7ffbb000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffba37fe640 1 -- 172.21.15.172:0/2293875579 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0009760 con 0x7ffbbc0c3610 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffba37fe640 1 -- 172.21.15.172:0/2293875579 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb000cc30 con 0x7ffbbc0c3610 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.582+0000 7ffba37fe640 1 -- 172.21.15.172:0/2293875579 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0013470 con 0x7ffbbc0c3610 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.583+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac009400 con 0x7ffbbc0c3610 2024-03-16T19:36:16.563 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.583+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc14cfe0 con 0x7ffbbc0c3610 2024-03-16T19:36:16.564 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.584+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc1f02a0 con 0x7ffbbc0c3610 2024-03-16T19:36:16.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.584+0000 7ffba37fe640 1 -- 172.21.15.172:0/2293875579 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7ffbb00135d0 con 0x7ffbbc0c3610 2024-03-16T19:36:16.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.585+0000 7ffba37fe640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffba806b3e0 0x7ffba806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:16.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.585+0000 7ffba37fe640 1 -- 172.21.15.172:0/2293875579 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7ffbb008a2b0 con 0x7ffbbc0c3610 2024-03-16T19:36:16.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.585+0000 7ffbbad9d640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffba806b3e0 0x7ffba806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:16.565 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.585+0000 7ffbbad9d640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffba806b3e0 0x7ffba806d8a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ffbac036000 tx=0x7ffbac03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:16.567 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.589+0000 7ffba37fe640 1 -- 172.21.15.172:0/2293875579 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffbbc1f02a0 con 0x7ffbbc0c3610 2024-03-16T19:36:16.860 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.882+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffbbc0cbb90 con 0x7ffba806b3e0 2024-03-16T19:36:16.869 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.890+0000 7ffba37fe640 1 -- 172.21.15.172:0/2293875579 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252929 (secure 0 0 0) 0x7ffbac044400 con 0x7ffba806b3e0 2024-03-16T19:36:16.872 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:36:16.873 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:36:16.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffba806b3e0 msgr2=0x7ffba806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:16.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffba806b3e0 0x7ffba806d8a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ffbac036000 tx=0x7ffbac03c040 comp rx=0 tx=0).stop 2024-03-16T19:36:16.874 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c3610 msgr2=0x7ffbbc0b2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:16.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c3610 0x7ffbbc0b2d70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000ec10 tx=0x7ffbb000c520 comp rx=0 tx=0).stop 2024-03-16T19:36:16.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 shutdown_connections 2024-03-16T19:36:16.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7ffba806b3e0 0x7ffba806d8a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbbc152840 0x7ffbbc1f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbbc0c3610 0x7ffbbc0b2d70 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 --2- 172.21.15.172:0/2293875579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbbc06fe10 0x7ffbbc14de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:16.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 >> 172.21.15.172:0/2293875579 conn(0x7ffbbc070860 msgr2=0x7ffbbc0c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:16.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 shutdown_connections 2024-03-16T19:36:16.875 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:16.895+0000 7ffbc11f4640 1 -- 172.21.15.172:0/2293875579 wait complete. 2024-03-16T19:36:16.888 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-03-16T19:36:16.280664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002568"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231608+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:33:38.231608+0000","last_peered":"2024-03-16T19:33:38.231608+0000","last_clean":"2024-03-16T19:33:38.231608+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:33:38.231608+0000","last_undegraded":"2024-03-16T19:33:38.231608+0000","last_fullsized":"2024-03-16T19:33:38.231608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241715+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:33:38.241715+0000","last_peered":"2024-03-16T19:33:38.241715+0000","last_clean":"2024-03-16T19:33:38.241715+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:33:38.241715+0000","last_undegraded":"2024-03-16T19:33:38.241715+0000","last_fullsized":"2024-03-16T19:33:38.241715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":114,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.235727+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:33:38.235727+0000","last_peered":"2024-03-16T19:33:38.235727+0000","last_clean":"2024-03-16T19:33:38.235727+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:33:38.235727+0000","last_undegraded":"2024-03-16T19:33:38.235727+0000","last_fullsized":"2024-03-16T19:33:38.235727+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230354+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:33:28.230354+0000","last_peered":"2024-03-16T19:33:28.230354+0000","last_clean":"2024-03-16T19:33:28.230354+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:33:28.230354+0000","last_undegraded":"2024-03-16T19:33:28.230354+0000","last_fullsized":"2024-03-16T19:33:28.230354+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193204+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193204+0000","last_peered":"2024-03-16T19:32:42.193204+0000","last_clean":"2024-03-16T19:32:42.193204+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193204+0000","last_undegraded":"2024-03-16T19:32:42.193204+0000","last_fullsized":"2024-03-16T19:32:42.193204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501752+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501752+0000","last_peered":"2024-03-16T19:32:43.501752+0000","last_clean":"2024-03-16T19:32:43.501752+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501752+0000","last_undegraded":"2024-03-16T19:32:43.501752+0000","last_fullsized":"2024-03-16T19:32:43.501752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180916+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180916+0000","last_peered":"2024-03-16T19:32:45.180916+0000","last_clean":"2024-03-16T19:32:45.180916+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180916+0000","last_undegraded":"2024-03-16T19:32:45.180916+0000","last_fullsized":"2024-03-16T19:32:45.180916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.142618+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:32:31.142618+0000","last_peered":"2024-03-16T19:32:31.142618+0000","last_clean":"2024-03-16T19:32:31.142618+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:32:31.142618+0000","last_undegraded":"2024-03-16T19:32:31.142618+0000","last_fullsized":"2024-03-16T19:32:31.142618+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050037+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050037+0000","last_peered":"2024-03-16T19:32:30.050037+0000","last_clean":"2024-03-16T19:32:30.050037+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050037+0000","last_undegraded":"2024-03-16T19:32:30.050037+0000","last_fullsized":"2024-03-16T19:32:30.050037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906526+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906526+0000","last_peered":"2024-03-16T19:32:36.906526+0000","last_clean":"2024-03-16T19:32:36.906526+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906526+0000","last_undegraded":"2024-03-16T19:32:36.906526+0000","last_fullsized":"2024-03-16T19:32:36.906526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946474+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946474+0000","last_peered":"2024-03-16T19:32:37.946474+0000","last_clean":"2024-03-16T19:32:37.946474+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946474+0000","last_undegraded":"2024-03-16T19:32:37.946474+0000","last_fullsized":"2024-03-16T19:32:37.946474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012365+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012365+0000","last_peered":"2024-03-16T19:32:29.012365+0000","last_clean":"2024-03-16T19:32:29.012365+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012365+0000","last_undegraded":"2024-03-16T19:32:29.012365+0000","last_fullsized":"2024-03-16T19:32:29.012365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.085970+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:32:34.085970+0000","last_peered":"2024-03-16T19:32:34.085970+0000","last_clean":"2024-03-16T19:32:34.085970+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:32:34.085970+0000","last_undegraded":"2024-03-16T19:32:34.085970+0000","last_fullsized":"2024-03-16T19:32:34.085970+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.194818+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:32:31.194818+0000","last_peered":"2024-03-16T19:32:31.194818+0000","last_clean":"2024-03-16T19:32:31.194818+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:32:31.194818+0000","last_undegraded":"2024-03-16T19:32:31.194818+0000","last_fullsized":"2024-03-16T19:32:31.194818+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.622169+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:32:30.622169+0000","last_peered":"2024-03-16T19:32:30.622169+0000","last_clean":"2024-03-16T19:32:30.622169+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:32:30.622169+0000","last_undegraded":"2024-03-16T19:32:30.622169+0000","last_fullsized":"2024-03-16T19:32:30.622169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.528006+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:32:41.528006+0000","last_peered":"2024-03-16T19:32:41.528006+0000","last_clean":"2024-03-16T19:32:41.528006+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:32:41.528006+0000","last_undegraded":"2024-03-16T19:32:41.528006+0000","last_fullsized":"2024-03-16T19:32:41.528006+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.668910+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:32:33.668910+0000","last_peered":"2024-03-16T19:32:33.668910+0000","last_clean":"2024-03-16T19:32:33.668910+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:32:33.668910+0000","last_undegraded":"2024-03-16T19:32:33.668910+0000","last_fullsized":"2024-03-16T19:32:33.668910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.158111+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:32:30.158111+0000","last_peered":"2024-03-16T19:32:30.158111+0000","last_clean":"2024-03-16T19:32:30.158111+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:32:30.158111+0000","last_undegraded":"2024-03-16T19:32:30.158111+0000","last_fullsized":"2024-03-16T19:32:30.158111+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233025+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:33:58.233025+0000","last_peered":"2024-03-16T19:33:58.233025+0000","last_clean":"2024-03-16T19:33:58.233025+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:33:58.233025+0000","last_undegraded":"2024-03-16T19:33:58.233025+0000","last_fullsized":"2024-03-16T19:33:58.233025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.231988+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:33:58.231988+0000","last_peered":"2024-03-16T19:33:58.231988+0000","last_clean":"2024-03-16T19:33:58.231988+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:33:58.231988+0000","last_undegraded":"2024-03-16T19:33:58.231988+0000","last_fullsized":"2024-03-16T19:33:58.231988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.202050+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:32:26.202050+0000","last_peered":"2024-03-16T19:32:26.202050+0000","last_clean":"2024-03-16T19:32:26.202050+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:32:26.202050+0000","last_undegraded":"2024-03-16T19:32:26.202050+0000","last_fullsized":"2024-03-16T19:32:26.202050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.113826+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:32:31.113826+0000","last_peered":"2024-03-16T19:32:31.113826+0000","last_clean":"2024-03-16T19:32:31.113826+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:32:31.113826+0000","last_undegraded":"2024-03-16T19:32:31.113826+0000","last_fullsized":"2024-03-16T19:32:31.113826+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237819+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:33:43.237819+0000","last_peered":"2024-03-16T19:33:43.237819+0000","last_clean":"2024-03-16T19:33:43.237819+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:33:43.237819+0000","last_undegraded":"2024-03-16T19:33:43.237819+0000","last_fullsized":"2024-03-16T19:33:43.237819+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231859+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:33:48.231859+0000","last_peered":"2024-03-16T19:33:48.231859+0000","last_clean":"2024-03-16T19:33:48.231859+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:33:48.231859+0000","last_undegraded":"2024-03-16T19:33:48.231859+0000","last_fullsized":"2024-03-16T19:33:48.231859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":173,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.238369+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:33:48.238369+0000","last_peered":"2024-03-16T19:33:48.238369+0000","last_clean":"2024-03-16T19:33:48.238369+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:33:48.238369+0000","last_undegraded":"2024-03-16T19:33:48.238369+0000","last_fullsized":"2024-03-16T19:33:48.238369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231247+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:33:48.231247+0000","last_peered":"2024-03-16T19:33:48.231247+0000","last_clean":"2024-03-16T19:33:48.231247+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:33:48.231247+0000","last_undegraded":"2024-03-16T19:33:48.231247+0000","last_fullsized":"2024-03-16T19:33:48.231247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242533+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:33:48.242533+0000","last_peered":"2024-03-16T19:33:48.242533+0000","last_clean":"2024-03-16T19:33:48.242533+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:33:48.242533+0000","last_undegraded":"2024-03-16T19:33:48.242533+0000","last_fullsized":"2024-03-16T19:33:48.242533+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237366+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:33:38.237366+0000","last_peered":"2024-03-16T19:33:38.237366+0000","last_clean":"2024-03-16T19:33:38.237366+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:33:38.237366+0000","last_undegraded":"2024-03-16T19:33:38.237366+0000","last_fullsized":"2024-03-16T19:33:38.237366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237508+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:33:58.237508+0000","last_peered":"2024-03-16T19:33:58.237508+0000","last_clean":"2024-03-16T19:33:58.237508+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:33:58.237508+0000","last_undegraded":"2024-03-16T19:33:58.237508+0000","last_fullsized":"2024-03-16T19:33:58.237508+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231143+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:33:38.231143+0000","last_peered":"2024-03-16T19:33:38.231143+0000","last_clean":"2024-03-16T19:33:38.231143+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:33:38.231143+0000","last_undegraded":"2024-03-16T19:33:38.231143+0000","last_fullsized":"2024-03-16T19:33:38.231143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":74,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989304+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:33:52.989304+0000","last_peered":"2024-03-16T19:33:52.989304+0000","last_clean":"2024-03-16T19:33:52.989304+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:33:52.989304+0000","last_undegraded":"2024-03-16T19:33:52.989304+0000","last_fullsized":"2024-03-16T19:33:52.989304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.230545+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:33:38.230545+0000","last_peered":"2024-03-16T19:33:38.230545+0000","last_clean":"2024-03-16T19:33:38.230545+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:33:38.230545+0000","last_undegraded":"2024-03-16T19:33:38.230545+0000","last_fullsized":"2024-03-16T19:33:38.230545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002778+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002778+0000","last_peered":"2024-03-16T19:32:28.002778+0000","last_clean":"2024-03-16T19:32:28.002778+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002778+0000","last_undegraded":"2024-03-16T19:32:28.002778+0000","last_fullsized":"2024-03-16T19:32:28.002778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.207579+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:32:28.207579+0000","last_peered":"2024-03-16T19:32:28.207579+0000","last_clean":"2024-03-16T19:32:28.207579+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:32:28.207579+0000","last_undegraded":"2024-03-16T19:32:28.207579+0000","last_fullsized":"2024-03-16T19:32:28.207579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265410+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265410+0000","last_peered":"2024-03-16T19:32:35.265410+0000","last_clean":"2024-03-16T19:32:35.265410+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265410+0000","last_undegraded":"2024-03-16T19:32:35.265410+0000","last_fullsized":"2024-03-16T19:32:35.265410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913659+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913659+0000","last_peered":"2024-03-16T19:32:32.913659+0000","last_clean":"2024-03-16T19:32:32.913659+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913659+0000","last_undegraded":"2024-03-16T19:32:32.913659+0000","last_fullsized":"2024-03-16T19:32:32.913659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.105281+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:32:32.105281+0000","last_peered":"2024-03-16T19:32:32.105281+0000","last_clean":"2024-03-16T19:32:32.105281+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:32:32.105281+0000","last_undegraded":"2024-03-16T19:32:32.105281+0000","last_fullsized":"2024-03-16T19:32:32.105281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":95,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.062784+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:32:38.062784+0000","last_peered":"2024-03-16T19:32:38.062784+0000","last_clean":"2024-03-16T19:32:38.062784+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:32:38.062784+0000","last_undegraded":"2024-03-16T19:32:38.062784+0000","last_fullsized":"2024-03-16T19:32:38.062784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.686005+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.686005+0000","last_peered":"2024-03-16T19:32:28.686005+0000","last_clean":"2024-03-16T19:32:28.686005+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.686005+0000","last_undegraded":"2024-03-16T19:32:28.686005+0000","last_fullsized":"2024-03-16T19:32:28.686005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691258+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691258+0000","last_peered":"2024-03-16T19:32:32.691258+0000","last_clean":"2024-03-16T19:32:32.691258+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691258+0000","last_undegraded":"2024-03-16T19:32:32.691258+0000","last_fullsized":"2024-03-16T19:32:32.691258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944590+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944590+0000","last_peered":"2024-03-16T19:32:30.944590+0000","last_clean":"2024-03-16T19:32:30.944590+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944590+0000","last_undegraded":"2024-03-16T19:32:30.944590+0000","last_fullsized":"2024-03-16T19:32:30.944590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706805+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706805+0000","last_peered":"2024-03-16T19:32:34.706805+0000","last_clean":"2024-03-16T19:32:34.706805+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706805+0000","last_undegraded":"2024-03-16T19:32:34.706805+0000","last_fullsized":"2024-03-16T19:32:34.706805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646648+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646648+0000","last_peered":"2024-03-16T19:32:29.646648+0000","last_clean":"2024-03-16T19:32:29.646648+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646648+0000","last_undegraded":"2024-03-16T19:32:29.646648+0000","last_fullsized":"2024-03-16T19:32:29.646648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223858+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223858+0000","last_peered":"2024-03-16T19:32:30.223858+0000","last_clean":"2024-03-16T19:32:30.223858+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223858+0000","last_undegraded":"2024-03-16T19:32:30.223858+0000","last_fullsized":"2024-03-16T19:32:30.223858+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538424+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538424+0000","last_peered":"2024-03-16T19:32:26.538424+0000","last_clean":"2024-03-16T19:32:26.538424+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538424+0000","last_undegraded":"2024-03-16T19:32:26.538424+0000","last_fullsized":"2024-03-16T19:32:26.538424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":31,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076164+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076164+0000","last_peered":"2024-03-16T19:32:35.076164+0000","last_clean":"2024-03-16T19:32:35.076164+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076164+0000","last_undegraded":"2024-03-16T19:32:35.076164+0000","last_fullsized":"2024-03-16T19:32:35.076164+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":247,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.954039+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:32:38.954039+0000","last_peered":"2024-03-16T19:32:38.954039+0000","last_clean":"2024-03-16T19:32:38.954039+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:32:38.954039+0000","last_undegraded":"2024-03-16T19:32:38.954039+0000","last_fullsized":"2024-03-16T19:32:38.954039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179941+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179941+0000","last_peered":"2024-03-16T19:32:41.179941+0000","last_clean":"2024-03-16T19:32:41.179941+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179941+0000","last_undegraded":"2024-03-16T19:32:41.179941+0000","last_fullsized":"2024-03-16T19:32:41.179941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.245702+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:32:36.245702+0000","last_peered":"2024-03-16T19:32:36.245702+0000","last_clean":"2024-03-16T19:32:36.245702+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:32:36.245702+0000","last_undegraded":"2024-03-16T19:32:36.245702+0000","last_fullsized":"2024-03-16T19:32:36.245702+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:27.197908+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:32:27.197908+0000","last_peered":"2024-03-16T19:32:27.197908+0000","last_clean":"2024-03-16T19:32:27.197908+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:32:27.197908+0000","last_undegraded":"2024-03-16T19:32:27.197908+0000","last_fullsized":"2024-03-16T19:32:27.197908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.696958+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:32:36.696958+0000","last_peered":"2024-03-16T19:32:36.696958+0000","last_clean":"2024-03-16T19:32:36.696958+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:32:36.696958+0000","last_undegraded":"2024-03-16T19:32:36.696958+0000","last_fullsized":"2024-03-16T19:32:36.696958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.190754+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:32:37.190754+0000","last_peered":"2024-03-16T19:32:37.190754+0000","last_clean":"2024-03-16T19:32:37.190754+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:32:37.190754+0000","last_undegraded":"2024-03-16T19:32:37.190754+0000","last_fullsized":"2024-03-16T19:32:37.190754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":190,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.938129+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:32:29.938129+0000","last_peered":"2024-03-16T19:32:29.938129+0000","last_clean":"2024-03-16T19:32:29.938129+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:32:29.938129+0000","last_undegraded":"2024-03-16T19:32:29.938129+0000","last_fullsized":"2024-03-16T19:32:29.938129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:39.932726+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:32:39.932726+0000","last_peered":"2024-03-16T19:32:39.932726+0000","last_clean":"2024-03-16T19:32:39.932726+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:32:39.932726+0000","last_undegraded":"2024-03-16T19:32:39.932726+0000","last_fullsized":"2024-03-16T19:32:39.932726+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":78,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:35:47.999711+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:35:47.999711+0000","last_peered":"2024-03-16T19:35:47.999711+0000","last_clean":"2024-03-16T19:35:47.999711+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:35:47.999711+0000","last_undegraded":"2024-03-16T19:35:47.999711+0000","last_fullsized":"2024-03-16T19:35:47.999711+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 213s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:38.176466+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:32:38.176466+0000","last_peered":"2024-03-16T19:32:38.176466+0000","last_clean":"2024-03-16T19:32:38.176466+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:32:38.176466+0000","last_undegraded":"2024-03-16T19:32:38.176466+0000","last_fullsized":"2024-03-16T19:32:38.176466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232760+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:33:58.232760+0000","last_peered":"2024-03-16T19:33:58.232760+0000","last_clean":"2024-03-16T19:33:58.232760+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:33:58.232760+0000","last_undegraded":"2024-03-16T19:33:58.232760+0000","last_fullsized":"2024-03-16T19:33:58.232760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233035+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:33:58.233035+0000","last_peered":"2024-03-16T19:33:58.233035+0000","last_clean":"2024-03-16T19:33:58.233035+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:33:58.233035+0000","last_undegraded":"2024-03-16T19:33:58.233035+0000","last_fullsized":"2024-03-16T19:33:58.233035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.108585+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:32:32.108585+0000","last_peered":"2024-03-16T19:32:32.108585+0000","last_clean":"2024-03-16T19:32:32.108585+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:32:32.108585+0000","last_undegraded":"2024-03-16T19:32:32.108585+0000","last_fullsized":"2024-03-16T19:32:32.108585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233045+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:33:58.233045+0000","last_peered":"2024-03-16T19:33:58.233045+0000","last_clean":"2024-03-16T19:33:58.233045+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:33:58.233045+0000","last_undegraded":"2024-03-16T19:33:58.233045+0000","last_fullsized":"2024-03-16T19:33:58.233045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.243408+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:33:58.243408+0000","last_peered":"2024-03-16T19:33:58.243408+0000","last_clean":"2024-03-16T19:33:58.243408+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:33:58.243408+0000","last_undegraded":"2024-03-16T19:33:58.243408+0000","last_fullsized":"2024-03-16T19:33:58.243408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232756+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:33:58.232756+0000","last_peered":"2024-03-16T19:33:58.232756+0000","last_clean":"2024-03-16T19:33:58.232756+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:33:58.232756+0000","last_undegraded":"2024-03-16T19:33:58.232756+0000","last_fullsized":"2024-03-16T19:33:58.232756+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":61,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.205835+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:32:28.205835+0000","last_peered":"2024-03-16T19:32:28.205835+0000","last_clean":"2024-03-16T19:32:28.205835+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:32:28.205835+0000","last_undegraded":"2024-03-16T19:32:28.205835+0000","last_fullsized":"2024-03-16T19:32:28.205835+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":166,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166603+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166603+0000","last_peered":"2024-03-16T19:32:40.166603+0000","last_clean":"2024-03-16T19:32:40.166603+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166603+0000","last_undegraded":"2024-03-16T19:32:40.166603+0000","last_fullsized":"2024-03-16T19:32:40.166603+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":376,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:18.230159+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:33:18.230159+0000","last_peered":"2024-03-16T19:33:18.230159+0000","last_clean":"2024-03-16T19:33:18.230159+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:33:18.230159+0000","last_undegraded":"2024-03-16T19:33:18.230159+0000","last_fullsized":"2024-03-16T19:33:18.230159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231177+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:33:38.231177+0000","last_peered":"2024-03-16T19:33:38.231177+0000","last_clean":"2024-03-16T19:33:38.231177+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:33:38.231177+0000","last_undegraded":"2024-03-16T19:33:38.231177+0000","last_fullsized":"2024-03-16T19:33:38.231177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:33.236883+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:33:33.236883+0000","last_peered":"2024-03-16T19:33:33.236883+0000","last_clean":"2024-03-16T19:33:33.236883+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:33:33.236883+0000","last_undegraded":"2024-03-16T19:33:33.236883+0000","last_fullsized":"2024-03-16T19:33:33.236883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230834+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:33:28.230834+0000","last_peered":"2024-03-16T19:33:28.230834+0000","last_clean":"2024-03-16T19:33:28.230834+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:33:28.230834+0000","last_undegraded":"2024-03-16T19:33:28.230834+0000","last_fullsized":"2024-03-16T19:33:28.230834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237468+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:33:58.237468+0000","last_peered":"2024-03-16T19:33:58.237468+0000","last_clean":"2024-03-16T19:33:58.237468+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:33:58.237468+0000","last_undegraded":"2024-03-16T19:33:58.237468+0000","last_fullsized":"2024-03-16T19:33:58.237468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989289+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:33:52.989289+0000","last_peered":"2024-03-16T19:33:52.989289+0000","last_clean":"2024-03-16T19:33:52.989289+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:33:52.989289+0000","last_undegraded":"2024-03-16T19:33:52.989289+0000","last_fullsized":"2024-03-16T19:33:52.989289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":205,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:35:53.253146+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:35:53.253146+0000","last_peered":"2024-03-16T19:35:53.253146+0000","last_clean":"2024-03-16T19:35:53.253146+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:35:53.253146+0000","last_undegraded":"2024-03-16T19:35:53.253146+0000","last_fullsized":"2024-03-16T19:35:53.253146+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 212s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231164+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:33:38.231164+0000","last_peered":"2024-03-16T19:33:38.231164+0000","last_clean":"2024-03-16T19:33:38.231164+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:33:38.231164+0000","last_undegraded":"2024-03-16T19:33:38.231164+0000","last_fullsized":"2024-03-16T19:33:38.231164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231343+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:33:38.231343+0000","last_peered":"2024-03-16T19:33:38.231343+0000","last_clean":"2024-03-16T19:33:38.231343+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:33:38.231343+0000","last_undegraded":"2024-03-16T19:33:38.231343+0000","last_fullsized":"2024-03-16T19:33:38.231343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241695+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:33:38.241695+0000","last_peered":"2024-03-16T19:33:38.241695+0000","last_clean":"2024-03-16T19:33:38.241695+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:33:38.241695+0000","last_undegraded":"2024-03-16T19:33:38.241695+0000","last_fullsized":"2024-03-16T19:33:38.241695+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231208+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:33:38.231208+0000","last_peered":"2024-03-16T19:33:38.231208+0000","last_clean":"2024-03-16T19:33:38.231208+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:33:38.231208+0000","last_undegraded":"2024-03-16T19:33:38.231208+0000","last_fullsized":"2024-03-16T19:33:38.231208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231584+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:33:38.231584+0000","last_peered":"2024-03-16T19:33:38.231584+0000","last_clean":"2024-03-16T19:33:38.231584+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:33:38.231584+0000","last_undegraded":"2024-03-16T19:33:38.231584+0000","last_fullsized":"2024-03-16T19:33:38.231584+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:35:53.253191+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:35:53.253191+0000","last_peered":"2024-03-16T19:35:53.253191+0000","last_clean":"2024-03-16T19:35:53.253191+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:35:53.253191+0000","last_undegraded":"2024-03-16T19:35:53.253191+0000","last_fullsized":"2024-03-16T19:35:53.253191+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 213s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242558+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:33:48.242558+0000","last_peered":"2024-03-16T19:33:48.242558+0000","last_clean":"2024-03-16T19:33:48.242558+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:33:48.242558+0000","last_undegraded":"2024-03-16T19:33:48.242558+0000","last_fullsized":"2024-03-16T19:33:48.242558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237393+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:33:38.237393+0000","last_peered":"2024-03-16T19:33:38.237393+0000","last_clean":"2024-03-16T19:33:38.237393+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:33:38.237393+0000","last_undegraded":"2024-03-16T19:33:38.237393+0000","last_fullsized":"2024-03-16T19:33:38.237393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231258+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:33:48.231258+0000","last_peered":"2024-03-16T19:33:48.231258+0000","last_clean":"2024-03-16T19:33:48.231258+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:33:48.231258+0000","last_undegraded":"2024-03-16T19:33:48.231258+0000","last_fullsized":"2024-03-16T19:33:48.231258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:03.237913+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:34:03.237913+0000","last_peered":"2024-03-16T19:34:03.237913+0000","last_clean":"2024-03-16T19:34:03.237913+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:34:03.237913+0000","last_undegraded":"2024-03-16T19:34:03.237913+0000","last_fullsized":"2024-03-16T19:34:03.237913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232268+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:33:48.232268+0000","last_peered":"2024-03-16T19:33:48.232268+0000","last_clean":"2024-03-16T19:33:48.232268+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:33:48.232268+0000","last_undegraded":"2024-03-16T19:33:48.232268+0000","last_fullsized":"2024-03-16T19:33:48.232268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232070+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:33:48.232070+0000","last_peered":"2024-03-16T19:33:48.232070+0000","last_clean":"2024-03-16T19:33:48.232070+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:33:48.232070+0000","last_undegraded":"2024-03-16T19:33:48.232070+0000","last_fullsized":"2024-03-16T19:33:48.232070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242493+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:33:48.242493+0000","last_peered":"2024-03-16T19:33:48.242493+0000","last_clean":"2024-03-16T19:33:48.242493+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:33:48.242493+0000","last_undegraded":"2024-03-16T19:33:48.242493+0000","last_fullsized":"2024-03-16T19:33:48.242493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:35:53.000132+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:35:53.000132+0000","last_peered":"2024-03-16T19:35:53.000132+0000","last_clean":"2024-03-16T19:35:53.000132+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:35:53.000132+0000","last_undegraded":"2024-03-16T19:35:53.000132+0000","last_fullsized":"2024-03-16T19:35:53.000132+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 213s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242516+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:33:48.242516+0000","last_peered":"2024-03-16T19:33:48.242516+0000","last_clean":"2024-03-16T19:33:48.242516+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:33:48.242516+0000","last_undegraded":"2024-03-16T19:33:48.242516+0000","last_fullsized":"2024-03-16T19:33:48.242516+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237853+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:33:43.237853+0000","last_peered":"2024-03-16T19:33:43.237853+0000","last_clean":"2024-03-16T19:33:43.237853+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:33:43.237853+0000","last_undegraded":"2024-03-16T19:33:43.237853+0000","last_fullsized":"2024-03-16T19:33:43.237853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233004+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:33:58.233004+0000","last_peered":"2024-03-16T19:33:58.233004+0000","last_clean":"2024-03-16T19:33:58.233004+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:33:58.233004+0000","last_undegraded":"2024-03-16T19:33:58.233004+0000","last_fullsized":"2024-03-16T19:33:58.233004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231221+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:33:48.231221+0000","last_peered":"2024-03-16T19:33:48.231221+0000","last_clean":"2024-03-16T19:33:48.231221+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:33:48.231221+0000","last_undegraded":"2024-03-16T19:33:48.231221+0000","last_fullsized":"2024-03-16T19:33:48.231221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695513+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695513+0000","last_peered":"2024-03-16T19:32:31.695513+0000","last_clean":"2024-03-16T19:32:31.695513+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695513+0000","last_undegraded":"2024-03-16T19:32:31.695513+0000","last_fullsized":"2024-03-16T19:32:31.695513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102649+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102649+0000","last_peered":"2024-03-16T19:32:32.102649+0000","last_clean":"2024-03-16T19:32:32.102649+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102649+0000","last_undegraded":"2024-03-16T19:32:32.102649+0000","last_fullsized":"2024-03-16T19:32:32.102649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696352+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696352+0000","last_peered":"2024-03-16T19:32:32.696352+0000","last_clean":"2024-03-16T19:32:32.696352+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696352+0000","last_undegraded":"2024-03-16T19:32:32.696352+0000","last_fullsized":"2024-03-16T19:32:32.696352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177567+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177567+0000","last_peered":"2024-03-16T19:32:35.177567+0000","last_clean":"2024-03-16T19:32:35.177567+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177567+0000","last_undegraded":"2024-03-16T19:32:35.177567+0000","last_fullsized":"2024-03-16T19:32:35.177567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230775+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230775+0000","last_peered":"2024-03-16T19:32:33.230775+0000","last_clean":"2024-03-16T19:32:33.230775+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230775+0000","last_undegraded":"2024-03-16T19:32:33.230775+0000","last_fullsized":"2024-03-16T19:32:33.230775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236387+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:33:28.236387+0000","last_peered":"2024-03-16T19:33:28.236387+0000","last_clean":"2024-03-16T19:33:28.236387+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:33:28.236387+0000","last_undegraded":"2024-03-16T19:33:28.236387+0000","last_fullsized":"2024-03-16T19:33:28.236387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236413+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:33:28.236413+0000","last_peered":"2024-03-16T19:33:28.236413+0000","last_clean":"2024-03-16T19:33:28.236413+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:33:28.236413+0000","last_undegraded":"2024-03-16T19:33:28.236413+0000","last_fullsized":"2024-03-16T19:33:28.236413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237534+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:33:58.237534+0000","last_peered":"2024-03-16T19:33:58.237534+0000","last_clean":"2024-03-16T19:33:58.237534+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:33:58.237534+0000","last_undegraded":"2024-03-16T19:33:58.237534+0000","last_fullsized":"2024-03-16T19:33:58.237534+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574936,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574937,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574937,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122103,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:36:16.893 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:16.893 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:16.893 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:16.893 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:16.894 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:36:16.976 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:16.998+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:17.463 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:17.484+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:17.985 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:18.007+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:18.462 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:18.484+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:19.024 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:19.045+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:19.479 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:19.501+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:20.034 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:20.055+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:20.435 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:20.457+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:21.024 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:21.046+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:21.427 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:21.449+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:21.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:36:21.945 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:36:21.983 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:22.005+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:22.426 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:22.447+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:23.012 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:23.034+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:23.441 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:23.463+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:23.986 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:24.008+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:24.424 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:24.446+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:24.969 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:24.990+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:25.389 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:25.410+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:26.016 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:26.037+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:26.369 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:26.390+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:27.061 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:27.082+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:27.352 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:27.372+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:28.018 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:28.039+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:28.387 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:28.408+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:29.058 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:29.078+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:29.429 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:29.450+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:30.075 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:30.096+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:30.478 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:30.499+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:31.061 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:31.082+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:31.518 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:31.539+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:32.076 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:32.037+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:32.538 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:32.559+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:33.028 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:33.049+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:33.499 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:33.520+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:34.066 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:34.087+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:34.465 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:34.486+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:35.042 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:35.062+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:35.472 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:35.493+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:36.091 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:36.112+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:36.430 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:36.451+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:36.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:36:37.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.048+0000 7f2c40de2640 1 Processor -- start 2024-03-16T19:36:37.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.048+0000 7f2c40de2640 1 -- start start 2024-03-16T19:36:37.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.049+0000 7f2c40de2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c151230 0x7f2c3c06bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:37.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.049+0000 7f2c40de2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2c3c06c500 0x7f2c3c156e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:37.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.049+0000 7f2c40de2640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c151610 con 0x7f2c3c0c9650 2024-03-16T19:36:37.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.049+0000 7f2c40de2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c3c06bb30 con 0x7f2c3c06c500 2024-03-16T19:36:37.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.049+0000 7f2c40de2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0c2440 con 0x7f2c3c151230 2024-03-16T19:36:37.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.049+0000 7f2c3a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c151230 0x7f2c3c06bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:37.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.049+0000 7f2c39d9b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2c3c06c500 0x7f2c3c156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:37.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.050+0000 7f2c3a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c151230 0x7f2c3c06bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:37072/0 (socket says 172.21.15.172:37072) 2024-03-16T19:36:37.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.050+0000 7f2c3a59c640 1 -- 172.21.15.172:0/3108661096 learned_addr learned my addr 172.21.15.172:0/3108661096 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:36:37.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.050+0000 7f2c39d9b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2c3c06c500 0x7f2c3c156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:42220/0 (socket says 172.21.15.172:42220) 2024-03-16T19:36:37.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.050+0000 7f2c3a59c640 1 -- 172.21.15.172:0/3108661096 >> v2:172.21.15.174:3300/0 conn(0x7f2c3c06c500 msgr2=0x7f2c3c156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:36:37.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.050+0000 7f2c3a59c640 1 --2- 172.21.15.172:0/3108661096 >> v2:172.21.15.174:3300/0 conn(0x7f2c3c06c500 0x7f2c3c156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:37.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.050+0000 7f2c3a59c640 1 -- 172.21.15.172:0/3108661096 >> v1:172.21.15.172:6789/0 conn(0x7f2c3c0c9650 legacy=0x7f2c3c157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:37.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.050+0000 7f2c3a59c640 1 -- 172.21.15.172:0/3108661096 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c0c6a30 con 0x7f2c3c151230 2024-03-16T19:36:37.030 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.050+0000 7f2c39d9b640 1 --2- 172.21.15.172:0/3108661096 >> v2:172.21.15.174:3300/0 conn(0x7f2c3c06c500 0x7f2c3c156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:36:37.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.051+0000 7f2c3a59c640 1 --2- 172.21.15.172:0/3108661096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c151230 0x7f2c3c06bfc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009870 tx=0x7f2c2c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97b9182b12e1649a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:37.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.051+0000 7f2c3959a640 1 -- 172.21.15.172:0/3108661096 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c031740 con 0x7f2c3c151230 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.051+0000 7f2c3959a640 1 -- 172.21.15.172:0/3108661096 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2c0318a0 con 0x7f2c3c151230 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.051+0000 7f2c3959a640 1 -- 172.21.15.172:0/3108661096 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c031c30 con 0x7f2c3c151230 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.052+0000 7f2c40de2640 1 -- 172.21.15.172:0/3108661096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c151230 msgr2=0x7f2c3c06bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.052+0000 7f2c40de2640 1 --2- 172.21.15.172:0/3108661096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c151230 0x7f2c3c06bfc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009870 tx=0x7f2c2c02b860 comp rx=0 tx=0).stop 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.052+0000 7f2c40de2640 1 -- 172.21.15.172:0/3108661096 shutdown_connections 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.052+0000 7f2c40de2640 1 --2- 172.21.15.172:0/3108661096 >> v2:172.21.15.174:3300/0 conn(0x7f2c3c06c500 0x7f2c3c156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.052+0000 7f2c40de2640 1 --2- 172.21.15.172:0/3108661096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c151230 0x7f2c3c06bfc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.052+0000 7f2c40de2640 1 -- 172.21.15.172:0/3108661096 >> 172.21.15.172:0/3108661096 conn(0x7f2c3c0c8b30 msgr2=0x7f2c3c0c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.052+0000 7f2c40de2640 1 -- 172.21.15.172:0/3108661096 shutdown_connections 2024-03-16T19:36:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.052+0000 7f2c40de2640 1 -- 172.21.15.172:0/3108661096 wait complete. 2024-03-16T19:36:37.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.054+0000 7f2c40de2640 1 Processor -- start 2024-03-16T19:36:37.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.054+0000 7f2c40de2640 1 -- start start 2024-03-16T19:36:37.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c40de2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c3c06c500 0x7f2c3c157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:37.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c40de2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c0c9650 0x7f2c3c1ecca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:37.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c40de2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c3c151230 0x7f2c3c147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:37.035 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c40de2640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0c78a0 con 0x7f2c3c06c500 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c40de2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0c9f30 con 0x7f2c3c151230 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c40de2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0c7ab0 con 0x7f2c3c0c9650 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c3a59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c3c06c500 0x7f2c3c157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c3a59c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c3c06c500 0x7f2c3c157d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:42900/0 (socket says 172.21.15.172:42900) 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c3a59c640 1 -- 172.21.15.172:0/3804115137 learned_addr learned my addr 172.21.15.172:0/3804115137 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c3ad9d640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c3c151230 0x7f2c3c147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.055+0000 7f2c39d9b640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c0c9650 0x7f2c3c1ecca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.056+0000 7f2c39d9b640 1 -- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c3c151230 msgr2=0x7f2c3c147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.056+0000 7f2c39d9b640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c3c151230 0x7f2c3c147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.056+0000 7f2c39d9b640 1 -- 172.21.15.172:0/3804115137 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c3c06c500 msgr2=0x7f2c3c157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.056+0000 7f2c39d9b640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c3c06c500 0x7f2c3c157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.056+0000 7f2c39d9b640 1 -- 172.21.15.172:0/3804115137 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3000a5a0 con 0x7f2c3c0c9650 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.056+0000 7f2c3ad9d640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c3c151230 0x7f2c3c147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:36:37.036 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.056+0000 7f2c39d9b640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c0c9650 0x7f2c3c1ecca0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0cd0f0 tx=0x7f2c3000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:37.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.056+0000 7f2c2affd640 1 -- 172.21.15.172:0/3804115137 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30004440 con 0x7f2c3c0c9650 2024-03-16T19:36:37.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.056+0000 7f2c2affd640 1 -- 172.21.15.172:0/3804115137 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3000ae80 con 0x7f2c3c0c9650 2024-03-16T19:36:37.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.057+0000 7f2c2affd640 1 -- 172.21.15.172:0/3804115137 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3000ca50 con 0x7f2c3c0c9650 2024-03-16T19:36:37.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.057+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c0094d0 con 0x7f2c3c0c9650 2024-03-16T19:36:37.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.057+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c0ca1f0 con 0x7f2c3c0c9650 2024-03-16T19:36:37.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.058+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c0681c0 con 0x7f2c3c0c9650 2024-03-16T19:36:37.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.058+0000 7f2c2affd640 1 -- 172.21.15.172:0/3804115137 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f2c3000cbb0 con 0x7f2c3c0c9650 2024-03-16T19:36:37.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.059+0000 7f2c2affd640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2c1c06b3b0 0x7f2c1c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:37.038 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.059+0000 7f2c2affd640 1 -- 172.21.15.172:0/3804115137 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f2c30089bf0 con 0x7f2c3c0c9650 2024-03-16T19:36:37.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.059+0000 7f2c3a59c640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2c1c06b3b0 0x7f2c1c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:37.039 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.059+0000 7f2c3a59c640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2c1c06b3b0 0x7f2c1c06d870 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c02b8a0 tx=0x7f2c2c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:37.042 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.063+0000 7f2c2affd640 1 -- 172.21.15.172:0/3804115137 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c300547d0 con 0x7f2c3c0c9650 2024-03-16T19:36:37.044 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:37.065+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:37.295 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.315+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c3c06b800 con 0x7f2c1c06b3b0 2024-03-16T19:36:37.303 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.324+0000 7f2c2affd640 1 -- 172.21.15.172:0/3804115137 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252929 (secure 0 0 0) 0x7f2c2c041070 con 0x7f2c1c06b3b0 2024-03-16T19:36:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:36:37.307 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:36:37.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.327+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2c1c06b3b0 msgr2=0x7f2c1c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:37.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.327+0000 7f2c40de2640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2c1c06b3b0 0x7f2c1c06d870 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c02b8a0 tx=0x7f2c2c008030 comp rx=0 tx=0).stop 2024-03-16T19:36:37.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.327+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c0c9650 msgr2=0x7f2c3c1ecca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:37.308 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.327+0000 7f2c40de2640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c0c9650 0x7f2c3c1ecca0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0cd0f0 tx=0x7f2c3000ed70 comp rx=0 tx=0).stop 2024-03-16T19:36:37.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.328+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 shutdown_connections 2024-03-16T19:36:37.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.328+0000 7f2c40de2640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c3c151230 0x7f2c3c147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:37.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.328+0000 7f2c40de2640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f2c1c06b3b0 0x7f2c1c06d870 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:37.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.328+0000 7f2c40de2640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2c3c0c9650 0x7f2c3c1ecca0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:37.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.328+0000 7f2c40de2640 1 --2- 172.21.15.172:0/3804115137 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2c3c06c500 0x7f2c3c157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:37.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.328+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 >> 172.21.15.172:0/3804115137 conn(0x7f2c3c0c8b30 msgr2=0x7f2c3c071710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:37.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.328+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 shutdown_connections 2024-03-16T19:36:37.309 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:37.328+0000 7f2c40de2640 1 -- 172.21.15.172:0/3804115137 wait complete. 2024-03-16T19:36:37.320 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-03-16T19:36:36.284790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002635"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231608+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:33:38.231608+0000","last_peered":"2024-03-16T19:33:38.231608+0000","last_clean":"2024-03-16T19:33:38.231608+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:33:38.231608+0000","last_undegraded":"2024-03-16T19:33:38.231608+0000","last_fullsized":"2024-03-16T19:33:38.231608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241715+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:33:38.241715+0000","last_peered":"2024-03-16T19:33:38.241715+0000","last_clean":"2024-03-16T19:33:38.241715+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:33:38.241715+0000","last_undegraded":"2024-03-16T19:33:38.241715+0000","last_fullsized":"2024-03-16T19:33:38.241715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":114,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.235727+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:33:38.235727+0000","last_peered":"2024-03-16T19:33:38.235727+0000","last_clean":"2024-03-16T19:33:38.235727+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:33:38.235727+0000","last_undegraded":"2024-03-16T19:33:38.235727+0000","last_fullsized":"2024-03-16T19:33:38.235727+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230354+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:33:28.230354+0000","last_peered":"2024-03-16T19:33:28.230354+0000","last_clean":"2024-03-16T19:33:28.230354+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:33:28.230354+0000","last_undegraded":"2024-03-16T19:33:28.230354+0000","last_fullsized":"2024-03-16T19:33:28.230354+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193204+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193204+0000","last_peered":"2024-03-16T19:32:42.193204+0000","last_clean":"2024-03-16T19:32:42.193204+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193204+0000","last_undegraded":"2024-03-16T19:32:42.193204+0000","last_fullsized":"2024-03-16T19:32:42.193204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501752+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501752+0000","last_peered":"2024-03-16T19:32:43.501752+0000","last_clean":"2024-03-16T19:32:43.501752+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501752+0000","last_undegraded":"2024-03-16T19:32:43.501752+0000","last_fullsized":"2024-03-16T19:32:43.501752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180916+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180916+0000","last_peered":"2024-03-16T19:32:45.180916+0000","last_clean":"2024-03-16T19:32:45.180916+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180916+0000","last_undegraded":"2024-03-16T19:32:45.180916+0000","last_fullsized":"2024-03-16T19:32:45.180916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239229+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:33:58.239229+0000","last_peered":"2024-03-16T19:33:58.239229+0000","last_clean":"2024-03-16T19:33:58.239229+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:33:58.239229+0000","last_undegraded":"2024-03-16T19:33:58.239229+0000","last_fullsized":"2024-03-16T19:33:58.239229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.050037+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:32:30.050037+0000","last_peered":"2024-03-16T19:32:30.050037+0000","last_clean":"2024-03-16T19:32:30.050037+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:32:30.050037+0000","last_undegraded":"2024-03-16T19:32:30.050037+0000","last_fullsized":"2024-03-16T19:32:30.050037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:36.906526+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:32:36.906526+0000","last_peered":"2024-03-16T19:32:36.906526+0000","last_clean":"2024-03-16T19:32:36.906526+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:32:36.906526+0000","last_undegraded":"2024-03-16T19:32:36.906526+0000","last_fullsized":"2024-03-16T19:32:36.906526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:37.946474+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:32:37.946474+0000","last_peered":"2024-03-16T19:32:37.946474+0000","last_clean":"2024-03-16T19:32:37.946474+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:32:37.946474+0000","last_undegraded":"2024-03-16T19:32:37.946474+0000","last_fullsized":"2024-03-16T19:32:37.946474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.012365+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:32:29.012365+0000","last_peered":"2024-03-16T19:32:29.012365+0000","last_clean":"2024-03-16T19:32:29.012365+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:32:29.012365+0000","last_undegraded":"2024-03-16T19:32:29.012365+0000","last_fullsized":"2024-03-16T19:32:29.012365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238872+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:33:53.238872+0000","last_peered":"2024-03-16T19:33:53.238872+0000","last_clean":"2024-03-16T19:33:53.238872+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:33:53.238872+0000","last_undegraded":"2024-03-16T19:33:53.238872+0000","last_fullsized":"2024-03-16T19:33:53.238872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238806+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:34:13.238806+0000","last_peered":"2024-03-16T19:34:13.238806+0000","last_clean":"2024-03-16T19:34:13.238806+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:34:13.238806+0000","last_undegraded":"2024-03-16T19:34:13.238806+0000","last_fullsized":"2024-03-16T19:34:13.238806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.244177+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:34:08.244177+0000","last_peered":"2024-03-16T19:34:08.244177+0000","last_clean":"2024-03-16T19:34:08.244177+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:34:08.244177+0000","last_undegraded":"2024-03-16T19:34:08.244177+0000","last_fullsized":"2024-03-16T19:34:08.244177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233445+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:34:08.233445+0000","last_peered":"2024-03-16T19:34:08.233445+0000","last_clean":"2024-03-16T19:34:08.233445+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:34:08.233445+0000","last_undegraded":"2024-03-16T19:34:08.233445+0000","last_fullsized":"2024-03-16T19:34:08.233445+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233511+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:34:08.233511+0000","last_peered":"2024-03-16T19:34:08.233511+0000","last_clean":"2024-03-16T19:34:08.233511+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:34:08.233511+0000","last_undegraded":"2024-03-16T19:34:08.233511+0000","last_fullsized":"2024-03-16T19:34:08.233511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:07.990588+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:34:07.990588+0000","last_peered":"2024-03-16T19:34:07.990588+0000","last_clean":"2024-03-16T19:34:07.990588+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:34:07.990588+0000","last_undegraded":"2024-03-16T19:34:07.990588+0000","last_fullsized":"2024-03-16T19:34:07.990588+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233025+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:33:58.233025+0000","last_peered":"2024-03-16T19:33:58.233025+0000","last_clean":"2024-03-16T19:33:58.233025+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:33:58.233025+0000","last_undegraded":"2024-03-16T19:33:58.233025+0000","last_fullsized":"2024-03-16T19:33:58.233025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.231988+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:33:58.231988+0000","last_peered":"2024-03-16T19:33:58.231988+0000","last_clean":"2024-03-16T19:33:58.231988+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:33:58.231988+0000","last_undegraded":"2024-03-16T19:33:58.231988+0000","last_fullsized":"2024-03-16T19:33:58.231988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.238290+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:34:08.238290+0000","last_peered":"2024-03-16T19:34:08.238290+0000","last_clean":"2024-03-16T19:34:08.238290+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:34:08.238290+0000","last_undegraded":"2024-03-16T19:34:08.238290+0000","last_fullsized":"2024-03-16T19:34:08.238290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990165+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:34:02.990165+0000","last_peered":"2024-03-16T19:34:02.990165+0000","last_clean":"2024-03-16T19:34:02.990165+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:34:02.990165+0000","last_undegraded":"2024-03-16T19:34:02.990165+0000","last_fullsized":"2024-03-16T19:34:02.990165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237819+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:33:43.237819+0000","last_peered":"2024-03-16T19:33:43.237819+0000","last_clean":"2024-03-16T19:33:43.237819+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:33:43.237819+0000","last_undegraded":"2024-03-16T19:33:43.237819+0000","last_fullsized":"2024-03-16T19:33:43.237819+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231859+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:33:48.231859+0000","last_peered":"2024-03-16T19:33:48.231859+0000","last_clean":"2024-03-16T19:33:48.231859+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:33:48.231859+0000","last_undegraded":"2024-03-16T19:33:48.231859+0000","last_fullsized":"2024-03-16T19:33:48.231859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":173,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.238369+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:33:48.238369+0000","last_peered":"2024-03-16T19:33:48.238369+0000","last_clean":"2024-03-16T19:33:48.238369+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:33:48.238369+0000","last_undegraded":"2024-03-16T19:33:48.238369+0000","last_fullsized":"2024-03-16T19:33:48.238369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231247+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:33:48.231247+0000","last_peered":"2024-03-16T19:33:48.231247+0000","last_clean":"2024-03-16T19:33:48.231247+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:33:48.231247+0000","last_undegraded":"2024-03-16T19:33:48.231247+0000","last_fullsized":"2024-03-16T19:33:48.231247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242533+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:33:48.242533+0000","last_peered":"2024-03-16T19:33:48.242533+0000","last_clean":"2024-03-16T19:33:48.242533+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:33:48.242533+0000","last_undegraded":"2024-03-16T19:33:48.242533+0000","last_fullsized":"2024-03-16T19:33:48.242533+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237366+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:33:38.237366+0000","last_peered":"2024-03-16T19:33:38.237366+0000","last_clean":"2024-03-16T19:33:38.237366+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:33:38.237366+0000","last_undegraded":"2024-03-16T19:33:38.237366+0000","last_fullsized":"2024-03-16T19:33:38.237366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237508+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:33:58.237508+0000","last_peered":"2024-03-16T19:33:58.237508+0000","last_clean":"2024-03-16T19:33:58.237508+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:33:58.237508+0000","last_undegraded":"2024-03-16T19:33:58.237508+0000","last_fullsized":"2024-03-16T19:33:58.237508+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231143+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:33:38.231143+0000","last_peered":"2024-03-16T19:33:38.231143+0000","last_clean":"2024-03-16T19:33:38.231143+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:33:38.231143+0000","last_undegraded":"2024-03-16T19:33:38.231143+0000","last_fullsized":"2024-03-16T19:33:38.231143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":74,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989304+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:33:52.989304+0000","last_peered":"2024-03-16T19:33:52.989304+0000","last_clean":"2024-03-16T19:33:52.989304+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:33:52.989304+0000","last_undegraded":"2024-03-16T19:33:52.989304+0000","last_fullsized":"2024-03-16T19:33:52.989304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.230545+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:33:38.230545+0000","last_peered":"2024-03-16T19:33:38.230545+0000","last_clean":"2024-03-16T19:33:38.230545+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:33:38.230545+0000","last_undegraded":"2024-03-16T19:33:38.230545+0000","last_fullsized":"2024-03-16T19:33:38.230545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002778+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002778+0000","last_peered":"2024-03-16T19:32:28.002778+0000","last_clean":"2024-03-16T19:32:28.002778+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002778+0000","last_undegraded":"2024-03-16T19:32:28.002778+0000","last_fullsized":"2024-03-16T19:32:28.002778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239257+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:33:58.239257+0000","last_peered":"2024-03-16T19:33:58.239257+0000","last_clean":"2024-03-16T19:33:58.239257+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:33:58.239257+0000","last_undegraded":"2024-03-16T19:33:58.239257+0000","last_fullsized":"2024-03-16T19:33:58.239257+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.265410+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:32:35.265410+0000","last_peered":"2024-03-16T19:32:35.265410+0000","last_clean":"2024-03-16T19:32:35.265410+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:32:35.265410+0000","last_undegraded":"2024-03-16T19:32:35.265410+0000","last_fullsized":"2024-03-16T19:32:35.265410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913659+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913659+0000","last_peered":"2024-03-16T19:32:32.913659+0000","last_clean":"2024-03-16T19:32:32.913659+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913659+0000","last_undegraded":"2024-03-16T19:32:32.913659+0000","last_fullsized":"2024-03-16T19:32:32.913659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238854+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:33:53.238854+0000","last_peered":"2024-03-16T19:33:53.238854+0000","last_clean":"2024-03-16T19:33:53.238854+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:33:53.238854+0000","last_undegraded":"2024-03-16T19:33:53.238854+0000","last_fullsized":"2024-03-16T19:33:53.238854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238825+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:33:53.238825+0000","last_peered":"2024-03-16T19:33:53.238825+0000","last_clean":"2024-03-16T19:33:53.238825+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:33:53.238825+0000","last_undegraded":"2024-03-16T19:33:53.238825+0000","last_fullsized":"2024-03-16T19:33:53.238825+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.686005+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:32:28.686005+0000","last_peered":"2024-03-16T19:32:28.686005+0000","last_clean":"2024-03-16T19:32:28.686005+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:32:28.686005+0000","last_undegraded":"2024-03-16T19:32:28.686005+0000","last_fullsized":"2024-03-16T19:32:28.686005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.691258+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:32:32.691258+0000","last_peered":"2024-03-16T19:32:32.691258+0000","last_clean":"2024-03-16T19:32:32.691258+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:32:32.691258+0000","last_undegraded":"2024-03-16T19:32:32.691258+0000","last_fullsized":"2024-03-16T19:32:32.691258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.944590+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:32:30.944590+0000","last_peered":"2024-03-16T19:32:30.944590+0000","last_clean":"2024-03-16T19:32:30.944590+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:32:30.944590+0000","last_undegraded":"2024-03-16T19:32:30.944590+0000","last_fullsized":"2024-03-16T19:32:30.944590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:34.706805+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:32:34.706805+0000","last_peered":"2024-03-16T19:32:34.706805+0000","last_clean":"2024-03-16T19:32:34.706805+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:32:34.706805+0000","last_undegraded":"2024-03-16T19:32:34.706805+0000","last_fullsized":"2024-03-16T19:32:34.706805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:29.646648+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:32:29.646648+0000","last_peered":"2024-03-16T19:32:29.646648+0000","last_clean":"2024-03-16T19:32:29.646648+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:32:29.646648+0000","last_undegraded":"2024-03-16T19:32:29.646648+0000","last_fullsized":"2024-03-16T19:32:29.646648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:30.223858+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:32:30.223858+0000","last_peered":"2024-03-16T19:32:30.223858+0000","last_clean":"2024-03-16T19:32:30.223858+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:32:30.223858+0000","last_undegraded":"2024-03-16T19:32:30.223858+0000","last_fullsized":"2024-03-16T19:32:30.223858+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:26.538424+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:32:26.538424+0000","last_peered":"2024-03-16T19:32:26.538424+0000","last_clean":"2024-03-16T19:32:26.538424+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:32:26.538424+0000","last_undegraded":"2024-03-16T19:32:26.538424+0000","last_fullsized":"2024-03-16T19:32:26.538424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":31,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.076164+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:32:35.076164+0000","last_peered":"2024-03-16T19:32:35.076164+0000","last_clean":"2024-03-16T19:32:35.076164+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:32:35.076164+0000","last_undegraded":"2024-03-16T19:32:35.076164+0000","last_fullsized":"2024-03-16T19:32:35.076164+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":248,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232779+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:34:08.232779+0000","last_peered":"2024-03-16T19:34:08.232779+0000","last_clean":"2024-03-16T19:34:08.232779+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:34:08.232779+0000","last_undegraded":"2024-03-16T19:34:08.232779+0000","last_fullsized":"2024-03-16T19:34:08.232779+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:41.179941+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:32:41.179941+0000","last_peered":"2024-03-16T19:32:41.179941+0000","last_clean":"2024-03-16T19:32:41.179941+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:32:41.179941+0000","last_undegraded":"2024-03-16T19:32:41.179941+0000","last_fullsized":"2024-03-16T19:32:41.179941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238829+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:34:13.238829+0000","last_peered":"2024-03-16T19:34:13.238829+0000","last_clean":"2024-03-16T19:34:13.238829+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:34:13.238829+0000","last_undegraded":"2024-03-16T19:34:13.238829+0000","last_fullsized":"2024-03-16T19:34:13.238829+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238778+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:34:13.238778+0000","last_peered":"2024-03-16T19:34:13.238778+0000","last_clean":"2024-03-16T19:34:13.238778+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:34:13.238778+0000","last_undegraded":"2024-03-16T19:34:13.238778+0000","last_fullsized":"2024-03-16T19:34:13.238778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233485+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:34:08.233485+0000","last_peered":"2024-03-16T19:34:08.233485+0000","last_clean":"2024-03-16T19:34:08.233485+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:34:08.233485+0000","last_undegraded":"2024-03-16T19:34:08.233485+0000","last_fullsized":"2024-03-16T19:34:08.233485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233709+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:34:08.233709+0000","last_peered":"2024-03-16T19:34:08.233709+0000","last_clean":"2024-03-16T19:34:08.233709+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:34:08.233709+0000","last_undegraded":"2024-03-16T19:34:08.233709+0000","last_fullsized":"2024-03-16T19:34:08.233709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232766+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:34:08.232766+0000","last_peered":"2024-03-16T19:34:08.232766+0000","last_clean":"2024-03-16T19:34:08.232766+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:34:08.232766+0000","last_undegraded":"2024-03-16T19:34:08.232766+0000","last_fullsized":"2024-03-16T19:34:08.232766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232748+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:34:08.232748+0000","last_peered":"2024-03-16T19:34:08.232748+0000","last_clean":"2024-03-16T19:34:08.232748+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:34:08.232748+0000","last_undegraded":"2024-03-16T19:34:08.232748+0000","last_fullsized":"2024-03-16T19:34:08.232748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":80,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:36:18.002633+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:36:18.002633+0000","last_peered":"2024-03-16T19:36:18.002633+0000","last_clean":"2024-03-16T19:36:18.002633+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:36:18.002633+0000","last_undegraded":"2024-03-16T19:36:18.002633+0000","last_fullsized":"2024-03-16T19:36:18.002633+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 243s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233693+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:34:08.233693+0000","last_peered":"2024-03-16T19:34:08.233693+0000","last_clean":"2024-03-16T19:34:08.233693+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:34:08.233693+0000","last_undegraded":"2024-03-16T19:34:08.233693+0000","last_fullsized":"2024-03-16T19:34:08.233693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232760+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:33:58.232760+0000","last_peered":"2024-03-16T19:33:58.232760+0000","last_clean":"2024-03-16T19:33:58.232760+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:33:58.232760+0000","last_undegraded":"2024-03-16T19:33:58.232760+0000","last_fullsized":"2024-03-16T19:33:58.232760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233035+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:33:58.233035+0000","last_peered":"2024-03-16T19:33:58.233035+0000","last_clean":"2024-03-16T19:33:58.233035+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:33:58.233035+0000","last_undegraded":"2024-03-16T19:33:58.233035+0000","last_fullsized":"2024-03-16T19:33:58.233035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990184+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:34:02.990184+0000","last_peered":"2024-03-16T19:34:02.990184+0000","last_clean":"2024-03-16T19:34:02.990184+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:34:02.990184+0000","last_undegraded":"2024-03-16T19:34:02.990184+0000","last_fullsized":"2024-03-16T19:34:02.990184+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233045+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:33:58.233045+0000","last_peered":"2024-03-16T19:33:58.233045+0000","last_clean":"2024-03-16T19:33:58.233045+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:33:58.233045+0000","last_undegraded":"2024-03-16T19:33:58.233045+0000","last_fullsized":"2024-03-16T19:33:58.233045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.243408+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:33:58.243408+0000","last_peered":"2024-03-16T19:33:58.243408+0000","last_clean":"2024-03-16T19:33:58.243408+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:33:58.243408+0000","last_undegraded":"2024-03-16T19:33:58.243408+0000","last_fullsized":"2024-03-16T19:33:58.243408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232756+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:33:58.232756+0000","last_peered":"2024-03-16T19:33:58.232756+0000","last_clean":"2024-03-16T19:33:58.232756+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:33:58.232756+0000","last_undegraded":"2024-03-16T19:33:58.232756+0000","last_fullsized":"2024-03-16T19:33:58.232756+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990175+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:34:02.990175+0000","last_peered":"2024-03-16T19:34:02.990175+0000","last_clean":"2024-03-16T19:34:02.990175+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:34:02.990175+0000","last_undegraded":"2024-03-16T19:34:02.990175+0000","last_fullsized":"2024-03-16T19:34:02.990175+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":166,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166603+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166603+0000","last_peered":"2024-03-16T19:32:40.166603+0000","last_clean":"2024-03-16T19:32:40.166603+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166603+0000","last_undegraded":"2024-03-16T19:32:40.166603+0000","last_fullsized":"2024-03-16T19:32:40.166603+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":376,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:18.230159+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:33:18.230159+0000","last_peered":"2024-03-16T19:33:18.230159+0000","last_clean":"2024-03-16T19:33:18.230159+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:33:18.230159+0000","last_undegraded":"2024-03-16T19:33:18.230159+0000","last_fullsized":"2024-03-16T19:33:18.230159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231177+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:33:38.231177+0000","last_peered":"2024-03-16T19:33:38.231177+0000","last_clean":"2024-03-16T19:33:38.231177+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:33:38.231177+0000","last_undegraded":"2024-03-16T19:33:38.231177+0000","last_fullsized":"2024-03-16T19:33:38.231177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:33.236883+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:33:33.236883+0000","last_peered":"2024-03-16T19:33:33.236883+0000","last_clean":"2024-03-16T19:33:33.236883+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:33:33.236883+0000","last_undegraded":"2024-03-16T19:33:33.236883+0000","last_fullsized":"2024-03-16T19:33:33.236883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230834+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:33:28.230834+0000","last_peered":"2024-03-16T19:33:28.230834+0000","last_clean":"2024-03-16T19:33:28.230834+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:33:28.230834+0000","last_undegraded":"2024-03-16T19:33:28.230834+0000","last_fullsized":"2024-03-16T19:33:28.230834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237468+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:33:58.237468+0000","last_peered":"2024-03-16T19:33:58.237468+0000","last_clean":"2024-03-16T19:33:58.237468+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:33:58.237468+0000","last_undegraded":"2024-03-16T19:33:58.237468+0000","last_fullsized":"2024-03-16T19:33:58.237468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989289+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:33:52.989289+0000","last_peered":"2024-03-16T19:33:52.989289+0000","last_clean":"2024-03-16T19:33:52.989289+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:33:52.989289+0000","last_undegraded":"2024-03-16T19:33:52.989289+0000","last_fullsized":"2024-03-16T19:33:52.989289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":206,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:36:08.254283+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:36:08.254283+0000","last_peered":"2024-03-16T19:36:08.254283+0000","last_clean":"2024-03-16T19:36:08.254283+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:36:08.254283+0000","last_undegraded":"2024-03-16T19:36:08.254283+0000","last_fullsized":"2024-03-16T19:36:08.254283+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 227s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231164+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:33:38.231164+0000","last_peered":"2024-03-16T19:33:38.231164+0000","last_clean":"2024-03-16T19:33:38.231164+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:33:38.231164+0000","last_undegraded":"2024-03-16T19:33:38.231164+0000","last_fullsized":"2024-03-16T19:33:38.231164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231343+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:33:38.231343+0000","last_peered":"2024-03-16T19:33:38.231343+0000","last_clean":"2024-03-16T19:33:38.231343+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:33:38.231343+0000","last_undegraded":"2024-03-16T19:33:38.231343+0000","last_fullsized":"2024-03-16T19:33:38.231343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241695+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:33:38.241695+0000","last_peered":"2024-03-16T19:33:38.241695+0000","last_clean":"2024-03-16T19:33:38.241695+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:33:38.241695+0000","last_undegraded":"2024-03-16T19:33:38.241695+0000","last_fullsized":"2024-03-16T19:33:38.241695+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231208+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:33:38.231208+0000","last_peered":"2024-03-16T19:33:38.231208+0000","last_clean":"2024-03-16T19:33:38.231208+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:33:38.231208+0000","last_undegraded":"2024-03-16T19:33:38.231208+0000","last_fullsized":"2024-03-16T19:33:38.231208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231584+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:33:38.231584+0000","last_peered":"2024-03-16T19:33:38.231584+0000","last_clean":"2024-03-16T19:33:38.231584+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:33:38.231584+0000","last_undegraded":"2024-03-16T19:33:38.231584+0000","last_fullsized":"2024-03-16T19:33:38.231584+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:36:08.254328+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:36:08.254328+0000","last_peered":"2024-03-16T19:36:08.254328+0000","last_clean":"2024-03-16T19:36:08.254328+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:36:08.254328+0000","last_undegraded":"2024-03-16T19:36:08.254328+0000","last_fullsized":"2024-03-16T19:36:08.254328+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 228s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242558+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:33:48.242558+0000","last_peered":"2024-03-16T19:33:48.242558+0000","last_clean":"2024-03-16T19:33:48.242558+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:33:48.242558+0000","last_undegraded":"2024-03-16T19:33:48.242558+0000","last_fullsized":"2024-03-16T19:33:48.242558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237393+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:33:38.237393+0000","last_peered":"2024-03-16T19:33:38.237393+0000","last_clean":"2024-03-16T19:33:38.237393+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:33:38.237393+0000","last_undegraded":"2024-03-16T19:33:38.237393+0000","last_fullsized":"2024-03-16T19:33:38.237393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231258+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:33:48.231258+0000","last_peered":"2024-03-16T19:33:48.231258+0000","last_clean":"2024-03-16T19:33:48.231258+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:33:48.231258+0000","last_undegraded":"2024-03-16T19:33:48.231258+0000","last_fullsized":"2024-03-16T19:33:48.231258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:03.237913+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:34:03.237913+0000","last_peered":"2024-03-16T19:34:03.237913+0000","last_clean":"2024-03-16T19:34:03.237913+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:34:03.237913+0000","last_undegraded":"2024-03-16T19:34:03.237913+0000","last_fullsized":"2024-03-16T19:34:03.237913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232268+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:33:48.232268+0000","last_peered":"2024-03-16T19:33:48.232268+0000","last_clean":"2024-03-16T19:33:48.232268+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:33:48.232268+0000","last_undegraded":"2024-03-16T19:33:48.232268+0000","last_fullsized":"2024-03-16T19:33:48.232268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232070+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:33:48.232070+0000","last_peered":"2024-03-16T19:33:48.232070+0000","last_clean":"2024-03-16T19:33:48.232070+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:33:48.232070+0000","last_undegraded":"2024-03-16T19:33:48.232070+0000","last_fullsized":"2024-03-16T19:33:48.232070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242493+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:33:48.242493+0000","last_peered":"2024-03-16T19:33:48.242493+0000","last_clean":"2024-03-16T19:33:48.242493+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:33:48.242493+0000","last_undegraded":"2024-03-16T19:33:48.242493+0000","last_fullsized":"2024-03-16T19:33:48.242493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:36:08.001545+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:36:08.001545+0000","last_peered":"2024-03-16T19:36:08.001545+0000","last_clean":"2024-03-16T19:36:08.001545+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:36:08.001545+0000","last_undegraded":"2024-03-16T19:36:08.001545+0000","last_fullsized":"2024-03-16T19:36:08.001545+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 228s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242516+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:33:48.242516+0000","last_peered":"2024-03-16T19:33:48.242516+0000","last_clean":"2024-03-16T19:33:48.242516+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:33:48.242516+0000","last_undegraded":"2024-03-16T19:33:48.242516+0000","last_fullsized":"2024-03-16T19:33:48.242516+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237853+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:33:43.237853+0000","last_peered":"2024-03-16T19:33:43.237853+0000","last_clean":"2024-03-16T19:33:43.237853+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:33:43.237853+0000","last_undegraded":"2024-03-16T19:33:43.237853+0000","last_fullsized":"2024-03-16T19:33:43.237853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233004+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:33:58.233004+0000","last_peered":"2024-03-16T19:33:58.233004+0000","last_clean":"2024-03-16T19:33:58.233004+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:33:58.233004+0000","last_undegraded":"2024-03-16T19:33:58.233004+0000","last_fullsized":"2024-03-16T19:33:58.233004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231221+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:33:48.231221+0000","last_peered":"2024-03-16T19:33:48.231221+0000","last_clean":"2024-03-16T19:33:48.231221+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:33:48.231221+0000","last_undegraded":"2024-03-16T19:33:48.231221+0000","last_fullsized":"2024-03-16T19:33:48.231221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695513+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695513+0000","last_peered":"2024-03-16T19:32:31.695513+0000","last_clean":"2024-03-16T19:32:31.695513+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695513+0000","last_undegraded":"2024-03-16T19:32:31.695513+0000","last_fullsized":"2024-03-16T19:32:31.695513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102649+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102649+0000","last_peered":"2024-03-16T19:32:32.102649+0000","last_clean":"2024-03-16T19:32:32.102649+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102649+0000","last_undegraded":"2024-03-16T19:32:32.102649+0000","last_fullsized":"2024-03-16T19:32:32.102649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696352+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696352+0000","last_peered":"2024-03-16T19:32:32.696352+0000","last_clean":"2024-03-16T19:32:32.696352+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696352+0000","last_undegraded":"2024-03-16T19:32:32.696352+0000","last_fullsized":"2024-03-16T19:32:32.696352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177567+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177567+0000","last_peered":"2024-03-16T19:32:35.177567+0000","last_clean":"2024-03-16T19:32:35.177567+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177567+0000","last_undegraded":"2024-03-16T19:32:35.177567+0000","last_fullsized":"2024-03-16T19:32:35.177567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":181,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:33.230775+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:32:33.230775+0000","last_peered":"2024-03-16T19:32:33.230775+0000","last_clean":"2024-03-16T19:32:33.230775+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:32:33.230775+0000","last_undegraded":"2024-03-16T19:32:33.230775+0000","last_fullsized":"2024-03-16T19:32:33.230775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236387+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:33:28.236387+0000","last_peered":"2024-03-16T19:33:28.236387+0000","last_clean":"2024-03-16T19:33:28.236387+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:33:28.236387+0000","last_undegraded":"2024-03-16T19:33:28.236387+0000","last_fullsized":"2024-03-16T19:33:28.236387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236413+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:33:28.236413+0000","last_peered":"2024-03-16T19:33:28.236413+0000","last_clean":"2024-03-16T19:33:28.236413+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:33:28.236413+0000","last_undegraded":"2024-03-16T19:33:28.236413+0000","last_fullsized":"2024-03-16T19:33:28.236413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237534+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:33:58.237534+0000","last_peered":"2024-03-16T19:33:58.237534+0000","last_clean":"2024-03-16T19:33:58.237534+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:33:58.237534+0000","last_undegraded":"2024-03-16T19:33:58.237534+0000","last_fullsized":"2024-03-16T19:33:58.237534+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574940,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574941,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574939,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574940,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122107,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574940,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:36:37.324 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:37.325 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:37.325 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:37.325 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:37.325 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:36:37.387 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:37.408+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:38.028 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:38.049+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:38.391 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:38.412+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:39.018 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:39.038+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:39.420 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:39.441+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:39.977 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:39.998+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:40.412 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:40.433+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:40.991 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:41.012+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:41.386 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:41.407+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:42.070 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:42.058+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:42.415 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:42.435+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:43.001 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:43.022+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:43.428 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:43.449+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:44.011 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:44.032+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:44.380 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:44.401+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:44.997 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:45.019+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:45.366 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:45.388+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:45.976 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:45.998+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:46.338 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:46.360+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:46.993 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:47.015+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:47.321 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:47.343+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:47.989 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:48.011+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:48.352 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:48.374+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:48.944 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:48.966+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:49.349 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:49.371+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:49.912 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:49.934+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:50.333 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:50.355+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:50.900 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:50.922+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:51.298 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:51.320+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:51.853 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:51.875+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:52.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:36:52.082 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-16T19:36:52.324 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:36:52.346+0000 7f3900bdc640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 69054) UID: 0 2024-03-16T19:36:52.324 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:52.346+0000 7fda8712b640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 69054) UID: 0 2024-03-16T19:36:52.324 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:36:52.346+0000 7fbcd8ae2640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 69054) UID: 0 2024-03-16T19:36:52.325 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:52.346+0000 7f1aabbce640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 69054) UID: 0 2024-03-16T19:36:52.334 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:52.356+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:52.889 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:52.911+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:53.357 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:53.379+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:53.936 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:53.958+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:54.401 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:54.423+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:54.969 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:54.991+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:55.382 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:55.404+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:55.981 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:56.003+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:56.336 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:56.358+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:57.008 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:57.030+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:57.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:36:57.360 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:57.383+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:57.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.682+0000 7f969b607640 1 Processor -- start 2024-03-16T19:36:57.660 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.682+0000 7f969b607640 1 -- start start 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.683+0000 7f969b607640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f969406bb10 0x7f96940cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.683+0000 7f969b607640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bef0 0x7f96940cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.683+0000 7f969b607640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96940cc720 con 0x7f96940cc260 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.683+0000 7f969b607640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96940cc9c0 con 0x7f969406bb10 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.683+0000 7f969b607640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969415de30 con 0x7f969406bef0 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.683+0000 7f96993a3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f969406bb10 0x7f96940cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.683+0000 7f9699ba4640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f96940cc260 0x7f9694154660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:38422/0 (socket says 172.21.15.172:38422) 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.683+0000 7f9699ba4640 1 -- 172.21.15.172:0/1078148333 learned_addr learned my addr 172.21.15.172:0/1078148333 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.683+0000 7f9698ba2640 1 --2- 172.21.15.172:0/1078148333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bef0 0x7f96940cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.684+0000 7f9698ba2640 1 -- 172.21.15.172:0/1078148333 >> v2:172.21.15.174:3300/0 conn(0x7f969406bb10 msgr2=0x7f96940cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:36:57.662 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.684+0000 7f9698ba2640 1 --2- 172.21.15.172:0/1078148333 >> v2:172.21.15.174:3300/0 conn(0x7f969406bb10 0x7f96940cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:57.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.684+0000 7f9698ba2640 1 -- 172.21.15.172:0/1078148333 >> v1:172.21.15.174:6789/0 conn(0x7f96940cc260 legacy=0x7f9694154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:57.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.684+0000 7f9698ba2640 1 -- 172.21.15.172:0/1078148333 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969415e110 con 0x7f969406bef0 2024-03-16T19:36:57.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.684+0000 7f968bfff640 1 -- 172.21.15.172:0/1078148333 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848982243 0 0) 0x7f96940cc720 con 0x7f96940cc260 2024-03-16T19:36:57.663 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.684+0000 7f96993a3640 1 --2- 172.21.15.172:0/1078148333 >> v2:172.21.15.174:3300/0 conn(0x7f969406bb10 0x7f96940cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:36:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f9698ba2640 1 --2- 172.21.15.172:0/1078148333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bef0 0x7f96940cb390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f96840097a0 tx=0x7f968402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=208bfb1089f0a8f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f968bfff640 1 -- 172.21.15.172:0/1078148333 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684030610 con 0x7f969406bef0 2024-03-16T19:36:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f968bfff640 1 -- 172.21.15.172:0/1078148333 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9684030770 con 0x7f969406bef0 2024-03-16T19:36:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f968bfff640 1 -- 172.21.15.172:0/1078148333 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684030b00 con 0x7f969406bef0 2024-03-16T19:36:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f969b607640 1 -- 172.21.15.172:0/1078148333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bef0 msgr2=0x7f96940cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f969b607640 1 --2- 172.21.15.172:0/1078148333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bef0 0x7f96940cb390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f96840097a0 tx=0x7f968402b960 comp rx=0 tx=0).stop 2024-03-16T19:36:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f969b607640 1 -- 172.21.15.172:0/1078148333 shutdown_connections 2024-03-16T19:36:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f969b607640 1 --2- 172.21.15.172:0/1078148333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bef0 0x7f96940cb390 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f969b607640 1 --2- 172.21.15.172:0/1078148333 >> v2:172.21.15.174:3300/0 conn(0x7f969406bb10 0x7f96940cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:57.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.685+0000 7f969b607640 1 -- 172.21.15.172:0/1078148333 >> 172.21.15.172:0/1078148333 conn(0x7f9694067250 msgr2=0x7f96940d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:57.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.686+0000 7f969b607640 1 -- 172.21.15.172:0/1078148333 shutdown_connections 2024-03-16T19:36:57.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.686+0000 7f969b607640 1 -- 172.21.15.172:0/1078148333 wait complete. 2024-03-16T19:36:57.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.687+0000 7f969b607640 1 Processor -- start 2024-03-16T19:36:57.665 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.687+0000 7f969b607640 1 -- start start 2024-03-16T19:36:57.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f969b607640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bb10 0x7f969415d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:57.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f969b607640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f969406bef0 0x7f969415c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:57.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f969b607640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96940cc260 0x7f96941483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:57.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f969b607640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941e6730 con 0x7f969406bef0 2024-03-16T19:36:57.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f969b607640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96940b5230 con 0x7f96940cc260 2024-03-16T19:36:57.667 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f969b607640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96940b4f90 con 0x7f969406bb10 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f96993a3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bb10 0x7f969415d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f9698ba2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f969406bef0 0x7f969415c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f9699ba4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96940cc260 0x7f96941483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f96993a3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bb10 0x7f969415d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:38376/0 (socket says 172.21.15.172:38376) 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.688+0000 7f96993a3640 1 -- 172.21.15.172:0/4191949213 learned_addr learned my addr 172.21.15.172:0/4191949213 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f9699ba4640 1 -- 172.21.15.172:0/4191949213 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bb10 msgr2=0x7f969415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f9699ba4640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bb10 0x7f969415d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f9699ba4640 1 -- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f969406bef0 msgr2=0x7f969415c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f9699ba4640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f969406bef0 0x7f969415c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f9699ba4640 1 -- 172.21.15.172:0/4191949213 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9690009400 con 0x7f96940cc260 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f9698ba2640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f969406bef0 0x7f969415c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f9699ba4640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96940cc260 0x7f96941483a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f968c00c750 tx=0x7f968c00cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f9689ffb640 1 -- 172.21.15.172:0/4191949213 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c004d60 con 0x7f96940cc260 2024-03-16T19:36:57.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f9689ffb640 1 -- 172.21.15.172:0/4191949213 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c00a640 con 0x7f96940cc260 2024-03-16T19:36:57.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.689+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9684009400 con 0x7f96940cc260 2024-03-16T19:36:57.669 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.690+0000 7f9689ffb640 1 -- 172.21.15.172:0/4191949213 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c00a9f0 con 0x7f96940cc260 2024-03-16T19:36:57.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.690+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96940b4700 con 0x7f96940cc260 2024-03-16T19:36:57.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.691+0000 7f9689ffb640 1 -- 172.21.15.172:0/4191949213 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f968c00ab50 con 0x7f96940cc260 2024-03-16T19:36:57.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.691+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9694073220 con 0x7f96940cc260 2024-03-16T19:36:57.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.691+0000 7f9689ffb640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f967806b3e0 0x7f967806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:57.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.692+0000 7f9689ffb640 1 -- 172.21.15.172:0/4191949213 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f968c0892a0 con 0x7f96940cc260 2024-03-16T19:36:57.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.692+0000 7f96993a3640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f967806b3e0 0x7f967806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:57.670 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.692+0000 7f96993a3640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f967806b3e0 0x7f967806d8a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9690009dd0 tx=0x7f9690009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:57.674 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.696+0000 7f9689ffb640 1 -- 172.21.15.172:0/4191949213 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f968c057ab0 con 0x7f96940cc260 2024-03-16T19:36:57.949 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.971+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9694150a30 con 0x7f967806b3e0 2024-03-16T19:36:57.956 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.978+0000 7f9689ffb640 1 -- 172.21.15.172:0/4191949213 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252938 (secure 0 0 0) 0x7f9694150a30 con 0x7f967806b3e0 2024-03-16T19:36:57.959 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:36:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:36:57.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f967806b3e0 msgr2=0x7f967806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:57.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f967806b3e0 0x7f967806d8a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9690009dd0 tx=0x7f9690009190 comp rx=0 tx=0).stop 2024-03-16T19:36:57.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96940cc260 msgr2=0x7f96941483a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:57.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96940cc260 0x7f96941483a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f968c00c750 tx=0x7f968c00cc10 comp rx=0 tx=0).stop 2024-03-16T19:36:57.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 shutdown_connections 2024-03-16T19:36:57.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96940cc260 0x7f96941483a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:57.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f969406bef0 0x7f969415c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:57.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f967806b3e0 0x7f967806d8a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:57.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 --2- 172.21.15.172:0/4191949213 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f969406bb10 0x7f969415d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:57.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 >> 172.21.15.172:0/4191949213 conn(0x7f9694067250 msgr2=0x7f9694152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:57.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 shutdown_connections 2024-03-16T19:36:57.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:57.983+0000 7f969b607640 1 -- 172.21.15.172:0/4191949213 wait complete. 2024-03-16T19:36:57.975 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-03-16T19:36:56.288825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002561"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231608+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:33:38.231608+0000","last_peered":"2024-03-16T19:33:38.231608+0000","last_clean":"2024-03-16T19:33:38.231608+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:33:38.231608+0000","last_undegraded":"2024-03-16T19:33:38.231608+0000","last_fullsized":"2024-03-16T19:33:38.231608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241715+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:33:38.241715+0000","last_peered":"2024-03-16T19:33:38.241715+0000","last_clean":"2024-03-16T19:33:38.241715+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:33:38.241715+0000","last_undegraded":"2024-03-16T19:33:38.241715+0000","last_fullsized":"2024-03-16T19:33:38.241715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":114,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.235727+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:33:38.235727+0000","last_peered":"2024-03-16T19:33:38.235727+0000","last_clean":"2024-03-16T19:33:38.235727+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:33:38.235727+0000","last_undegraded":"2024-03-16T19:33:38.235727+0000","last_fullsized":"2024-03-16T19:33:38.235727+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230354+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:33:28.230354+0000","last_peered":"2024-03-16T19:33:28.230354+0000","last_clean":"2024-03-16T19:33:28.230354+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:33:28.230354+0000","last_undegraded":"2024-03-16T19:33:28.230354+0000","last_fullsized":"2024-03-16T19:33:28.230354+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:42.193204+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:32:42.193204+0000","last_peered":"2024-03-16T19:32:42.193204+0000","last_clean":"2024-03-16T19:32:42.193204+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:32:42.193204+0000","last_undegraded":"2024-03-16T19:32:42.193204+0000","last_fullsized":"2024-03-16T19:32:42.193204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:43.501752+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:32:43.501752+0000","last_peered":"2024-03-16T19:32:43.501752+0000","last_clean":"2024-03-16T19:32:43.501752+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:32:43.501752+0000","last_undegraded":"2024-03-16T19:32:43.501752+0000","last_fullsized":"2024-03-16T19:32:43.501752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:45.180916+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:32:45.180916+0000","last_peered":"2024-03-16T19:32:45.180916+0000","last_clean":"2024-03-16T19:32:45.180916+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:32:45.180916+0000","last_undegraded":"2024-03-16T19:32:45.180916+0000","last_fullsized":"2024-03-16T19:32:45.180916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239229+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:33:58.239229+0000","last_peered":"2024-03-16T19:33:58.239229+0000","last_clean":"2024-03-16T19:33:58.239229+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:33:58.239229+0000","last_undegraded":"2024-03-16T19:33:58.239229+0000","last_fullsized":"2024-03-16T19:33:58.239229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234382+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:34:18.234382+0000","last_peered":"2024-03-16T19:34:18.234382+0000","last_clean":"2024-03-16T19:34:18.234382+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:34:18.234382+0000","last_undegraded":"2024-03-16T19:34:18.234382+0000","last_fullsized":"2024-03-16T19:34:18.234382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233484+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:34:18.233484+0000","last_peered":"2024-03-16T19:34:18.233484+0000","last_clean":"2024-03-16T19:34:18.233484+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:34:18.233484+0000","last_undegraded":"2024-03-16T19:34:18.233484+0000","last_fullsized":"2024-03-16T19:34:18.233484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233466+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:34:18.233466+0000","last_peered":"2024-03-16T19:34:18.233466+0000","last_clean":"2024-03-16T19:34:18.233466+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:34:18.233466+0000","last_undegraded":"2024-03-16T19:34:18.233466+0000","last_fullsized":"2024-03-16T19:34:18.233466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234110+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:34:18.234110+0000","last_peered":"2024-03-16T19:34:18.234110+0000","last_clean":"2024-03-16T19:34:18.234110+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:34:18.234110+0000","last_undegraded":"2024-03-16T19:34:18.234110+0000","last_fullsized":"2024-03-16T19:34:18.234110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238872+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:33:53.238872+0000","last_peered":"2024-03-16T19:33:53.238872+0000","last_clean":"2024-03-16T19:33:53.238872+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:33:53.238872+0000","last_undegraded":"2024-03-16T19:33:53.238872+0000","last_fullsized":"2024-03-16T19:33:53.238872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238806+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:34:13.238806+0000","last_peered":"2024-03-16T19:34:13.238806+0000","last_clean":"2024-03-16T19:34:13.238806+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:34:13.238806+0000","last_undegraded":"2024-03-16T19:34:13.238806+0000","last_fullsized":"2024-03-16T19:34:13.238806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.244177+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:34:08.244177+0000","last_peered":"2024-03-16T19:34:08.244177+0000","last_clean":"2024-03-16T19:34:08.244177+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:34:08.244177+0000","last_undegraded":"2024-03-16T19:34:08.244177+0000","last_fullsized":"2024-03-16T19:34:08.244177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233445+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:34:08.233445+0000","last_peered":"2024-03-16T19:34:08.233445+0000","last_clean":"2024-03-16T19:34:08.233445+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:34:08.233445+0000","last_undegraded":"2024-03-16T19:34:08.233445+0000","last_fullsized":"2024-03-16T19:34:08.233445+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233511+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:34:08.233511+0000","last_peered":"2024-03-16T19:34:08.233511+0000","last_clean":"2024-03-16T19:34:08.233511+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:34:08.233511+0000","last_undegraded":"2024-03-16T19:34:08.233511+0000","last_fullsized":"2024-03-16T19:34:08.233511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:07.990588+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:34:07.990588+0000","last_peered":"2024-03-16T19:34:07.990588+0000","last_clean":"2024-03-16T19:34:07.990588+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:34:07.990588+0000","last_undegraded":"2024-03-16T19:34:07.990588+0000","last_fullsized":"2024-03-16T19:34:07.990588+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233025+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:33:58.233025+0000","last_peered":"2024-03-16T19:33:58.233025+0000","last_clean":"2024-03-16T19:33:58.233025+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:33:58.233025+0000","last_undegraded":"2024-03-16T19:33:58.233025+0000","last_fullsized":"2024-03-16T19:33:58.233025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.231988+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:33:58.231988+0000","last_peered":"2024-03-16T19:33:58.231988+0000","last_clean":"2024-03-16T19:33:58.231988+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:33:58.231988+0000","last_undegraded":"2024-03-16T19:33:58.231988+0000","last_fullsized":"2024-03-16T19:33:58.231988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.238290+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:34:08.238290+0000","last_peered":"2024-03-16T19:34:08.238290+0000","last_clean":"2024-03-16T19:34:08.238290+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:34:08.238290+0000","last_undegraded":"2024-03-16T19:34:08.238290+0000","last_fullsized":"2024-03-16T19:34:08.238290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990165+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:34:02.990165+0000","last_peered":"2024-03-16T19:34:02.990165+0000","last_clean":"2024-03-16T19:34:02.990165+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:34:02.990165+0000","last_undegraded":"2024-03-16T19:34:02.990165+0000","last_fullsized":"2024-03-16T19:34:02.990165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237819+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:33:43.237819+0000","last_peered":"2024-03-16T19:33:43.237819+0000","last_clean":"2024-03-16T19:33:43.237819+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:33:43.237819+0000","last_undegraded":"2024-03-16T19:33:43.237819+0000","last_fullsized":"2024-03-16T19:33:43.237819+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231859+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:33:48.231859+0000","last_peered":"2024-03-16T19:33:48.231859+0000","last_clean":"2024-03-16T19:33:48.231859+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:33:48.231859+0000","last_undegraded":"2024-03-16T19:33:48.231859+0000","last_fullsized":"2024-03-16T19:33:48.231859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":173,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.238369+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:33:48.238369+0000","last_peered":"2024-03-16T19:33:48.238369+0000","last_clean":"2024-03-16T19:33:48.238369+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:33:48.238369+0000","last_undegraded":"2024-03-16T19:33:48.238369+0000","last_fullsized":"2024-03-16T19:33:48.238369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231247+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:33:48.231247+0000","last_peered":"2024-03-16T19:33:48.231247+0000","last_clean":"2024-03-16T19:33:48.231247+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:33:48.231247+0000","last_undegraded":"2024-03-16T19:33:48.231247+0000","last_fullsized":"2024-03-16T19:33:48.231247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242533+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:33:48.242533+0000","last_peered":"2024-03-16T19:33:48.242533+0000","last_clean":"2024-03-16T19:33:48.242533+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:33:48.242533+0000","last_undegraded":"2024-03-16T19:33:48.242533+0000","last_fullsized":"2024-03-16T19:33:48.242533+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237366+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:33:38.237366+0000","last_peered":"2024-03-16T19:33:38.237366+0000","last_clean":"2024-03-16T19:33:38.237366+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:33:38.237366+0000","last_undegraded":"2024-03-16T19:33:38.237366+0000","last_fullsized":"2024-03-16T19:33:38.237366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237508+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:33:58.237508+0000","last_peered":"2024-03-16T19:33:58.237508+0000","last_clean":"2024-03-16T19:33:58.237508+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:33:58.237508+0000","last_undegraded":"2024-03-16T19:33:58.237508+0000","last_fullsized":"2024-03-16T19:33:58.237508+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231143+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:33:38.231143+0000","last_peered":"2024-03-16T19:33:38.231143+0000","last_clean":"2024-03-16T19:33:38.231143+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:33:38.231143+0000","last_undegraded":"2024-03-16T19:33:38.231143+0000","last_fullsized":"2024-03-16T19:33:38.231143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":74,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989304+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:33:52.989304+0000","last_peered":"2024-03-16T19:33:52.989304+0000","last_clean":"2024-03-16T19:33:52.989304+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:33:52.989304+0000","last_undegraded":"2024-03-16T19:33:52.989304+0000","last_fullsized":"2024-03-16T19:33:52.989304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.230545+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:33:38.230545+0000","last_peered":"2024-03-16T19:33:38.230545+0000","last_clean":"2024-03-16T19:33:38.230545+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:33:38.230545+0000","last_undegraded":"2024-03-16T19:33:38.230545+0000","last_fullsized":"2024-03-16T19:33:38.230545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:28.002778+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:32:28.002778+0000","last_peered":"2024-03-16T19:32:28.002778+0000","last_clean":"2024-03-16T19:32:28.002778+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:32:28.002778+0000","last_undegraded":"2024-03-16T19:32:28.002778+0000","last_fullsized":"2024-03-16T19:32:28.002778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239257+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:33:58.239257+0000","last_peered":"2024-03-16T19:33:58.239257+0000","last_clean":"2024-03-16T19:33:58.239257+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:33:58.239257+0000","last_undegraded":"2024-03-16T19:33:58.239257+0000","last_fullsized":"2024-03-16T19:33:58.239257+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:23.239683+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:34:23.239683+0000","last_peered":"2024-03-16T19:34:23.239683+0000","last_clean":"2024-03-16T19:34:23.239683+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:34:23.239683+0000","last_undegraded":"2024-03-16T19:34:23.239683+0000","last_fullsized":"2024-03-16T19:34:23.239683+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.913659+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:32:32.913659+0000","last_peered":"2024-03-16T19:32:32.913659+0000","last_clean":"2024-03-16T19:32:32.913659+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:32:32.913659+0000","last_undegraded":"2024-03-16T19:32:32.913659+0000","last_fullsized":"2024-03-16T19:32:32.913659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238854+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:33:53.238854+0000","last_peered":"2024-03-16T19:33:53.238854+0000","last_clean":"2024-03-16T19:33:53.238854+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:33:53.238854+0000","last_undegraded":"2024-03-16T19:33:53.238854+0000","last_fullsized":"2024-03-16T19:33:53.238854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238825+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:33:53.238825+0000","last_peered":"2024-03-16T19:33:53.238825+0000","last_clean":"2024-03-16T19:33:53.238825+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:33:53.238825+0000","last_undegraded":"2024-03-16T19:33:53.238825+0000","last_fullsized":"2024-03-16T19:33:53.238825+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234171+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:34:18.234171+0000","last_peered":"2024-03-16T19:34:18.234171+0000","last_clean":"2024-03-16T19:34:18.234171+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:34:18.234171+0000","last_undegraded":"2024-03-16T19:34:18.234171+0000","last_fullsized":"2024-03-16T19:34:18.234171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.244981+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:34:18.244981+0000","last_peered":"2024-03-16T19:34:18.244981+0000","last_clean":"2024-03-16T19:34:18.244981+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:34:18.244981+0000","last_undegraded":"2024-03-16T19:34:18.244981+0000","last_fullsized":"2024-03-16T19:34:18.244981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233493+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:34:18.233493+0000","last_peered":"2024-03-16T19:34:18.233493+0000","last_clean":"2024-03-16T19:34:18.233493+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:34:18.233493+0000","last_undegraded":"2024-03-16T19:34:18.233493+0000","last_fullsized":"2024-03-16T19:34:18.233493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234208+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:34:18.234208+0000","last_peered":"2024-03-16T19:34:18.234208+0000","last_clean":"2024-03-16T19:34:18.234208+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:34:18.234208+0000","last_undegraded":"2024-03-16T19:34:18.234208+0000","last_fullsized":"2024-03-16T19:34:18.234208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234218+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:34:18.234218+0000","last_peered":"2024-03-16T19:34:18.234218+0000","last_clean":"2024-03-16T19:34:18.234218+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:34:18.234218+0000","last_undegraded":"2024-03-16T19:34:18.234218+0000","last_fullsized":"2024-03-16T19:34:18.234218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.239240+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:34:18.239240+0000","last_peered":"2024-03-16T19:34:18.239240+0000","last_clean":"2024-03-16T19:34:18.239240+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:34:18.239240+0000","last_undegraded":"2024-03-16T19:34:18.239240+0000","last_fullsized":"2024-03-16T19:34:18.239240+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.245012+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:34:18.245012+0000","last_peered":"2024-03-16T19:34:18.245012+0000","last_clean":"2024-03-16T19:34:18.245012+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:34:18.245012+0000","last_undegraded":"2024-03-16T19:34:18.245012+0000","last_fullsized":"2024-03-16T19:34:18.245012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":32,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:27.992368+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:34:27.992368+0000","last_peered":"2024-03-16T19:34:27.992368+0000","last_clean":"2024-03-16T19:34:27.992368+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:34:27.992368+0000","last_undegraded":"2024-03-16T19:34:27.992368+0000","last_fullsized":"2024-03-16T19:34:27.992368+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":248,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232779+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:34:08.232779+0000","last_peered":"2024-03-16T19:34:08.232779+0000","last_clean":"2024-03-16T19:34:08.232779+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:34:08.232779+0000","last_undegraded":"2024-03-16T19:34:08.232779+0000","last_fullsized":"2024-03-16T19:34:08.232779+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234360+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:34:18.234360+0000","last_peered":"2024-03-16T19:34:18.234360+0000","last_clean":"2024-03-16T19:34:18.234360+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:34:18.234360+0000","last_undegraded":"2024-03-16T19:34:18.234360+0000","last_fullsized":"2024-03-16T19:34:18.234360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238829+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:34:13.238829+0000","last_peered":"2024-03-16T19:34:13.238829+0000","last_clean":"2024-03-16T19:34:13.238829+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:34:13.238829+0000","last_undegraded":"2024-03-16T19:34:13.238829+0000","last_fullsized":"2024-03-16T19:34:13.238829+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238778+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:34:13.238778+0000","last_peered":"2024-03-16T19:34:13.238778+0000","last_clean":"2024-03-16T19:34:13.238778+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:34:13.238778+0000","last_undegraded":"2024-03-16T19:34:13.238778+0000","last_fullsized":"2024-03-16T19:34:13.238778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233485+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:34:08.233485+0000","last_peered":"2024-03-16T19:34:08.233485+0000","last_clean":"2024-03-16T19:34:08.233485+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:34:08.233485+0000","last_undegraded":"2024-03-16T19:34:08.233485+0000","last_fullsized":"2024-03-16T19:34:08.233485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233709+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:34:08.233709+0000","last_peered":"2024-03-16T19:34:08.233709+0000","last_clean":"2024-03-16T19:34:08.233709+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:34:08.233709+0000","last_undegraded":"2024-03-16T19:34:08.233709+0000","last_fullsized":"2024-03-16T19:34:08.233709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232766+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:34:08.232766+0000","last_peered":"2024-03-16T19:34:08.232766+0000","last_clean":"2024-03-16T19:34:08.232766+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:34:08.232766+0000","last_undegraded":"2024-03-16T19:34:08.232766+0000","last_fullsized":"2024-03-16T19:34:08.232766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232748+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:34:08.232748+0000","last_peered":"2024-03-16T19:34:08.232748+0000","last_clean":"2024-03-16T19:34:08.232748+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:34:08.232748+0000","last_undegraded":"2024-03-16T19:34:08.232748+0000","last_fullsized":"2024-03-16T19:34:08.232748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":81,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:36:33.004016+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:36:33.004016+0000","last_peered":"2024-03-16T19:36:33.004016+0000","last_clean":"2024-03-16T19:36:33.004016+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:36:33.004016+0000","last_undegraded":"2024-03-16T19:36:33.004016+0000","last_fullsized":"2024-03-16T19:36:33.004016+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 258s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233693+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:34:08.233693+0000","last_peered":"2024-03-16T19:34:08.233693+0000","last_clean":"2024-03-16T19:34:08.233693+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:34:08.233693+0000","last_undegraded":"2024-03-16T19:34:08.233693+0000","last_fullsized":"2024-03-16T19:34:08.233693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232760+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:33:58.232760+0000","last_peered":"2024-03-16T19:33:58.232760+0000","last_clean":"2024-03-16T19:33:58.232760+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:33:58.232760+0000","last_undegraded":"2024-03-16T19:33:58.232760+0000","last_fullsized":"2024-03-16T19:33:58.232760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233035+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:33:58.233035+0000","last_peered":"2024-03-16T19:33:58.233035+0000","last_clean":"2024-03-16T19:33:58.233035+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:33:58.233035+0000","last_undegraded":"2024-03-16T19:33:58.233035+0000","last_fullsized":"2024-03-16T19:33:58.233035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990184+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:34:02.990184+0000","last_peered":"2024-03-16T19:34:02.990184+0000","last_clean":"2024-03-16T19:34:02.990184+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:34:02.990184+0000","last_undegraded":"2024-03-16T19:34:02.990184+0000","last_fullsized":"2024-03-16T19:34:02.990184+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233045+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:33:58.233045+0000","last_peered":"2024-03-16T19:33:58.233045+0000","last_clean":"2024-03-16T19:33:58.233045+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:33:58.233045+0000","last_undegraded":"2024-03-16T19:33:58.233045+0000","last_fullsized":"2024-03-16T19:33:58.233045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.243408+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:33:58.243408+0000","last_peered":"2024-03-16T19:33:58.243408+0000","last_clean":"2024-03-16T19:33:58.243408+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:33:58.243408+0000","last_undegraded":"2024-03-16T19:33:58.243408+0000","last_fullsized":"2024-03-16T19:33:58.243408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232756+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:33:58.232756+0000","last_peered":"2024-03-16T19:33:58.232756+0000","last_clean":"2024-03-16T19:33:58.232756+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:33:58.232756+0000","last_undegraded":"2024-03-16T19:33:58.232756+0000","last_fullsized":"2024-03-16T19:33:58.232756+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990175+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:34:02.990175+0000","last_peered":"2024-03-16T19:34:02.990175+0000","last_clean":"2024-03-16T19:34:02.990175+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:34:02.990175+0000","last_undegraded":"2024-03-16T19:34:02.990175+0000","last_fullsized":"2024-03-16T19:34:02.990175+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":166,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:40.166603+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:32:40.166603+0000","last_peered":"2024-03-16T19:32:40.166603+0000","last_clean":"2024-03-16T19:32:40.166603+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:32:40.166603+0000","last_undegraded":"2024-03-16T19:32:40.166603+0000","last_fullsized":"2024-03-16T19:32:40.166603+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":376,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:18.230159+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:33:18.230159+0000","last_peered":"2024-03-16T19:33:18.230159+0000","last_clean":"2024-03-16T19:33:18.230159+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:33:18.230159+0000","last_undegraded":"2024-03-16T19:33:18.230159+0000","last_fullsized":"2024-03-16T19:33:18.230159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231177+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:33:38.231177+0000","last_peered":"2024-03-16T19:33:38.231177+0000","last_clean":"2024-03-16T19:33:38.231177+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:33:38.231177+0000","last_undegraded":"2024-03-16T19:33:38.231177+0000","last_fullsized":"2024-03-16T19:33:38.231177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:33.236883+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:33:33.236883+0000","last_peered":"2024-03-16T19:33:33.236883+0000","last_clean":"2024-03-16T19:33:33.236883+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:33:33.236883+0000","last_undegraded":"2024-03-16T19:33:33.236883+0000","last_fullsized":"2024-03-16T19:33:33.236883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230834+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:33:28.230834+0000","last_peered":"2024-03-16T19:33:28.230834+0000","last_clean":"2024-03-16T19:33:28.230834+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:33:28.230834+0000","last_undegraded":"2024-03-16T19:33:28.230834+0000","last_fullsized":"2024-03-16T19:33:28.230834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237468+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:33:58.237468+0000","last_peered":"2024-03-16T19:33:58.237468+0000","last_clean":"2024-03-16T19:33:58.237468+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:33:58.237468+0000","last_undegraded":"2024-03-16T19:33:58.237468+0000","last_fullsized":"2024-03-16T19:33:58.237468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989289+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:33:52.989289+0000","last_peered":"2024-03-16T19:33:52.989289+0000","last_clean":"2024-03-16T19:33:52.989289+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:33:52.989289+0000","last_undegraded":"2024-03-16T19:33:52.989289+0000","last_fullsized":"2024-03-16T19:33:52.989289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":208,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:36:38.256858+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:36:38.256858+0000","last_peered":"2024-03-16T19:36:38.256858+0000","last_clean":"2024-03-16T19:36:38.256858+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:36:38.256858+0000","last_undegraded":"2024-03-16T19:36:38.256858+0000","last_fullsized":"2024-03-16T19:36:38.256858+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 257s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231164+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:33:38.231164+0000","last_peered":"2024-03-16T19:33:38.231164+0000","last_clean":"2024-03-16T19:33:38.231164+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:33:38.231164+0000","last_undegraded":"2024-03-16T19:33:38.231164+0000","last_fullsized":"2024-03-16T19:33:38.231164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231343+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:33:38.231343+0000","last_peered":"2024-03-16T19:33:38.231343+0000","last_clean":"2024-03-16T19:33:38.231343+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:33:38.231343+0000","last_undegraded":"2024-03-16T19:33:38.231343+0000","last_fullsized":"2024-03-16T19:33:38.231343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241695+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:33:38.241695+0000","last_peered":"2024-03-16T19:33:38.241695+0000","last_clean":"2024-03-16T19:33:38.241695+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:33:38.241695+0000","last_undegraded":"2024-03-16T19:33:38.241695+0000","last_fullsized":"2024-03-16T19:33:38.241695+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231208+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:33:38.231208+0000","last_peered":"2024-03-16T19:33:38.231208+0000","last_clean":"2024-03-16T19:33:38.231208+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:33:38.231208+0000","last_undegraded":"2024-03-16T19:33:38.231208+0000","last_fullsized":"2024-03-16T19:33:38.231208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231584+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:33:38.231584+0000","last_peered":"2024-03-16T19:33:38.231584+0000","last_clean":"2024-03-16T19:33:38.231584+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:33:38.231584+0000","last_undegraded":"2024-03-16T19:33:38.231584+0000","last_fullsized":"2024-03-16T19:33:38.231584+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":208,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:36:38.256905+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:36:38.256905+0000","last_peered":"2024-03-16T19:36:38.256905+0000","last_clean":"2024-03-16T19:36:38.256905+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:36:38.256905+0000","last_undegraded":"2024-03-16T19:36:38.256905+0000","last_fullsized":"2024-03-16T19:36:38.256905+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 258s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242558+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:33:48.242558+0000","last_peered":"2024-03-16T19:33:48.242558+0000","last_clean":"2024-03-16T19:33:48.242558+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:33:48.242558+0000","last_undegraded":"2024-03-16T19:33:48.242558+0000","last_fullsized":"2024-03-16T19:33:48.242558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237393+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:33:38.237393+0000","last_peered":"2024-03-16T19:33:38.237393+0000","last_clean":"2024-03-16T19:33:38.237393+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:33:38.237393+0000","last_undegraded":"2024-03-16T19:33:38.237393+0000","last_fullsized":"2024-03-16T19:33:38.237393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231258+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:33:48.231258+0000","last_peered":"2024-03-16T19:33:48.231258+0000","last_clean":"2024-03-16T19:33:48.231258+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:33:48.231258+0000","last_undegraded":"2024-03-16T19:33:48.231258+0000","last_fullsized":"2024-03-16T19:33:48.231258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:03.237913+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:34:03.237913+0000","last_peered":"2024-03-16T19:34:03.237913+0000","last_clean":"2024-03-16T19:34:03.237913+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:34:03.237913+0000","last_undegraded":"2024-03-16T19:34:03.237913+0000","last_fullsized":"2024-03-16T19:34:03.237913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232268+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:33:48.232268+0000","last_peered":"2024-03-16T19:33:48.232268+0000","last_clean":"2024-03-16T19:33:48.232268+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:33:48.232268+0000","last_undegraded":"2024-03-16T19:33:48.232268+0000","last_fullsized":"2024-03-16T19:33:48.232268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232070+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:33:48.232070+0000","last_peered":"2024-03-16T19:33:48.232070+0000","last_clean":"2024-03-16T19:33:48.232070+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:33:48.232070+0000","last_undegraded":"2024-03-16T19:33:48.232070+0000","last_fullsized":"2024-03-16T19:33:48.232070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242493+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:33:48.242493+0000","last_peered":"2024-03-16T19:33:48.242493+0000","last_clean":"2024-03-16T19:33:48.242493+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:33:48.242493+0000","last_undegraded":"2024-03-16T19:33:48.242493+0000","last_fullsized":"2024-03-16T19:33:48.242493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:36:38.004527+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:36:38.004527+0000","last_peered":"2024-03-16T19:36:38.004527+0000","last_clean":"2024-03-16T19:36:38.004527+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:36:38.004527+0000","last_undegraded":"2024-03-16T19:36:38.004527+0000","last_fullsized":"2024-03-16T19:36:38.004527+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 258s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242516+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:33:48.242516+0000","last_peered":"2024-03-16T19:33:48.242516+0000","last_clean":"2024-03-16T19:33:48.242516+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:33:48.242516+0000","last_undegraded":"2024-03-16T19:33:48.242516+0000","last_fullsized":"2024-03-16T19:33:48.242516+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237853+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:33:43.237853+0000","last_peered":"2024-03-16T19:33:43.237853+0000","last_clean":"2024-03-16T19:33:43.237853+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:33:43.237853+0000","last_undegraded":"2024-03-16T19:33:43.237853+0000","last_fullsized":"2024-03-16T19:33:43.237853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233004+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:33:58.233004+0000","last_peered":"2024-03-16T19:33:58.233004+0000","last_clean":"2024-03-16T19:33:58.233004+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:33:58.233004+0000","last_undegraded":"2024-03-16T19:33:58.233004+0000","last_fullsized":"2024-03-16T19:33:58.233004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231221+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:33:48.231221+0000","last_peered":"2024-03-16T19:33:48.231221+0000","last_clean":"2024-03-16T19:33:48.231221+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:33:48.231221+0000","last_undegraded":"2024-03-16T19:33:48.231221+0000","last_fullsized":"2024-03-16T19:33:48.231221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:31.695513+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:32:31.695513+0000","last_peered":"2024-03-16T19:32:31.695513+0000","last_clean":"2024-03-16T19:32:31.695513+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:32:31.695513+0000","last_undegraded":"2024-03-16T19:32:31.695513+0000","last_fullsized":"2024-03-16T19:32:31.695513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.102649+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:32:32.102649+0000","last_peered":"2024-03-16T19:32:32.102649+0000","last_clean":"2024-03-16T19:32:32.102649+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:32:32.102649+0000","last_undegraded":"2024-03-16T19:32:32.102649+0000","last_fullsized":"2024-03-16T19:32:32.102649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:32.696352+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:32:32.696352+0000","last_peered":"2024-03-16T19:32:32.696352+0000","last_clean":"2024-03-16T19:32:32.696352+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:32:32.696352+0000","last_undegraded":"2024-03-16T19:32:32.696352+0000","last_fullsized":"2024-03-16T19:32:32.696352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:32:35.177567+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:32:35.177567+0000","last_peered":"2024-03-16T19:32:35.177567+0000","last_clean":"2024-03-16T19:32:35.177567+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:32:35.177567+0000","last_undegraded":"2024-03-16T19:32:35.177567+0000","last_fullsized":"2024-03-16T19:32:35.177567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":182,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:38.241036+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:34:38.241036+0000","last_peered":"2024-03-16T19:34:38.241036+0000","last_clean":"2024-03-16T19:34:38.241036+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:34:38.241036+0000","last_undegraded":"2024-03-16T19:34:38.241036+0000","last_fullsized":"2024-03-16T19:34:38.241036+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236387+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:33:28.236387+0000","last_peered":"2024-03-16T19:33:28.236387+0000","last_clean":"2024-03-16T19:33:28.236387+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:33:28.236387+0000","last_undegraded":"2024-03-16T19:33:28.236387+0000","last_fullsized":"2024-03-16T19:33:28.236387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236413+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:33:28.236413+0000","last_peered":"2024-03-16T19:33:28.236413+0000","last_clean":"2024-03-16T19:33:28.236413+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:33:28.236413+0000","last_undegraded":"2024-03-16T19:33:28.236413+0000","last_fullsized":"2024-03-16T19:33:28.236413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237534+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:33:58.237534+0000","last_peered":"2024-03-16T19:33:58.237534+0000","last_clean":"2024-03-16T19:33:58.237534+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:33:58.237534+0000","last_undegraded":"2024-03-16T19:33:58.237534+0000","last_fullsized":"2024-03-16T19:33:58.237534+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574944,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574945,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574943,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574944,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574945,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122111,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574944,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:36:57.980 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:57.980 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:57.980 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:57.980 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:36:57.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 2.10 deep-scrub 2024-03-16T19:36:58.030 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:58.053+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:58.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.184+0000 7f7a2e781640 1 Processor -- start 2024-03-16T19:36:58.163 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.184+0000 7f7a2e781640 1 -- start start 2024-03-16T19:36:58.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a2e781640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a280c82d0 0x7f7a28150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a2e781640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f7a28157720 0x7f7a280c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a2e781640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a280c7590 con 0x7f7a280c3360 2024-03-16T19:36:58.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a2e781640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a28067330 con 0x7f7a28157720 2024-03-16T19:36:58.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a2e781640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a280c94c0 con 0x7f7a280c82d0 2024-03-16T19:36:58.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a27fff640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f7a280c3360 0x7f7a28066c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:34570/0 (socket says 172.21.15.172:34570) 2024-03-16T19:36:58.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a27fff640 1 -- 172.21.15.172:0/1152914464 learned_addr learned my addr 172.21.15.172:0/1152914464 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:36:58.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a277fe640 1 --2- 172.21.15.172:0/1152914464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a280c82d0 0x7f7a28150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.164 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a2cd1e640 1 --2- 172.21.15.172:0/1152914464 >> v2:172.21.15.172:3300/0 conn(0x7f7a28157720 0x7f7a280c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a26ffd640 1 -- 172.21.15.172:0/1152914464 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780292981 0 0) 0x7f7a280c7590 con 0x7f7a280c3360 2024-03-16T19:36:58.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a26ffd640 1 -- 172.21.15.172:0/1152914464 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a10003520 con 0x7f7a280c3360 2024-03-16T19:36:58.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a277fe640 1 -- 172.21.15.172:0/1152914464 >> v2:172.21.15.172:3300/0 conn(0x7f7a28157720 msgr2=0x7f7a280c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:36:58.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a277fe640 1 --2- 172.21.15.172:0/1152914464 >> v2:172.21.15.172:3300/0 conn(0x7f7a28157720 0x7f7a280c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a277fe640 1 -- 172.21.15.172:0/1152914464 >> v1:172.21.15.172:6789/0 conn(0x7f7a280c3360 legacy=0x7f7a28066c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.165 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.186+0000 7f7a277fe640 1 -- 172.21.15.172:0/1152914464 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a28157b50 con 0x7f7a280c82d0 2024-03-16T19:36:58.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.187+0000 7f7a277fe640 1 --2- 172.21.15.172:0/1152914464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a280c82d0 0x7f7a28150fd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0097a0 tx=0x7f7a1c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c33c58cf50e6f196 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.188+0000 7f7a26ffd640 1 -- 172.21.15.172:0/1152914464 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c02d040 con 0x7f7a280c82d0 2024-03-16T19:36:58.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.188+0000 7f7a26ffd640 1 -- 172.21.15.172:0/1152914464 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c031810 con 0x7f7a280c82d0 2024-03-16T19:36:58.166 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.188+0000 7f7a26ffd640 1 -- 172.21.15.172:0/1152914464 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c031ba0 con 0x7f7a280c82d0 2024-03-16T19:36:58.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.189+0000 7f7a2e781640 1 -- 172.21.15.172:0/1152914464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a280c82d0 msgr2=0x7f7a28150fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.189+0000 7f7a2e781640 1 --2- 172.21.15.172:0/1152914464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a280c82d0 0x7f7a28150fd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0097a0 tx=0x7f7a1c02b8b0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.189+0000 7f7a2e781640 1 -- 172.21.15.172:0/1152914464 shutdown_connections 2024-03-16T19:36:58.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.189+0000 7f7a2e781640 1 --2- 172.21.15.172:0/1152914464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a280c82d0 0x7f7a28150fd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.189+0000 7f7a2e781640 1 --2- 172.21.15.172:0/1152914464 >> v2:172.21.15.172:3300/0 conn(0x7f7a28157720 0x7f7a280c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.189+0000 7f7a2e781640 1 -- 172.21.15.172:0/1152914464 >> 172.21.15.172:0/1152914464 conn(0x7f7a2806d510 msgr2=0x7f7a2806aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:58.167 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.189+0000 7f7a2e781640 1 -- 172.21.15.172:0/1152914464 shutdown_connections 2024-03-16T19:36:58.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.189+0000 7f7a2e781640 1 -- 172.21.15.172:0/1152914464 wait complete. 2024-03-16T19:36:58.168 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.191+0000 7f7a2e781640 1 Processor -- start 2024-03-16T19:36:58.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.191+0000 7f7a2e781640 1 -- start start 2024-03-16T19:36:58.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.192+0000 7f7a2e781640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a280c3360 0x7f7a281f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.192+0000 7f7a2e781640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a280c82d0 0x7f7a28152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.171 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.192+0000 7f7a2e781640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a28157720 0x7f7a28153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.192+0000 7f7a27fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a280c3360 0x7f7a281f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.192+0000 7f7a27fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a280c3360 0x7f7a281f0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49696/0 (socket says 172.21.15.172:49696) 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.192+0000 7f7a277fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a280c82d0 0x7f7a28152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.192+0000 7f7a277fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a280c82d0 0x7f7a28152ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:49222/0 (socket says 172.21.15.172:49222) 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.192+0000 7f7a277fe640 1 -- 172.21.15.172:0/1353214166 learned_addr learned my addr 172.21.15.172:0/1353214166 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.192+0000 7f7a2cd1e640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a28157720 0x7f7a28153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2814def0 con 0x7f7a280c3360 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2814a810 con 0x7f7a280c82d0 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2814acd0 con 0x7f7a28157720 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a277fe640 1 -- 172.21.15.172:0/1353214166 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a28157720 msgr2=0x7f7a28153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a277fe640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a28157720 0x7f7a28153530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a277fe640 1 -- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a280c3360 msgr2=0x7f7a281f0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a277fe640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7a280c3360 0x7f7a281f0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.172 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a277fe640 1 -- 172.21.15.172:0/1353214166 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c009400 con 0x7f7a280c82d0 2024-03-16T19:36:58.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a277fe640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a280c82d0 0x7f7a28152ff0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c02bec0 tx=0x7f7a1c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.193+0000 7f7a24ff9640 1 -- 172.21.15.172:0/1353214166 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c031610 con 0x7f7a280c82d0 2024-03-16T19:36:58.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.194+0000 7f7a24ff9640 1 -- 172.21.15.172:0/1353214166 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c03ad60 con 0x7f7a280c82d0 2024-03-16T19:36:58.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.194+0000 7f7a24ff9640 1 -- 172.21.15.172:0/1353214166 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c042630 con 0x7f7a280c82d0 2024-03-16T19:36:58.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.194+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2814bbb0 con 0x7f7a280c82d0 2024-03-16T19:36:58.173 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.194+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2814a9f0 con 0x7f7a280c82d0 2024-03-16T19:36:58.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.195+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7a28150b30 con 0x7f7a280c82d0 2024-03-16T19:36:58.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.195+0000 7f7a24ff9640 1 -- 172.21.15.172:0/1353214166 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7a1c042790 con 0x7f7a280c82d0 2024-03-16T19:36:58.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.196+0000 7f7a24ff9640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7a1406b3e0 0x7f7a1406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.196+0000 7f7a24ff9640 1 -- 172.21.15.172:0/1353214166 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f7a1c0aba80 con 0x7f7a280c82d0 2024-03-16T19:36:58.174 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.196+0000 7f7a27fff640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7a1406b3e0 0x7f7a1406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.196+0000 7f7a24ff9640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f7a14073840 0x7f7a14075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.196+0000 7f7a24ff9640 1 -- 172.21.15.172:0/1353214166 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 1: {"prefix": "get_command_descriptions", "pgid": "2.10"}) v1 -- 0x7f7a140763d0 con 0x7f7a14073840 2024-03-16T19:36:58.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.196+0000 7f7a24ff9640 1 -- 172.21.15.172:0/1353214166 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a1c0b02a0 con 0x7f7a280c82d0 2024-03-16T19:36:58.175 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.197+0000 7f7a27fff640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7a1406b3e0 0x7f7a1406d8a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7a18005880 tx=0x7f7a18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.200+0000 7f7a2cd1e640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f7a14073840 0x7f7a14075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.178 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.200+0000 7f7a2cd1e640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f7a14073840 0x7f7a14075d00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.179 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.201+0000 7f7a24ff9640 1 -- 172.21.15.172:0/1353214166 <== osd.2 v2:172.21.15.172:6823/3572014952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7a140763d0 con 0x7f7a14073840 2024-03-16T19:36:58.197 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.219+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 2: {"prefix": "deep-scrub", "pgid": "2.10"}) v1 -- 0x7f7a280c9150 con 0x7f7a14073840 2024-03-16T19:36:58.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.226+0000 7f7a24ff9640 1 -- 172.21.15.172:0/1353214166 <== osd.2 v2:172.21.15.172:6823/3572014952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+64 (crc 0 0 0) 0x7f7a20012720 con 0x7f7a14073840 2024-03-16T19:36:58.204 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:36:58.204 INFO:teuthology.orchestra.run.smithi172.stdout: "deep": true, 2024-03-16T19:36:58.204 INFO:teuthology.orchestra.run.smithi172.stdout: "must": true, 2024-03-16T19:36:58.204 INFO:teuthology.orchestra.run.smithi172.stdout: "stamp": "0.000000" 2024-03-16T19:36:58.204 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:36:58.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f7a14073840 msgr2=0x7f7a14075d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f7a14073840 0x7f7a14075d00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7a1406b3e0 msgr2=0x7f7a1406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f7a1406b3e0 0x7f7a1406d8a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7a18005880 tx=0x7f7a18009040 comp rx=0 tx=0).stop 2024-03-16T19:36:58.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a280c82d0 msgr2=0x7f7a28152ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 --2- 172.21.15.172:0/1353214166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a280c82d0 0x7f7a28152ff0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c02bec0 tx=0x7f7a1c0046c0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2cd1e640 1 -- 172.21.15.172:0/1353214166 reap_dead start 2024-03-16T19:36:58.205 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 shutdown_connections 2024-03-16T19:36:58.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 >> 172.21.15.172:0/1353214166 conn(0x7f7a2806d510 msgr2=0x7f7a2814f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:58.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 shutdown_connections 2024-03-16T19:36:58.206 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.227+0000 7f7a2e781640 1 -- 172.21.15.172:0/1353214166 wait complete. 2024-03-16T19:36:58.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 3.0 deep-scrub 2024-03-16T19:36:58.328 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:58.350+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:58.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.423+0000 7f78c3af4640 1 Processor -- start 2024-03-16T19:36:58.401 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.423+0000 7f78c3af4640 1 -- start start 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.424+0000 7f78c3af4640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f78bc158230 0x7f78bc06b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.424+0000 7f78c3af4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc06b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.424+0000 7f78c3af4640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78bc0719e0 con 0x7f78bc157e50 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.424+0000 7f78c3af4640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78bc06aed0 con 0x7f78bc158230 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.424+0000 7f78c3af4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0c7b50 con 0x7f78bc0c9530 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.424+0000 7f78c1890640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f78bc157e50 0x7f78bc0ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:34584/0 (socket says 172.21.15.172:34584) 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.424+0000 7f78c1890640 1 -- 172.21.15.172:0/3327625362 learned_addr learned my addr 172.21.15.172:0/3327625362 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.425+0000 7f78c108f640 1 --2- 172.21.15.172:0/3327625362 >> v2:172.21.15.172:3300/0 conn(0x7f78bc158230 0x7f78bc06b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.425+0000 7f78c2091640 1 --2- 172.21.15.172:0/3327625362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc06b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.425+0000 7f78c2091640 1 -- 172.21.15.172:0/3327625362 >> v2:172.21.15.172:3300/0 conn(0x7f78bc158230 msgr2=0x7f78bc06b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:36:58.403 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.425+0000 7f78c2091640 1 --2- 172.21.15.172:0/3327625362 >> v2:172.21.15.172:3300/0 conn(0x7f78bc158230 0x7f78bc06b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.404 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.425+0000 7f78c2091640 1 -- 172.21.15.172:0/3327625362 >> v1:172.21.15.172:6789/0 conn(0x7f78bc157e50 legacy=0x7f78bc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:36:58.404 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.425+0000 7f78c2091640 1 -- 172.21.15.172:0/3327625362 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc0c7e10 con 0x7f78bc0c9530 2024-03-16T19:36:58.404 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.426+0000 7f78c2091640 1 --2- 172.21.15.172:0/3327625362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc06b860 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009d90 tx=0x7f78b402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8bfc5bbc9a1cb2a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.404 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.426+0000 7f78c088e640 1 -- 172.21.15.172:0/3327625362 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b402cbc0 con 0x7f78bc0c9530 2024-03-16T19:36:58.404 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.426+0000 7f78c088e640 1 -- 172.21.15.172:0/3327625362 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b402cd20 con 0x7f78bc0c9530 2024-03-16T19:36:58.404 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.426+0000 7f78c088e640 1 -- 172.21.15.172:0/3327625362 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4034660 con 0x7f78bc0c9530 2024-03-16T19:36:58.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.427+0000 7f78c3af4640 1 -- 172.21.15.172:0/3327625362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 msgr2=0x7f78bc06b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.427+0000 7f78c3af4640 1 --2- 172.21.15.172:0/3327625362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc06b860 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009d90 tx=0x7f78b402be10 comp rx=0 tx=0).stop 2024-03-16T19:36:58.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.427+0000 7f78c3af4640 1 -- 172.21.15.172:0/3327625362 shutdown_connections 2024-03-16T19:36:58.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.427+0000 7f78c3af4640 1 --2- 172.21.15.172:0/3327625362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc06b860 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.405 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.427+0000 7f78c3af4640 1 --2- 172.21.15.172:0/3327625362 >> v2:172.21.15.172:3300/0 conn(0x7f78bc158230 0x7f78bc06b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.427+0000 7f78c3af4640 1 -- 172.21.15.172:0/3327625362 >> 172.21.15.172:0/3327625362 conn(0x7f78bc06db70 msgr2=0x7f78bc06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:58.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.427+0000 7f78c3af4640 1 -- 172.21.15.172:0/3327625362 shutdown_connections 2024-03-16T19:36:58.406 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.427+0000 7f78c3af4640 1 -- 172.21.15.172:0/3327625362 wait complete. 2024-03-16T19:36:58.407 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.429+0000 7f78c3af4640 1 Processor -- start 2024-03-16T19:36:58.407 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.429+0000 7f78c3af4640 1 -- start start 2024-03-16T19:36:58.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c3af4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc0cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c3af4640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78bc157e50 0x7f78bc0cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c3af4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78bc158230 0x7f78bc155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c1890640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc0cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c3af4640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0cb8f0 con 0x7f78bc157e50 2024-03-16T19:36:58.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c108f640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78bc157e50 0x7f78bc0cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c3af4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc1f3ae0 con 0x7f78bc158230 2024-03-16T19:36:58.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c1890640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc0cbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:38414/0 (socket says 172.21.15.172:38414) 2024-03-16T19:36:58.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc153890 con 0x7f78bc0c9530 2024-03-16T19:36:58.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.430+0000 7f78c1890640 1 -- 172.21.15.172:0/3240846648 learned_addr learned my addr 172.21.15.172:0/3240846648 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:36:58.409 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78c2091640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78bc158230 0x7f78bc155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.410 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78c1890640 1 -- 172.21.15.172:0/3240846648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78bc158230 msgr2=0x7f78bc155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.410 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78c1890640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78bc158230 0x7f78bc155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78c1890640 1 -- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78bc157e50 msgr2=0x7f78bc0cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78c1890640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78bc157e50 0x7f78bc0cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78c1890640 1 -- 172.21.15.172:0/3240846648 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4009a70 con 0x7f78bc0c9530 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78c108f640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78bc157e50 0x7f78bc0cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78c1890640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc0cbaf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c490 tx=0x7f78b800c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78b27fc640 1 -- 172.21.15.172:0/3240846648 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8005ce0 con 0x7f78bc0c9530 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78b27fc640 1 -- 172.21.15.172:0/3240846648 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b8005e60 con 0x7f78bc0c9530 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.431+0000 7f78b27fc640 1 -- 172.21.15.172:0/3240846648 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b80072b0 con 0x7f78bc0c9530 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.432+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78bc1f3d30 con 0x7f78bc0c9530 2024-03-16T19:36:58.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.432+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc1f42f0 con 0x7f78bc0c9530 2024-03-16T19:36:58.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.433+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78bc0c4180 con 0x7f78bc0c9530 2024-03-16T19:36:58.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.433+0000 7f78b27fc640 1 -- 172.21.15.172:0/3240846648 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f78b8007410 con 0x7f78bc0c9530 2024-03-16T19:36:58.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.434+0000 7f78b27fc640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f78a006b480 0x7f78a006d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.434+0000 7f78c108f640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f78a006b480 0x7f78a006d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.434+0000 7f78b27fc640 1 -- 172.21.15.172:0/3240846648 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f78b80896d0 con 0x7f78bc0c9530 2024-03-16T19:36:58.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.434+0000 7f78b27fc640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f78a0073860 0x7f78a0075d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.434+0000 7f78b27fc640 1 -- 172.21.15.172:0/3240846648 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 1: {"prefix": "get_command_descriptions", "pgid": "3.0"}) v1 -- 0x7f78a00763f0 con 0x7f78a0073860 2024-03-16T19:36:58.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.434+0000 7f78b27fc640 1 -- 172.21.15.172:0/3240846648 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78b80541e0 con 0x7f78bc0c9530 2024-03-16T19:36:58.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.435+0000 7f78c108f640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f78a006b480 0x7f78a006d940 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f78bc156c90 tx=0x7f78ac009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.435+0000 7f78c2091640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f78a0073860 0x7f78a0075d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.435+0000 7f78c2091640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f78a0073860 0x7f78a0075d20 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.436+0000 7f78b27fc640 1 -- 172.21.15.172:0/3240846648 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f78a00763f0 con 0x7f78a0073860 2024-03-16T19:36:58.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.461+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 2: {"prefix": "deep-scrub", "pgid": "3.0"}) v1 -- 0x7f78bc076570 con 0x7f78a0073860 2024-03-16T19:36:58.439 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.461+0000 7f78b27fc640 1 -- 172.21.15.172:0/3240846648 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+64 (crc 0 0 0) 0x7f78bc076570 con 0x7f78a0073860 2024-03-16T19:36:58.439 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:36:58.439 INFO:teuthology.orchestra.run.smithi172.stdout: "deep": true, 2024-03-16T19:36:58.440 INFO:teuthology.orchestra.run.smithi172.stdout: "must": true, 2024-03-16T19:36:58.440 INFO:teuthology.orchestra.run.smithi172.stdout: "stamp": "0.000000" 2024-03-16T19:36:58.440 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.462+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f78a0073860 msgr2=0x7f78a0075d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.462+0000 7f78c3af4640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f78a0073860 0x7f78a0075d20 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.462+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f78a006b480 msgr2=0x7f78a006d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.462+0000 7f78c3af4640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f78a006b480 0x7f78a006d940 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f78bc156c90 tx=0x7f78ac009190 comp rx=0 tx=0).stop 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.462+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 msgr2=0x7f78bc0cbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.462+0000 7f78c3af4640 1 --2- 172.21.15.172:0/3240846648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78bc0c9530 0x7f78bc0cbaf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c490 tx=0x7f78b800c950 comp rx=0 tx=0).stop 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.462+0000 7f78c2091640 1 -- 172.21.15.172:0/3240846648 reap_dead start 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.463+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 shutdown_connections 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.463+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 >> 172.21.15.172:0/3240846648 conn(0x7f78bc06db70 msgr2=0x7f78bc06f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.463+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 shutdown_connections 2024-03-16T19:36:58.441 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.463+0000 7f78c3af4640 1 -- 172.21.15.172:0/3240846648 wait complete. 2024-03-16T19:36:58.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 3.4 deep-scrub 2024-03-16T19:36:58.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.616+0000 7f857632c640 1 Processor -- start 2024-03-16T19:36:58.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.616+0000 7f857632c640 1 -- start start 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f857632c640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f8570157ec0 0x7f857015b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f857632c640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85700cba70 con 0x7f857006dc20 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f856f7fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f857006dc20 0x7f85700c9080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:34810/0 (socket says 172.21.15.172:34810) 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f856f7fe640 1 -- 172.21.15.172:0/2062675955 learned_addr learned my addr 172.21.15.172:0/2062675955 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f857632c640 1 -- 172.21.15.172:0/2062675955 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857015b950 con 0x7f857015b540 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f856ffff640 1 --2- 172.21.15.172:0/2062675955 >> v2:172.21.15.172:3300/0 conn(0x7f8570157ec0 0x7f857015b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f857632c640 1 -- 172.21.15.172:0/2062675955 --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85700c9ea0 con 0x7f8570157ec0 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f856effd640 1 -- 172.21.15.172:0/2062675955 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543111158 0 0) 0x7f857015b950 con 0x7f857015b540 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f856effd640 1 -- 172.21.15.172:0/2062675955 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8558003520 con 0x7f857015b540 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f856ffff640 1 -- 172.21.15.172:0/2062675955 >> v1:172.21.15.174:6789/0 conn(0x7f857015b540 legacy=0x7f85700c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f856ffff640 1 -- 172.21.15.172:0/2062675955 >> v1:172.21.15.172:6789/0 conn(0x7f857006dc20 legacy=0x7f85700c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f856ffff640 1 -- 172.21.15.172:0/2062675955 --> v2:172.21.15.172:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857006bfe0 con 0x7f8570157ec0 2024-03-16T19:36:58.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.617+0000 7f856effd640 1 -- 172.21.15.172:0/2062675955 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698423920 0 0) 0x7f85700cba70 con 0x7f857006dc20 2024-03-16T19:36:58.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.618+0000 7f856ffff640 1 --2- 172.21.15.172:0/2062675955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8570157ec0 0x7f857015b000 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f85600097a0 tx=0x7f856002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50687fe8b7e4820 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.618+0000 7f856effd640 1 -- 172.21.15.172:0/2062675955 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856002f810 con 0x7f8570157ec0 2024-03-16T19:36:58.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.619+0000 7f856effd640 1 -- 172.21.15.172:0/2062675955 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856002f970 con 0x7f8570157ec0 2024-03-16T19:36:58.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.619+0000 7f856effd640 1 -- 172.21.15.172:0/2062675955 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856002fd00 con 0x7f8570157ec0 2024-03-16T19:36:58.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.619+0000 7f857632c640 1 -- 172.21.15.172:0/2062675955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8570157ec0 msgr2=0x7f857015b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.619+0000 7f857632c640 1 --2- 172.21.15.172:0/2062675955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8570157ec0 0x7f857015b000 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f85600097a0 tx=0x7f856002b9f0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.620+0000 7f857632c640 1 -- 172.21.15.172:0/2062675955 shutdown_connections 2024-03-16T19:36:58.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.620+0000 7f857632c640 1 --2- 172.21.15.172:0/2062675955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8570157ec0 0x7f857015b000 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.620+0000 7f857632c640 1 -- 172.21.15.172:0/2062675955 >> 172.21.15.172:0/2062675955 conn(0x7f8570070810 msgr2=0x7f8570070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:58.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.620+0000 7f857632c640 1 -- 172.21.15.172:0/2062675955 shutdown_connections 2024-03-16T19:36:58.599 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.620+0000 7f857632c640 1 -- 172.21.15.172:0/2062675955 wait complete. 2024-03-16T19:36:58.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.622+0000 7f857632c640 1 Processor -- start 2024-03-16T19:36:58.600 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.622+0000 7f857632c640 1 -- start start 2024-03-16T19:36:58.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.623+0000 7f857632c640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f857006dc20 0x7f85700c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f857632c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8570157ec0 0x7f8570151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f856ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f857006dc20 0x7f85700c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f857632c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f857015b540 0x7f85701504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f856ffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f857006dc20 0x7f85700c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:50916/0 (socket says 172.21.15.172:50916) 2024-03-16T19:36:58.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f856ffff640 1 -- 172.21.15.172:0/2636343328 learned_addr learned my addr 172.21.15.172:0/2636343328 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f857632c640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85700c7180 con 0x7f857006dc20 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85700b2420 con 0x7f8570157ec0 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85700b2700 con 0x7f857015b540 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f85748c9640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f857015b540 0x7f85701504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f856f7fe640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8570157ec0 0x7f8570151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f85748c9640 1 -- 172.21.15.172:0/2636343328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8570157ec0 msgr2=0x7f8570151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f85748c9640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8570157ec0 0x7f8570151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f85748c9640 1 -- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f857006dc20 msgr2=0x7f85700c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f85748c9640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f857006dc20 0x7f85700c7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f85748c9640 1 -- 172.21.15.172:0/2636343328 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8560009400 con 0x7f857015b540 2024-03-16T19:36:58.604 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.624+0000 7f85748c9640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f857015b540 0x7f85701504d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8568002180 tx=0x7f856800cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.625+0000 7f856cff9640 1 -- 172.21.15.172:0/2636343328 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568003970 con 0x7f857015b540 2024-03-16T19:36:58.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.625+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8570151640 con 0x7f857015b540 2024-03-16T19:36:58.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.625+0000 7f856cff9640 1 -- 172.21.15.172:0/2636343328 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8568003af0 con 0x7f857015b540 2024-03-16T19:36:58.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.625+0000 7f856cff9640 1 -- 172.21.15.172:0/2636343328 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85680050d0 con 0x7f857015b540 2024-03-16T19:36:58.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.625+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8570152380 con 0x7f857015b540 2024-03-16T19:36:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.626+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f85701514c0 con 0x7f857015b540 2024-03-16T19:36:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.627+0000 7f856cff9640 1 -- 172.21.15.172:0/2636343328 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f8568005230 con 0x7f857015b540 2024-03-16T19:36:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.627+0000 7f856cff9640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f855c06b3b0 0x7f855c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.628+0000 7f856cff9640 1 -- 172.21.15.172:0/2636343328 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f856809a840 con 0x7f857015b540 2024-03-16T19:36:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.628+0000 7f856cff9640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f855c0737f0 0x7f855c075cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.628+0000 7f856cff9640 1 -- 172.21.15.172:0/2636343328 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 1: {"prefix": "get_command_descriptions", "pgid": "3.4"}) v1 -- 0x7f855c076380 con 0x7f855c0737f0 2024-03-16T19:36:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.628+0000 7f856cff9640 1 -- 172.21.15.172:0/2636343328 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8568065220 con 0x7f857015b540 2024-03-16T19:36:58.606 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.628+0000 7f856ffff640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f855c06b3b0 0x7f855c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.628+0000 7f856ffff640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f855c06b3b0 0x7f855c06d870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f856002ba30 tx=0x7f8560008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.608 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.631+0000 7f856f7fe640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f855c0737f0 0x7f855c075cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.631+0000 7f856f7fe640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f855c0737f0 0x7f855c075cb0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.610 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.632+0000 7f856cff9640 1 -- 172.21.15.172:0/2636343328 <== osd.1 v2:172.21.15.172:6808/3098410089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f855c076380 con 0x7f855c0737f0 2024-03-16T19:36:58.636 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.658+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 --> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] -- command(tid 2: {"prefix": "deep-scrub", "pgid": "3.4"}) v1 -- 0x7f85700c8da0 con 0x7f855c0737f0 2024-03-16T19:36:58.636 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f856cff9640 1 -- 172.21.15.172:0/2636343328 <== osd.1 v2:172.21.15.172:6808/3098410089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+64 (crc 0 0 0) 0x7f85640071f0 con 0x7f855c0737f0 2024-03-16T19:36:58.637 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:36:58.637 INFO:teuthology.orchestra.run.smithi172.stdout: "deep": true, 2024-03-16T19:36:58.637 INFO:teuthology.orchestra.run.smithi172.stdout: "must": true, 2024-03-16T19:36:58.637 INFO:teuthology.orchestra.run.smithi172.stdout: "stamp": "0.000000" 2024-03-16T19:36:58.637 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:36:58.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f855c0737f0 msgr2=0x7f855c075cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6808/3098410089,v1:172.21.15.172:6809/3098410089] conn(0x7f855c0737f0 0x7f855c075cb0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f855c06b3b0 msgr2=0x7f855c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f855c06b3b0 0x7f855c06d870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f856002ba30 tx=0x7f8560008030 comp rx=0 tx=0).stop 2024-03-16T19:36:58.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f857015b540 msgr2=0x7f85701504d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 --2- 172.21.15.172:0/2636343328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f857015b540 0x7f85701504d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8568002180 tx=0x7f856800cb90 comp rx=0 tx=0).stop 2024-03-16T19:36:58.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f85748c9640 1 -- 172.21.15.172:0/2636343328 reap_dead start 2024-03-16T19:36:58.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 shutdown_connections 2024-03-16T19:36:58.638 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 >> 172.21.15.172:0/2636343328 conn(0x7f8570070810 msgr2=0x7f85700c4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:58.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 shutdown_connections 2024-03-16T19:36:58.639 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.659+0000 7f857632c640 1 -- 172.21.15.172:0/2636343328 wait complete. 2024-03-16T19:36:58.652 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 3.8 deep-scrub 2024-03-16T19:36:58.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.807+0000 7f1eb0f19640 1 Processor -- start 2024-03-16T19:36:58.785 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.807+0000 7f1eb0f19640 1 -- start start 2024-03-16T19:36:58.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eb0f19640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1eac151a00 0x7f1eac0c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eb0f19640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac071e20 0x7f1eac0c5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.786 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eb0f19640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1eac0c5c40 0x7f1eac0c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1ea9d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac071e20 0x7f1eac0c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eaa59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1eac151a00 0x7f1eac0c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eaa59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1eac151a00 0x7f1eac0c9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:34944/0 (socket says 172.21.15.172:34944) 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1ea9d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac071e20 0x7f1eac0c5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:40290/0 (socket says 172.21.15.172:40290) 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eaad9d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1eac0c5c40 0x7f1eac0c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eaad9d640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f1eac0c5c40 0x7f1eac0c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:50918/0 (socket says 172.21.15.172:50918) 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eb0f19640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac0c91b0 con 0x7f1eac0c5c40 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eb0f19640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac0c9470 con 0x7f1eac151a00 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eb0f19640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac0c9750 con 0x7f1eac071e20 2024-03-16T19:36:58.787 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.808+0000 7f1eaa59c640 1 -- 172.21.15.172:0/292621934 learned_addr learned my addr 172.21.15.172:0/292621934 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:36:58.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.809+0000 7f1eaa59c640 1 -- 172.21.15.172:0/292621934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac071e20 msgr2=0x7f1eac0c5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:36:58.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.809+0000 7f1eaa59c640 1 --2- 172.21.15.172:0/292621934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac071e20 0x7f1eac0c5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.809+0000 7f1eaa59c640 1 -- 172.21.15.172:0/292621934 >> v2:172.21.15.172:3300/0 conn(0x7f1eac0c5c40 msgr2=0x7f1eac0c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:36:58.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.809+0000 7f1eaa59c640 1 --2- 172.21.15.172:0/292621934 >> v2:172.21.15.172:3300/0 conn(0x7f1eac0c5c40 0x7f1eac0c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.788 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.809+0000 7f1eaa59c640 1 -- 172.21.15.172:0/292621934 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac0711f0 con 0x7f1eac151a00 2024-03-16T19:36:58.789 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.810+0000 7f1eaa59c640 1 --2- 172.21.15.172:0/292621934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eac151a00 0x7f1eac0c9c60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0c3af0 tx=0x7f1ea002b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de3a1055f3282f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.810+0000 7f1ea959a640 1 -- 172.21.15.172:0/292621934 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0030850 con 0x7f1eac151a00 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.810+0000 7f1ea959a640 1 -- 172.21.15.172:0/292621934 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea00309b0 con 0x7f1eac151a00 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.810+0000 7f1ea959a640 1 -- 172.21.15.172:0/292621934 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0030d40 con 0x7f1eac151a00 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.811+0000 7f1eb0f19640 1 -- 172.21.15.172:0/292621934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eac151a00 msgr2=0x7f1eac0c9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.811+0000 7f1eb0f19640 1 --2- 172.21.15.172:0/292621934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eac151a00 0x7f1eac0c9c60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0c3af0 tx=0x7f1ea002b990 comp rx=0 tx=0).stop 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.811+0000 7f1eb0f19640 1 -- 172.21.15.172:0/292621934 shutdown_connections 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.811+0000 7f1eb0f19640 1 --2- 172.21.15.172:0/292621934 >> v2:172.21.15.172:3300/0 conn(0x7f1eac0c5c40 0x7f1eac0c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.811+0000 7f1eb0f19640 1 --2- 172.21.15.172:0/292621934 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac071e20 0x7f1eac0c5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.811+0000 7f1eb0f19640 1 --2- 172.21.15.172:0/292621934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eac151a00 0x7f1eac0c9c60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.811+0000 7f1eb0f19640 1 -- 172.21.15.172:0/292621934 >> 172.21.15.172:0/292621934 conn(0x7f1eac06d510 msgr2=0x7f1eac06aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.811+0000 7f1eb0f19640 1 -- 172.21.15.172:0/292621934 shutdown_connections 2024-03-16T19:36:58.790 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.811+0000 7f1eb0f19640 1 -- 172.21.15.172:0/292621934 wait complete. 2024-03-16T19:36:58.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.813+0000 7f1eb0f19640 1 Processor -- start 2024-03-16T19:36:58.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.814+0000 7f1eb0f19640 1 -- start start 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1eb0f19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eac071e20 0x7f1eac1dd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1eb0f19640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac0c5c40 0x7f1eac1f12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1eb0f19640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eac151a00 0x7f1eac14dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1eaa59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eac071e20 0x7f1eac1dd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1eaa59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eac071e20 0x7f1eac1dd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:34948/0 (socket says 172.21.15.172:34948) 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1ea9d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac0c5c40 0x7f1eac1f12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1ea9d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac0c5c40 0x7f1eac1f12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:40300/0 (socket says 172.21.15.172:40300) 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1eaad9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eac151a00 0x7f1eac14dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1eaad9d640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eac151a00 0x7f1eac14dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:50934/0 (socket says 172.21.15.172:50934) 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.815+0000 7f1eaa59c640 1 -- 172.21.15.172:0/4173370626 learned_addr learned my addr 172.21.15.172:0/4173370626 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac14ee60 con 0x7f1eac151a00 2024-03-16T19:36:58.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac1de760 con 0x7f1eac071e20 2024-03-16T19:36:58.795 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac1ed350 con 0x7f1eac0c5c40 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1ea9d9b640 1 -- 172.21.15.172:0/4173370626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eac071e20 msgr2=0x7f1eac1dd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1ea9d9b640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eac071e20 0x7f1eac1dd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1ea9d9b640 1 -- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eac151a00 msgr2=0x7f1eac14dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1ea9d9b640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eac151a00 0x7f1eac14dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1ea9d9b640 1 -- 172.21.15.172:0/4173370626 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea00094d0 con 0x7f1eac0c5c40 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1eaad9d640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1eac151a00 0x7f1eac14dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.816+0000 7f1ea9d9b640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac0c5c40 0x7f1eac1f12d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009f40 tx=0x7f1e9c00ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.817+0000 7f1e9affd640 1 -- 172.21.15.172:0/4173370626 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c00ee40 con 0x7f1eac0c5c40 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.817+0000 7f1e9affd640 1 -- 172.21.15.172:0/4173370626 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c00c940 con 0x7f1eac0c5c40 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.817+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac1f1810 con 0x7f1eac0c5c40 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.817+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac14e0f0 con 0x7f1eac0c5c40 2024-03-16T19:36:58.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.818+0000 7f1e9affd640 1 -- 172.21.15.172:0/4173370626 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c00ccf0 con 0x7f1eac0c5c40 2024-03-16T19:36:58.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.819+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e90001050 con 0x7f1eac0c5c40 2024-03-16T19:36:58.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.819+0000 7f1e9affd640 1 -- 172.21.15.172:0/4173370626 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1e9c011450 con 0x7f1eac0c5c40 2024-03-16T19:36:58.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.819+0000 7f1e9affd640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e8c06b3b0 0x7f1e8c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.819+0000 7f1e9affd640 1 -- 172.21.15.172:0/4173370626 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f1e9c08b7a0 con 0x7f1eac0c5c40 2024-03-16T19:36:58.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.820+0000 7f1eaa59c640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e8c06b3b0 0x7f1e8c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.820+0000 7f1e9affd640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f1e8c0736a0 0x7f1e8c075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:36:58.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.820+0000 7f1e9affd640 1 -- 172.21.15.172:0/4173370626 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 1: {"prefix": "get_command_descriptions", "pgid": "3.8"}) v1 -- 0x7f1e8c076230 con 0x7f1e8c0736a0 2024-03-16T19:36:58.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.820+0000 7f1e9affd640 1 -- 172.21.15.172:0/4173370626 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e9c08dd00 con 0x7f1eac0c5c40 2024-03-16T19:36:58.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.820+0000 7f1eaa59c640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e8c06b3b0 0x7f1e8c06d870 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00090b0 tx=0x7f1ea0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.812 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.834+0000 7f1eaad9d640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f1e8c0736a0 0x7f1e8c075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:36:58.815 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.837+0000 7f1eaad9d640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f1e8c0736a0 0x7f1e8c075b60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:36:58.817 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.839+0000 7f1e9affd640 1 -- 172.21.15.172:0/4173370626 <== osd.2 v2:172.21.15.172:6823/3572014952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1e8c076230 con 0x7f1e8c0736a0 2024-03-16T19:36:58.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.864+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 --> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] -- command(tid 2: {"prefix": "deep-scrub", "pgid": "3.8"}) v1 -- 0x7f1e90002d70 con 0x7f1e8c0736a0 2024-03-16T19:36:58.843 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.865+0000 7f1e9affd640 1 -- 172.21.15.172:0/4173370626 <== osd.2 v2:172.21.15.172:6823/3572014952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+64 (crc 0 0 0) 0x7f1e90002d70 con 0x7f1e8c0736a0 2024-03-16T19:36:58.844 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-16T19:36:58.844 INFO:teuthology.orchestra.run.smithi172.stdout: "deep": true, 2024-03-16T19:36:58.844 INFO:teuthology.orchestra.run.smithi172.stdout: "must": true, 2024-03-16T19:36:58.844 INFO:teuthology.orchestra.run.smithi172.stdout: "stamp": "0.000000" 2024-03-16T19:36:58.844 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f1e8c0736a0 msgr2=0x7f1e8c075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6823/3572014952,v1:172.21.15.172:6825/3572014952] conn(0x7f1e8c0736a0 0x7f1e8c075b60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e8c06b3b0 msgr2=0x7f1e8c06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f1e8c06b3b0 0x7f1e8c06d870 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00090b0 tx=0x7f1ea0008030 comp rx=0 tx=0).stop 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac0c5c40 msgr2=0x7f1eac1f12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 --2- 172.21.15.172:0/4173370626 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eac0c5c40 0x7f1eac1f12d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009f40 tx=0x7f1e9c00ec50 comp rx=0 tx=0).stop 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eaad9d640 1 -- 172.21.15.172:0/4173370626 reap_dead start 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 shutdown_connections 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 >> 172.21.15.172:0/4173370626 conn(0x7f1eac06d510 msgr2=0x7f1eac072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 shutdown_connections 2024-03-16T19:36:58.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:36:58.866+0000 7f1eb0f19640 1 -- 172.21.15.172:0/4173370626 wait complete. 2024-03-16T19:36:58.860 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:36:59.068 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:36:59.091+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:36:59.308 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:36:59.330+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:00.069 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:00.091+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:00.321 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:00.343+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:01.062 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:01.084+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:01.356 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:01.378+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:02.029 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:02.051+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:02.365 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:02.387+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:03.029 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:03.052+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:03.340 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:03.362+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:04.037 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:04.059+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:04.312 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:04.334+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:04.991 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:05.013+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:05.347 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:05.369+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:06.031 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:06.053+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:06.318 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:06.340+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:07.058 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:07.080+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:07.293 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:07.315+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:08.060 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:08.082+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:08.341 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:08.363+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:09.093 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:09.114+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:09.391 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:09.412+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:10.095 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:10.116+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:10.361 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:10.382+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:11.058 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:11.079+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:11.333 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:11.355+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:12.067 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:12.088+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:12.355 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:12.376+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:13.035 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:13.056+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:13.340 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:13.361+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:13.992 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:14.014+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:14.390 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:14.411+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:14.955 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:14.976+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:15.362 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:15.383+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:15.936 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:15.957+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:16.355 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:16.376+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:16.982 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:17.003+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:17.308 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:17.329+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:17.992 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:18.013+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:18.263 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:18.284+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:18.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:37:18.959 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:18.980+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:19.278 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:19.300+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:19.923 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:19.945+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:20.293 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:20.315+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:20.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.606+0000 7fc1b6eeb640 1 Processor -- start 2024-03-16T19:37:20.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.606+0000 7fc1b6eeb640 1 -- start start 2024-03-16T19:37:20.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.606+0000 7fc1b6eeb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b016aeb0 0x7fc1b016b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:20.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.606+0000 7fc1b6eeb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc1b00732d0 0x7fc1b01648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:20.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.606+0000 7fc1b6eeb640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc1b00c9650 0x7fc1b0164e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.606+0000 7fc1b6eeb640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b00c25b0 con 0x7fc1b00c9650 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.606+0000 7fc1b6eeb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b006b630 con 0x7fc1b00732d0 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.607+0000 7fc1b6eeb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00c1b70 con 0x7fc1b016aeb0 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.607+0000 7fc1b5488640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc1b00c9650 0x7fc1b0164e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.607+0000 7fc1b5488640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7fc1b00c9650 0x7fc1b0164e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59918/0 (socket says 172.21.15.172:59918) 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.607+0000 7fc1b5488640 1 -- 172.21.15.172:0/2699555278 learned_addr learned my addr 172.21.15.172:0/2699555278 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.607+0000 7fc1affff640 1 --2- 172.21.15.172:0/2699555278 >> v2:172.21.15.174:3300/0 conn(0x7fc1b00732d0 0x7fc1b01648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.607+0000 7fc1b4c87640 1 --2- 172.21.15.172:0/2699555278 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b016aeb0 0x7fc1b016b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.613+0000 7fc1b4c87640 1 -- 172.21.15.172:0/2699555278 >> v2:172.21.15.174:3300/0 conn(0x7fc1b00732d0 msgr2=0x7fc1b01648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.613+0000 7fc1b4c87640 1 --2- 172.21.15.172:0/2699555278 >> v2:172.21.15.174:3300/0 conn(0x7fc1b00732d0 0x7fc1b01648e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.613+0000 7fc1b4c87640 1 -- 172.21.15.172:0/2699555278 >> v2:172.21.15.172:3300/0 conn(0x7fc1b00c9650 msgr2=0x7fc1b0164e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:37:20.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.613+0000 7fc1b4c87640 1 --2- 172.21.15.172:0/2699555278 >> v2:172.21.15.172:3300/0 conn(0x7fc1b00c9650 0x7fc1b0164e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.613+0000 7fc1b4c87640 1 -- 172.21.15.172:0/2699555278 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00c1e50 con 0x7fc1b016aeb0 2024-03-16T19:37:20.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.614+0000 7fc1b4c87640 1 --2- 172.21.15.172:0/2699555278 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b016aeb0 0x7fc1b016b290 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc1b00b22f0 tx=0x7fc1a402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f864b2c5d49ad029 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:37:20.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.614+0000 7fc1af7fe640 1 -- 172.21.15.172:0/2699555278 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4031830 con 0x7fc1b016aeb0 2024-03-16T19:37:20.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.614+0000 7fc1af7fe640 1 -- 172.21.15.172:0/2699555278 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a4031990 con 0x7fc1b016aeb0 2024-03-16T19:37:20.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.614+0000 7fc1af7fe640 1 -- 172.21.15.172:0/2699555278 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4031d20 con 0x7fc1b016aeb0 2024-03-16T19:37:20.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.615+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2699555278 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b016aeb0 msgr2=0x7fc1b016b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:20.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.615+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2699555278 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b016aeb0 0x7fc1b016b290 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc1b00b22f0 tx=0x7fc1a402b8d0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.615+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2699555278 shutdown_connections 2024-03-16T19:37:20.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.615+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2699555278 >> v2:172.21.15.172:3300/0 conn(0x7fc1b00c9650 0x7fc1b0164e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.615+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2699555278 >> v2:172.21.15.174:3300/0 conn(0x7fc1b00732d0 0x7fc1b01648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.615+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2699555278 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b016aeb0 0x7fc1b016b290 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.615+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2699555278 >> 172.21.15.172:0/2699555278 conn(0x7fc1b00c8b30 msgr2=0x7fc1b00c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:37:20.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.615+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2699555278 shutdown_connections 2024-03-16T19:37:20.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.615+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2699555278 wait complete. 2024-03-16T19:37:20.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.617+0000 7fc1b6eeb640 1 Processor -- start 2024-03-16T19:37:20.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.617+0000 7fc1b6eeb640 1 -- start start 2024-03-16T19:37:20.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.618+0000 7fc1b6eeb640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1b00732d0 0x7fc1b0167800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:20.597 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.618+0000 7fc1b4c87640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1b00732d0 0x7fc1b0167800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:20.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.618+0000 7fc1b4c87640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1b00732d0 0x7fc1b0167800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:59926/0 (socket says 172.21.15.172:59926) 2024-03-16T19:37:20.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.618+0000 7fc1b4c87640 1 -- 172.21.15.172:0/2850258931 learned_addr learned my addr 172.21.15.172:0/2850258931 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:37:20.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.618+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b00c9650 0x7fc1b00c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:20.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.618+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1b016aeb0 0x7fc1b006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:20.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.618+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00c3c90 con 0x7fc1b00732d0 2024-03-16T19:37:20.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.618+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b01fd450 con 0x7fc1b016aeb0 2024-03-16T19:37:20.598 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.618+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b01fd980 con 0x7fc1b00c9650 2024-03-16T19:37:20.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1affff640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b00c9650 0x7fc1b00c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:20.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1b5488640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1b016aeb0 0x7fc1b006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:20.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1b5488640 1 -- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b00c9650 msgr2=0x7fc1b00c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:20.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1b5488640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b00c9650 0x7fc1b00c3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1b5488640 1 -- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1b00732d0 msgr2=0x7fc1b0167800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:20.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1b5488640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1b00732d0 0x7fc1b0167800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1b5488640 1 -- 172.21.15.172:0/2850258931 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a4009400 con 0x7fc1b016aeb0 2024-03-16T19:37:20.601 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1affff640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b00c9650 0x7fc1b00c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1b4c87640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1b00732d0 0x7fc1b0167800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1b5488640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1b016aeb0 0x7fc1b006cc60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc1a80076c0 tx=0x7fc1a800e4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1ad7fa640 1 -- 172.21.15.172:0/2850258931 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800a0d0 con 0x7fc1b016aeb0 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1ad7fa640 1 -- 172.21.15.172:0/2850258931 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a800a250 con 0x7fc1b016aeb0 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.619+0000 7fc1ad7fa640 1 -- 172.21.15.172:0/2850258931 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800a600 con 0x7fc1b016aeb0 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.620+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b00c3800 con 0x7fc1b016aeb0 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.620+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b00cba30 con 0x7fc1b016aeb0 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.621+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b00c3990 con 0x7fc1b016aeb0 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.622+0000 7fc1ad7fa640 1 -- 172.21.15.172:0/2850258931 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc1a8008790 con 0x7fc1b016aeb0 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.622+0000 7fc1ad7fa640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc19406b400 0x7fc19406d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.622+0000 7fc1ad7fa640 1 -- 172.21.15.172:0/2850258931 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fc1a8010040 con 0x7fc1b016aeb0 2024-03-16T19:37:20.602 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.622+0000 7fc1b4c87640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc19406b400 0x7fc19406d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:20.603 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.623+0000 7fc1b4c87640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc19406b400 0x7fc19406d8c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc1a402b910 tx=0x7fc1a4045040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:37:20.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.626+0000 7fc1ad7fa640 1 -- 172.21.15.172:0/2850258931 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1a80550e0 con 0x7fc1b016aeb0 2024-03-16T19:37:20.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.917+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1b00c2130 con 0x7fc19406b400 2024-03-16T19:37:20.907 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.928+0000 7fc1ad7fa640 1 -- 172.21.15.172:0/2850258931 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252938 (secure 0 0 0) 0x7fc1a40393b0 con 0x7fc19406b400 2024-03-16T19:37:20.910 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:37:20.912 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:37:20.913 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:20.932+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:20.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc19406b400 msgr2=0x7fc19406d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:20.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc19406b400 0x7fc19406d8c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc1a402b910 tx=0x7fc1a4045040 comp rx=0 tx=0).stop 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1b016aeb0 msgr2=0x7fc1b006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1b016aeb0 0x7fc1b006cc60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc1a80076c0 tx=0x7fc1a800e4f0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 shutdown_connections 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1b016aeb0 0x7fc1b006cc60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fc19406b400 0x7fc19406d8c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1b00c9650 0x7fc1b00c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 --2- 172.21.15.172:0/2850258931 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc1b00732d0 0x7fc1b0167800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 >> 172.21.15.172:0/2850258931 conn(0x7fc1b00c8b30 msgr2=0x7fc1b01606f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 shutdown_connections 2024-03-16T19:37:20.914 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:20.933+0000 7fc1b6eeb640 1 -- 172.21.15.172:0/2850258931 wait complete. 2024-03-16T19:37:20.927 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-03-16T19:37:20.293892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002523"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231608+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:33:38.231608+0000","last_peered":"2024-03-16T19:33:38.231608+0000","last_clean":"2024-03-16T19:33:38.231608+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:33:38.231608+0000","last_undegraded":"2024-03-16T19:33:38.231608+0000","last_fullsized":"2024-03-16T19:33:38.231608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241715+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:33:38.241715+0000","last_peered":"2024-03-16T19:33:38.241715+0000","last_clean":"2024-03-16T19:33:38.241715+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:33:38.241715+0000","last_undegraded":"2024-03-16T19:33:38.241715+0000","last_fullsized":"2024-03-16T19:33:38.241715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":114,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.235727+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:33:38.235727+0000","last_peered":"2024-03-16T19:33:38.235727+0000","last_clean":"2024-03-16T19:33:38.235727+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:33:38.235727+0000","last_undegraded":"2024-03-16T19:33:38.235727+0000","last_fullsized":"2024-03-16T19:33:38.235727+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230354+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:33:28.230354+0000","last_peered":"2024-03-16T19:33:28.230354+0000","last_clean":"2024-03-16T19:33:28.230354+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:33:28.230354+0000","last_undegraded":"2024-03-16T19:33:28.230354+0000","last_fullsized":"2024-03-16T19:33:28.230354+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235065+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:34:28.235065+0000","last_peered":"2024-03-16T19:34:28.235065+0000","last_clean":"2024-03-16T19:34:28.235065+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:34:28.235065+0000","last_undegraded":"2024-03-16T19:34:28.235065+0000","last_fullsized":"2024-03-16T19:34:28.235065+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234972+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:34:28.234972+0000","last_peered":"2024-03-16T19:34:28.234972+0000","last_clean":"2024-03-16T19:34:28.234972+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:34:28.234972+0000","last_undegraded":"2024-03-16T19:34:28.234972+0000","last_fullsized":"2024-03-16T19:34:28.234972+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235046+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:34:28.235046+0000","last_peered":"2024-03-16T19:34:28.235046+0000","last_clean":"2024-03-16T19:34:28.235046+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:34:28.235046+0000","last_undegraded":"2024-03-16T19:34:28.235046+0000","last_fullsized":"2024-03-16T19:34:28.235046+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239229+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:33:58.239229+0000","last_peered":"2024-03-16T19:33:58.239229+0000","last_clean":"2024-03-16T19:33:58.239229+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:33:58.239229+0000","last_undegraded":"2024-03-16T19:33:58.239229+0000","last_fullsized":"2024-03-16T19:33:58.239229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234382+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:34:18.234382+0000","last_peered":"2024-03-16T19:34:18.234382+0000","last_clean":"2024-03-16T19:34:18.234382+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:34:18.234382+0000","last_undegraded":"2024-03-16T19:34:18.234382+0000","last_fullsized":"2024-03-16T19:34:18.234382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233484+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:34:18.233484+0000","last_peered":"2024-03-16T19:34:18.233484+0000","last_clean":"2024-03-16T19:34:18.233484+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:34:18.233484+0000","last_undegraded":"2024-03-16T19:34:18.233484+0000","last_fullsized":"2024-03-16T19:34:18.233484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233466+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:34:18.233466+0000","last_peered":"2024-03-16T19:34:18.233466+0000","last_clean":"2024-03-16T19:34:18.233466+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:34:18.233466+0000","last_undegraded":"2024-03-16T19:34:18.233466+0000","last_fullsized":"2024-03-16T19:34:18.233466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234110+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:34:18.234110+0000","last_peered":"2024-03-16T19:34:18.234110+0000","last_clean":"2024-03-16T19:34:18.234110+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:34:18.234110+0000","last_undegraded":"2024-03-16T19:34:18.234110+0000","last_fullsized":"2024-03-16T19:34:18.234110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238872+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:33:53.238872+0000","last_peered":"2024-03-16T19:33:53.238872+0000","last_clean":"2024-03-16T19:33:53.238872+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:33:53.238872+0000","last_undegraded":"2024-03-16T19:33:53.238872+0000","last_fullsized":"2024-03-16T19:33:53.238872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238806+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:34:13.238806+0000","last_peered":"2024-03-16T19:34:13.238806+0000","last_clean":"2024-03-16T19:34:13.238806+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:34:13.238806+0000","last_undegraded":"2024-03-16T19:34:13.238806+0000","last_fullsized":"2024-03-16T19:34:13.238806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.244177+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:34:08.244177+0000","last_peered":"2024-03-16T19:34:08.244177+0000","last_clean":"2024-03-16T19:34:08.244177+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:34:08.244177+0000","last_undegraded":"2024-03-16T19:34:08.244177+0000","last_fullsized":"2024-03-16T19:34:08.244177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233445+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:34:08.233445+0000","last_peered":"2024-03-16T19:34:08.233445+0000","last_clean":"2024-03-16T19:34:08.233445+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:34:08.233445+0000","last_undegraded":"2024-03-16T19:34:08.233445+0000","last_fullsized":"2024-03-16T19:34:08.233445+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233511+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:34:08.233511+0000","last_peered":"2024-03-16T19:34:08.233511+0000","last_clean":"2024-03-16T19:34:08.233511+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:34:08.233511+0000","last_undegraded":"2024-03-16T19:34:08.233511+0000","last_fullsized":"2024-03-16T19:34:08.233511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:07.990588+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:34:07.990588+0000","last_peered":"2024-03-16T19:34:07.990588+0000","last_clean":"2024-03-16T19:34:07.990588+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:34:07.990588+0000","last_undegraded":"2024-03-16T19:34:07.990588+0000","last_fullsized":"2024-03-16T19:34:07.990588+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233025+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:33:58.233025+0000","last_peered":"2024-03-16T19:33:58.233025+0000","last_clean":"2024-03-16T19:33:58.233025+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:33:58.233025+0000","last_undegraded":"2024-03-16T19:33:58.233025+0000","last_fullsized":"2024-03-16T19:33:58.233025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.231988+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:33:58.231988+0000","last_peered":"2024-03-16T19:33:58.231988+0000","last_clean":"2024-03-16T19:33:58.231988+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:33:58.231988+0000","last_undegraded":"2024-03-16T19:33:58.231988+0000","last_fullsized":"2024-03-16T19:33:58.231988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.238290+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:34:08.238290+0000","last_peered":"2024-03-16T19:34:08.238290+0000","last_clean":"2024-03-16T19:34:08.238290+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:34:08.238290+0000","last_undegraded":"2024-03-16T19:34:08.238290+0000","last_fullsized":"2024-03-16T19:34:08.238290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990165+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:34:02.990165+0000","last_peered":"2024-03-16T19:34:02.990165+0000","last_clean":"2024-03-16T19:34:02.990165+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:34:02.990165+0000","last_undegraded":"2024-03-16T19:34:02.990165+0000","last_fullsized":"2024-03-16T19:34:02.990165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237819+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:33:43.237819+0000","last_peered":"2024-03-16T19:33:43.237819+0000","last_clean":"2024-03-16T19:33:43.237819+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:33:43.237819+0000","last_undegraded":"2024-03-16T19:33:43.237819+0000","last_fullsized":"2024-03-16T19:33:43.237819+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231859+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:33:48.231859+0000","last_peered":"2024-03-16T19:33:48.231859+0000","last_clean":"2024-03-16T19:33:48.231859+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:33:48.231859+0000","last_undegraded":"2024-03-16T19:33:48.231859+0000","last_fullsized":"2024-03-16T19:33:48.231859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":173,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.238369+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:33:48.238369+0000","last_peered":"2024-03-16T19:33:48.238369+0000","last_clean":"2024-03-16T19:33:48.238369+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:33:48.238369+0000","last_undegraded":"2024-03-16T19:33:48.238369+0000","last_fullsized":"2024-03-16T19:33:48.238369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231247+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:33:48.231247+0000","last_peered":"2024-03-16T19:33:48.231247+0000","last_clean":"2024-03-16T19:33:48.231247+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:33:48.231247+0000","last_undegraded":"2024-03-16T19:33:48.231247+0000","last_fullsized":"2024-03-16T19:33:48.231247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242533+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:33:48.242533+0000","last_peered":"2024-03-16T19:33:48.242533+0000","last_clean":"2024-03-16T19:33:48.242533+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:33:48.242533+0000","last_undegraded":"2024-03-16T19:33:48.242533+0000","last_fullsized":"2024-03-16T19:33:48.242533+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237366+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:33:38.237366+0000","last_peered":"2024-03-16T19:33:38.237366+0000","last_clean":"2024-03-16T19:33:38.237366+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:33:38.237366+0000","last_undegraded":"2024-03-16T19:33:38.237366+0000","last_fullsized":"2024-03-16T19:33:38.237366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237508+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:33:58.237508+0000","last_peered":"2024-03-16T19:33:58.237508+0000","last_clean":"2024-03-16T19:33:58.237508+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:33:58.237508+0000","last_undegraded":"2024-03-16T19:33:58.237508+0000","last_fullsized":"2024-03-16T19:33:58.237508+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231143+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:33:38.231143+0000","last_peered":"2024-03-16T19:33:38.231143+0000","last_clean":"2024-03-16T19:33:38.231143+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:33:38.231143+0000","last_undegraded":"2024-03-16T19:33:38.231143+0000","last_fullsized":"2024-03-16T19:33:38.231143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":74,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989304+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:33:52.989304+0000","last_peered":"2024-03-16T19:33:52.989304+0000","last_clean":"2024-03-16T19:33:52.989304+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:33:52.989304+0000","last_undegraded":"2024-03-16T19:33:52.989304+0000","last_fullsized":"2024-03-16T19:33:52.989304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.230545+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:33:38.230545+0000","last_peered":"2024-03-16T19:33:38.230545+0000","last_clean":"2024-03-16T19:33:38.230545+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:33:38.230545+0000","last_undegraded":"2024-03-16T19:33:38.230545+0000","last_fullsized":"2024-03-16T19:33:38.230545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234753+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:34:28.234753+0000","last_peered":"2024-03-16T19:34:28.234753+0000","last_clean":"2024-03-16T19:34:28.234753+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:34:28.234753+0000","last_undegraded":"2024-03-16T19:34:28.234753+0000","last_fullsized":"2024-03-16T19:34:28.234753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239257+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:33:58.239257+0000","last_peered":"2024-03-16T19:33:58.239257+0000","last_clean":"2024-03-16T19:33:58.239257+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:33:58.239257+0000","last_undegraded":"2024-03-16T19:33:58.239257+0000","last_fullsized":"2024-03-16T19:33:58.239257+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:23.239683+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:34:23.239683+0000","last_peered":"2024-03-16T19:34:23.239683+0000","last_clean":"2024-03-16T19:34:23.239683+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:34:23.239683+0000","last_undegraded":"2024-03-16T19:34:23.239683+0000","last_fullsized":"2024-03-16T19:34:23.239683+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234192+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:34:28.234192+0000","last_peered":"2024-03-16T19:34:28.234192+0000","last_clean":"2024-03-16T19:34:28.234192+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:34:28.234192+0000","last_undegraded":"2024-03-16T19:34:28.234192+0000","last_fullsized":"2024-03-16T19:34:28.234192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238854+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:33:53.238854+0000","last_peered":"2024-03-16T19:33:53.238854+0000","last_clean":"2024-03-16T19:33:53.238854+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:33:53.238854+0000","last_undegraded":"2024-03-16T19:33:53.238854+0000","last_fullsized":"2024-03-16T19:33:53.238854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238825+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:33:53.238825+0000","last_peered":"2024-03-16T19:33:53.238825+0000","last_clean":"2024-03-16T19:33:53.238825+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:33:53.238825+0000","last_undegraded":"2024-03-16T19:33:53.238825+0000","last_fullsized":"2024-03-16T19:33:53.238825+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234171+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:34:18.234171+0000","last_peered":"2024-03-16T19:34:18.234171+0000","last_clean":"2024-03-16T19:34:18.234171+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:34:18.234171+0000","last_undegraded":"2024-03-16T19:34:18.234171+0000","last_fullsized":"2024-03-16T19:34:18.234171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.244981+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:34:18.244981+0000","last_peered":"2024-03-16T19:34:18.244981+0000","last_clean":"2024-03-16T19:34:18.244981+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:34:18.244981+0000","last_undegraded":"2024-03-16T19:34:18.244981+0000","last_fullsized":"2024-03-16T19:34:18.244981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233493+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:34:18.233493+0000","last_peered":"2024-03-16T19:34:18.233493+0000","last_clean":"2024-03-16T19:34:18.233493+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:34:18.233493+0000","last_undegraded":"2024-03-16T19:34:18.233493+0000","last_fullsized":"2024-03-16T19:34:18.233493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234208+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:34:18.234208+0000","last_peered":"2024-03-16T19:34:18.234208+0000","last_clean":"2024-03-16T19:34:18.234208+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:34:18.234208+0000","last_undegraded":"2024-03-16T19:34:18.234208+0000","last_fullsized":"2024-03-16T19:34:18.234208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234218+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:34:18.234218+0000","last_peered":"2024-03-16T19:34:18.234218+0000","last_clean":"2024-03-16T19:34:18.234218+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:34:18.234218+0000","last_undegraded":"2024-03-16T19:34:18.234218+0000","last_fullsized":"2024-03-16T19:34:18.234218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.239240+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:34:18.239240+0000","last_peered":"2024-03-16T19:34:18.239240+0000","last_clean":"2024-03-16T19:34:18.239240+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:34:18.239240+0000","last_undegraded":"2024-03-16T19:34:18.239240+0000","last_fullsized":"2024-03-16T19:34:18.239240+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.245012+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:34:18.245012+0000","last_peered":"2024-03-16T19:34:18.245012+0000","last_clean":"2024-03-16T19:34:18.245012+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:34:18.245012+0000","last_undegraded":"2024-03-16T19:34:18.245012+0000","last_fullsized":"2024-03-16T19:34:18.245012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":32,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:27.992368+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:34:27.992368+0000","last_peered":"2024-03-16T19:34:27.992368+0000","last_clean":"2024-03-16T19:34:27.992368+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:34:27.992368+0000","last_undegraded":"2024-03-16T19:34:27.992368+0000","last_fullsized":"2024-03-16T19:34:27.992368+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":248,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232779+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:34:08.232779+0000","last_peered":"2024-03-16T19:34:08.232779+0000","last_clean":"2024-03-16T19:34:08.232779+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:34:08.232779+0000","last_undegraded":"2024-03-16T19:34:08.232779+0000","last_fullsized":"2024-03-16T19:34:08.232779+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234360+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:34:18.234360+0000","last_peered":"2024-03-16T19:34:18.234360+0000","last_clean":"2024-03-16T19:34:18.234360+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:34:18.234360+0000","last_undegraded":"2024-03-16T19:34:18.234360+0000","last_fullsized":"2024-03-16T19:34:18.234360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238829+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:34:13.238829+0000","last_peered":"2024-03-16T19:34:13.238829+0000","last_clean":"2024-03-16T19:34:13.238829+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:34:13.238829+0000","last_undegraded":"2024-03-16T19:34:13.238829+0000","last_fullsized":"2024-03-16T19:34:13.238829+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238778+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:34:13.238778+0000","last_peered":"2024-03-16T19:34:13.238778+0000","last_clean":"2024-03-16T19:34:13.238778+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:34:13.238778+0000","last_undegraded":"2024-03-16T19:34:13.238778+0000","last_fullsized":"2024-03-16T19:34:13.238778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233485+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:34:08.233485+0000","last_peered":"2024-03-16T19:34:08.233485+0000","last_clean":"2024-03-16T19:34:08.233485+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:34:08.233485+0000","last_undegraded":"2024-03-16T19:34:08.233485+0000","last_fullsized":"2024-03-16T19:34:08.233485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233709+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:34:08.233709+0000","last_peered":"2024-03-16T19:34:08.233709+0000","last_clean":"2024-03-16T19:34:08.233709+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:34:08.233709+0000","last_undegraded":"2024-03-16T19:34:08.233709+0000","last_fullsized":"2024-03-16T19:34:08.233709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232766+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:34:08.232766+0000","last_peered":"2024-03-16T19:34:08.232766+0000","last_clean":"2024-03-16T19:34:08.232766+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:34:08.232766+0000","last_undegraded":"2024-03-16T19:34:08.232766+0000","last_fullsized":"2024-03-16T19:34:08.232766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232748+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:34:08.232748+0000","last_peered":"2024-03-16T19:34:08.232748+0000","last_clean":"2024-03-16T19:34:08.232748+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:34:08.232748+0000","last_undegraded":"2024-03-16T19:34:08.232748+0000","last_fullsized":"2024-03-16T19:34:08.232748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":84,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:37:03.006797+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:37:03.006797+0000","last_peered":"2024-03-16T19:37:03.006797+0000","last_clean":"2024-03-16T19:37:03.006797+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:37:03.006797+0000","last_undegraded":"2024-03-16T19:37:03.006797+0000","last_fullsized":"2024-03-16T19:37:03.006797+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 288s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233693+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:34:08.233693+0000","last_peered":"2024-03-16T19:34:08.233693+0000","last_clean":"2024-03-16T19:34:08.233693+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:34:08.233693+0000","last_undegraded":"2024-03-16T19:34:08.233693+0000","last_fullsized":"2024-03-16T19:34:08.233693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232760+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:33:58.232760+0000","last_peered":"2024-03-16T19:33:58.232760+0000","last_clean":"2024-03-16T19:33:58.232760+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:33:58.232760+0000","last_undegraded":"2024-03-16T19:33:58.232760+0000","last_fullsized":"2024-03-16T19:33:58.232760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233035+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:33:58.233035+0000","last_peered":"2024-03-16T19:33:58.233035+0000","last_clean":"2024-03-16T19:33:58.233035+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:33:58.233035+0000","last_undegraded":"2024-03-16T19:33:58.233035+0000","last_fullsized":"2024-03-16T19:33:58.233035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990184+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:34:02.990184+0000","last_peered":"2024-03-16T19:34:02.990184+0000","last_clean":"2024-03-16T19:34:02.990184+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:34:02.990184+0000","last_undegraded":"2024-03-16T19:34:02.990184+0000","last_fullsized":"2024-03-16T19:34:02.990184+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233045+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:33:58.233045+0000","last_peered":"2024-03-16T19:33:58.233045+0000","last_clean":"2024-03-16T19:33:58.233045+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:33:58.233045+0000","last_undegraded":"2024-03-16T19:33:58.233045+0000","last_fullsized":"2024-03-16T19:33:58.233045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.243408+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:33:58.243408+0000","last_peered":"2024-03-16T19:33:58.243408+0000","last_clean":"2024-03-16T19:33:58.243408+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:33:58.243408+0000","last_undegraded":"2024-03-16T19:33:58.243408+0000","last_fullsized":"2024-03-16T19:33:58.243408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232756+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:33:58.232756+0000","last_peered":"2024-03-16T19:33:58.232756+0000","last_clean":"2024-03-16T19:33:58.232756+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:33:58.232756+0000","last_undegraded":"2024-03-16T19:33:58.232756+0000","last_fullsized":"2024-03-16T19:33:58.232756+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990175+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:34:02.990175+0000","last_peered":"2024-03-16T19:34:02.990175+0000","last_clean":"2024-03-16T19:34:02.990175+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:34:02.990175+0000","last_undegraded":"2024-03-16T19:34:02.990175+0000","last_fullsized":"2024-03-16T19:34:02.990175+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":167,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:43.236034+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:34:43.236034+0000","last_peered":"2024-03-16T19:34:43.236034+0000","last_clean":"2024-03-16T19:34:43.236034+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:34:43.236034+0000","last_undegraded":"2024-03-16T19:34:43.236034+0000","last_fullsized":"2024-03-16T19:34:43.236034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":377,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:18.238443+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:35:18.238443+0000","last_peered":"2024-03-16T19:35:18.238443+0000","last_clean":"2024-03-16T19:35:18.238443+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:35:18.238443+0000","last_undegraded":"2024-03-16T19:35:18.238443+0000","last_fullsized":"2024-03-16T19:35:18.238443+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231177+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:33:38.231177+0000","last_peered":"2024-03-16T19:33:38.231177+0000","last_clean":"2024-03-16T19:33:38.231177+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:33:38.231177+0000","last_undegraded":"2024-03-16T19:33:38.231177+0000","last_fullsized":"2024-03-16T19:33:38.231177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:33.236883+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:33:33.236883+0000","last_peered":"2024-03-16T19:33:33.236883+0000","last_clean":"2024-03-16T19:33:33.236883+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:33:33.236883+0000","last_undegraded":"2024-03-16T19:33:33.236883+0000","last_fullsized":"2024-03-16T19:33:33.236883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.230834+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:33:28.230834+0000","last_peered":"2024-03-16T19:33:28.230834+0000","last_clean":"2024-03-16T19:33:28.230834+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:33:28.230834+0000","last_undegraded":"2024-03-16T19:33:28.230834+0000","last_fullsized":"2024-03-16T19:33:28.230834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237468+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:33:58.237468+0000","last_peered":"2024-03-16T19:33:58.237468+0000","last_clean":"2024-03-16T19:33:58.237468+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:33:58.237468+0000","last_undegraded":"2024-03-16T19:33:58.237468+0000","last_fullsized":"2024-03-16T19:33:58.237468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989289+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:33:52.989289+0000","last_peered":"2024-03-16T19:33:52.989289+0000","last_clean":"2024-03-16T19:33:52.989289+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:33:52.989289+0000","last_undegraded":"2024-03-16T19:33:52.989289+0000","last_fullsized":"2024-03-16T19:33:52.989289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":210,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:36:58.462156+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:36:58.462156+0000","last_peered":"2024-03-16T19:36:58.462156+0000","last_clean":"2024-03-16T19:36:58.462156+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:36:58.462156+0000","last_undegraded":"2024-03-16T19:36:58.462156+0000","last_fullsized":"2024-03-16T19:36:58.462156+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 272s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231164+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:33:38.231164+0000","last_peered":"2024-03-16T19:33:38.231164+0000","last_clean":"2024-03-16T19:33:38.231164+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:33:38.231164+0000","last_undegraded":"2024-03-16T19:33:38.231164+0000","last_fullsized":"2024-03-16T19:33:38.231164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231343+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:33:38.231343+0000","last_peered":"2024-03-16T19:33:38.231343+0000","last_clean":"2024-03-16T19:33:38.231343+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:33:38.231343+0000","last_undegraded":"2024-03-16T19:33:38.231343+0000","last_fullsized":"2024-03-16T19:33:38.231343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241695+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:33:38.241695+0000","last_peered":"2024-03-16T19:33:38.241695+0000","last_clean":"2024-03-16T19:33:38.241695+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:33:38.241695+0000","last_undegraded":"2024-03-16T19:33:38.241695+0000","last_fullsized":"2024-03-16T19:33:38.241695+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231208+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:33:38.231208+0000","last_peered":"2024-03-16T19:33:38.231208+0000","last_clean":"2024-03-16T19:33:38.231208+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:33:38.231208+0000","last_undegraded":"2024-03-16T19:33:38.231208+0000","last_fullsized":"2024-03-16T19:33:38.231208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231584+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:33:38.231584+0000","last_peered":"2024-03-16T19:33:38.231584+0000","last_clean":"2024-03-16T19:33:38.231584+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:33:38.231584+0000","last_undegraded":"2024-03-16T19:33:38.231584+0000","last_fullsized":"2024-03-16T19:33:38.231584+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":210,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:36:58.659474+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:36:58.659474+0000","last_peered":"2024-03-16T19:36:58.659474+0000","last_clean":"2024-03-16T19:36:58.659474+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:36:58.659474+0000","last_undegraded":"2024-03-16T19:36:58.659474+0000","last_fullsized":"2024-03-16T19:36:58.659474+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 273s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242558+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:33:48.242558+0000","last_peered":"2024-03-16T19:33:48.242558+0000","last_clean":"2024-03-16T19:33:48.242558+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:33:48.242558+0000","last_undegraded":"2024-03-16T19:33:48.242558+0000","last_fullsized":"2024-03-16T19:33:48.242558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237393+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:33:38.237393+0000","last_peered":"2024-03-16T19:33:38.237393+0000","last_clean":"2024-03-16T19:33:38.237393+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:33:38.237393+0000","last_undegraded":"2024-03-16T19:33:38.237393+0000","last_fullsized":"2024-03-16T19:33:38.237393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231258+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:33:48.231258+0000","last_peered":"2024-03-16T19:33:48.231258+0000","last_clean":"2024-03-16T19:33:48.231258+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:33:48.231258+0000","last_undegraded":"2024-03-16T19:33:48.231258+0000","last_fullsized":"2024-03-16T19:33:48.231258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:03.237913+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:34:03.237913+0000","last_peered":"2024-03-16T19:34:03.237913+0000","last_clean":"2024-03-16T19:34:03.237913+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:34:03.237913+0000","last_undegraded":"2024-03-16T19:34:03.237913+0000","last_fullsized":"2024-03-16T19:34:03.237913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232268+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:33:48.232268+0000","last_peered":"2024-03-16T19:33:48.232268+0000","last_clean":"2024-03-16T19:33:48.232268+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:33:48.232268+0000","last_undegraded":"2024-03-16T19:33:48.232268+0000","last_fullsized":"2024-03-16T19:33:48.232268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232070+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:33:48.232070+0000","last_peered":"2024-03-16T19:33:48.232070+0000","last_clean":"2024-03-16T19:33:48.232070+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:33:48.232070+0000","last_undegraded":"2024-03-16T19:33:48.232070+0000","last_fullsized":"2024-03-16T19:33:48.232070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242493+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:33:48.242493+0000","last_peered":"2024-03-16T19:33:48.242493+0000","last_clean":"2024-03-16T19:33:48.242493+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:33:48.242493+0000","last_undegraded":"2024-03-16T19:33:48.242493+0000","last_fullsized":"2024-03-16T19:33:48.242493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:36:58.866345+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:36:58.866345+0000","last_peered":"2024-03-16T19:36:58.866345+0000","last_clean":"2024-03-16T19:36:58.866345+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:36:58.866345+0000","last_undegraded":"2024-03-16T19:36:58.866345+0000","last_fullsized":"2024-03-16T19:36:58.866345+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 273s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242516+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:33:48.242516+0000","last_peered":"2024-03-16T19:33:48.242516+0000","last_clean":"2024-03-16T19:33:48.242516+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:33:48.242516+0000","last_undegraded":"2024-03-16T19:33:48.242516+0000","last_fullsized":"2024-03-16T19:33:48.242516+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237853+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:33:43.237853+0000","last_peered":"2024-03-16T19:33:43.237853+0000","last_clean":"2024-03-16T19:33:43.237853+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:33:43.237853+0000","last_undegraded":"2024-03-16T19:33:43.237853+0000","last_fullsized":"2024-03-16T19:33:43.237853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233004+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:33:58.233004+0000","last_peered":"2024-03-16T19:33:58.233004+0000","last_clean":"2024-03-16T19:33:58.233004+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:33:58.233004+0000","last_undegraded":"2024-03-16T19:33:58.233004+0000","last_fullsized":"2024-03-16T19:33:58.233004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231221+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:33:48.231221+0000","last_peered":"2024-03-16T19:33:48.231221+0000","last_clean":"2024-03-16T19:33:48.231221+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:33:48.231221+0000","last_undegraded":"2024-03-16T19:33:48.231221+0000","last_fullsized":"2024-03-16T19:33:48.231221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235008+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:34:28.235008+0000","last_peered":"2024-03-16T19:34:28.235008+0000","last_clean":"2024-03-16T19:34:28.235008+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:34:28.235008+0000","last_undegraded":"2024-03-16T19:34:28.235008+0000","last_fullsized":"2024-03-16T19:34:28.235008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235076+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:34:28.235076+0000","last_peered":"2024-03-16T19:34:28.235076+0000","last_clean":"2024-03-16T19:34:28.235076+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:34:28.235076+0000","last_undegraded":"2024-03-16T19:34:28.235076+0000","last_fullsized":"2024-03-16T19:34:28.235076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234995+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:34:28.234995+0000","last_peered":"2024-03-16T19:34:28.234995+0000","last_clean":"2024-03-16T19:34:28.234995+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:34:28.234995+0000","last_undegraded":"2024-03-16T19:34:28.234995+0000","last_fullsized":"2024-03-16T19:34:28.234995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235086+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:34:28.235086+0000","last_peered":"2024-03-16T19:34:28.235086+0000","last_clean":"2024-03-16T19:34:28.235086+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:34:28.235086+0000","last_undegraded":"2024-03-16T19:34:28.235086+0000","last_fullsized":"2024-03-16T19:34:28.235086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":182,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:38.241036+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:34:38.241036+0000","last_peered":"2024-03-16T19:34:38.241036+0000","last_clean":"2024-03-16T19:34:38.241036+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:34:38.241036+0000","last_undegraded":"2024-03-16T19:34:38.241036+0000","last_fullsized":"2024-03-16T19:34:38.241036+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236387+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:33:28.236387+0000","last_peered":"2024-03-16T19:33:28.236387+0000","last_clean":"2024-03-16T19:33:28.236387+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:33:28.236387+0000","last_undegraded":"2024-03-16T19:33:28.236387+0000","last_fullsized":"2024-03-16T19:33:28.236387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:28.236413+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:33:28.236413+0000","last_peered":"2024-03-16T19:33:28.236413+0000","last_clean":"2024-03-16T19:33:28.236413+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:33:28.236413+0000","last_undegraded":"2024-03-16T19:33:28.236413+0000","last_fullsized":"2024-03-16T19:33:28.236413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237534+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:33:58.237534+0000","last_peered":"2024-03-16T19:33:58.237534+0000","last_clean":"2024-03-16T19:33:58.237534+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:33:58.237534+0000","last_undegraded":"2024-03-16T19:33:58.237534+0000","last_fullsized":"2024-03-16T19:33:58.237534+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574949,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574948,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574949,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122116,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574949,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:37:20.932 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:37:20.932 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:37:20.932 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:37:20.932 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:37:20.932 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:37:21.320 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:21.340+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:21.918 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:21.940+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:22.336 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:22.357+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:22.871 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:22.893+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:23.346 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:23.367+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:23.822 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:23.843+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:24.367 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:24.388+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:24.836 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:24.857+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:25.401 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:25.422+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:25.822 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:25.843+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:26.428 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:26.450+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:26.787 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:26.808+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:27.420 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:27.441+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:27.823 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:27.844+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:28.396 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:28.417+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:28.789 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:28.810+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:29.355 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:29.378+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:29.808 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:29.830+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:30.387 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:30.410+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:30.786 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:30.809+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:31.349 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:31.371+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:31.819 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:31.841+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:32.366 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:32.388+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:32.818 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:32.841+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:33.320 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:33.342+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:33.848 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:33.870+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:34.302 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:34.324+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:34.825 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:34.847+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:35.260 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:35.282+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:35.821 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:35.843+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:36.306 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:36.328+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:36.807 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:36.829+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:37.347 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:37.369+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:37.792 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:37.815+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:38.375 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:38.397+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:38.766 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:38.788+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:39.379 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:39.401+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:39.807 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:39.830+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:40.339 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:40.361+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:40.790 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:40.813+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:40.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:37:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.341+0000 7fbcc6313640 1 Processor -- start 2024-03-16T19:37:41.319 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.341+0000 7fbcc6313640 1 -- start start 2024-03-16T19:37:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.342+0000 7fbcc6313640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.342+0000 7fbcc6313640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc01590a0 con 0x7fbcc00cb2b0 2024-03-16T19:37:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.342+0000 7fbcc6313640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc01595d0 con 0x7fbcc00c3610 2024-03-16T19:37:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.342+0000 7fbcc6313640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc00c4260 con 0x7fbcc0152840 2024-03-16T19:37:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.342+0000 7fbcbffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.342+0000 7fbcbffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc006e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.172:48626/0 (socket says 172.21.15.172:48626) 2024-03-16T19:37:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.342+0000 7fbcbffff640 1 -- 172.21.15.172:0/3413529031 learned_addr learned my addr 172.21.15.172:0/3413529031 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:37:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.342+0000 7fbcbffff640 1 -- 172.21.15.172:0/3413529031 >> v1:172.21.15.174:6789/0 conn(0x7fbcc00c3610 legacy=0x7fbcc00caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:41.321 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.343+0000 7fbcbffff640 1 -- 172.21.15.172:0/3413529031 >> v1:172.21.15.172:6789/0 conn(0x7fbcc00cb2b0 legacy=0x7fbcc0162fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:41.322 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.343+0000 7fbcbffff640 1 -- 172.21.15.172:0/3413529031 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc00cbb90 con 0x7fbcc0152840 2024-03-16T19:37:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.344+0000 7fbcbffff640 1 --2- 172.21.15.172:0/3413529031 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc006e330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbcb000f9e0 tx=0x7fbcb002fda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=366bc85f62b1853a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:37:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.344+0000 7fbcbeffd640 1 -- 172.21.15.172:0/3413529031 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0030b50 con 0x7fbcc0152840 2024-03-16T19:37:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.344+0000 7fbcbeffd640 1 -- 172.21.15.172:0/3413529031 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb0030cb0 con 0x7fbcc0152840 2024-03-16T19:37:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.344+0000 7fbcbeffd640 1 -- 172.21.15.172:0/3413529031 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0038660 con 0x7fbcc0152840 2024-03-16T19:37:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.344+0000 7fbcc6313640 1 -- 172.21.15.172:0/3413529031 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 msgr2=0x7fbcc006e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.344+0000 7fbcc6313640 1 --2- 172.21.15.172:0/3413529031 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc006e330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbcb000f9e0 tx=0x7fbcb002fda0 comp rx=0 tx=0).stop 2024-03-16T19:37:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.345+0000 7fbcc6313640 1 -- 172.21.15.172:0/3413529031 shutdown_connections 2024-03-16T19:37:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.345+0000 7fbcc6313640 1 --2- 172.21.15.172:0/3413529031 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc006e330 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.345+0000 7fbcc6313640 1 -- 172.21.15.172:0/3413529031 >> 172.21.15.172:0/3413529031 conn(0x7fbcc0070860 msgr2=0x7fbcc0070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:37:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.345+0000 7fbcc6313640 1 -- 172.21.15.172:0/3413529031 shutdown_connections 2024-03-16T19:37:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.345+0000 7fbcc6313640 1 -- 172.21.15.172:0/3413529031 wait complete. 2024-03-16T19:37:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.347+0000 7fbcc6313640 1 Processor -- start 2024-03-16T19:37:41.324 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.347+0000 7fbcc6313640 1 -- start start 2024-03-16T19:37:41.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.348+0000 7fbcc6313640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcc00c3610 0x7fbcc01509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:41.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.348+0000 7fbcc6313640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcc00cb2b0 0x7fbcc0146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:41.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.348+0000 7fbcbffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcc00c3610 0x7fbcc01509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:41.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.348+0000 7fbcbffff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcc00c3610 0x7fbcc01509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:60936/0 (socket says 172.21.15.172:60936) 2024-03-16T19:37:41.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.348+0000 7fbcbf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcc00cb2b0 0x7fbcc0146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:41.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.348+0000 7fbcbf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcc00cb2b0 0x7fbcc0146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:40314/0 (socket says 172.21.15.172:40314) 2024-03-16T19:37:41.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.348+0000 7fbcc6313640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc0146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:41.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.348+0000 7fbcc6313640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc0147e90 con 0x7fbcc00c3610 2024-03-16T19:37:41.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.348+0000 7fbcbffff640 1 -- 172.21.15.172:0/3027889782 learned_addr learned my addr 172.21.15.172:0/3027889782 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.349+0000 7fbcc48b0640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc0146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.349+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc0151950 con 0x7fbcc00cb2b0 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.349+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc01ec9f0 con 0x7fbcc0152840 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.349+0000 7fbcbffff640 1 -- 172.21.15.172:0/3027889782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 msgr2=0x7fbcc0146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.349+0000 7fbcbffff640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc0146880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.349+0000 7fbcbffff640 1 -- 172.21.15.172:0/3027889782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcc00cb2b0 msgr2=0x7fbcc0146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.349+0000 7fbcbffff640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcc00cb2b0 0x7fbcc0146340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.349+0000 7fbcbffff640 1 -- 172.21.15.172:0/3027889782 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb000f610 con 0x7fbcc00c3610 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.349+0000 7fbcbffff640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcc00c3610 0x7fbcc01509b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0000c00 tx=0x7fbcb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:37:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.350+0000 7fbcbcff9640 1 -- 172.21.15.172:0/3027889782 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0041600 con 0x7fbcc00c3610 2024-03-16T19:37:41.330 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:41.350+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:41.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.350+0000 7fbcbcff9640 1 -- 172.21.15.172:0/3027889782 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb0041760 con 0x7fbcc00c3610 2024-03-16T19:37:41.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.350+0000 7fbcbcff9640 1 -- 172.21.15.172:0/3027889782 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0041af0 con 0x7fbcc00c3610 2024-03-16T19:37:41.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.350+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc0147210 con 0x7fbcc00c3610 2024-03-16T19:37:41.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.350+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc01477d0 con 0x7fbcc00c3610 2024-03-16T19:37:41.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.351+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc00681c0 con 0x7fbcc00c3610 2024-03-16T19:37:41.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.352+0000 7fbcbcff9640 1 -- 172.21.15.172:0/3027889782 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbcb003f040 con 0x7fbcc00c3610 2024-03-16T19:37:41.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.352+0000 7fbcbcff9640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbcac06b3b0 0x7fbcac06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:37:41.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.352+0000 7fbcbcff9640 1 -- 172.21.15.172:0/3027889782 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7fbcb00b0100 con 0x7fbcc00c3610 2024-03-16T19:37:41.331 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.353+0000 7fbcbf7fe640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbcac06b3b0 0x7fbcac06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:37:41.332 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.355+0000 7fbcbf7fe640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbcac06b3b0 0x7fbcac06d870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbcc01e7020 tx=0x7fbcb4007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:37:41.335 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.357+0000 7fbcbcff9640 1 -- 172.21.15.172:0/3027889782 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcb007ac40 con 0x7fbcc00c3610 2024-03-16T19:37:41.637 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.659+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbcc00cb690 con 0x7fbcac06b3b0 2024-03-16T19:37:41.648 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.671+0000 7fbcbcff9640 1 -- 172.21.15.172:0/3027889782 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252938 (secure 0 0 0) 0x7fbcc00cb690 con 0x7fbcac06b3b0 2024-03-16T19:37:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:37:41.652 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:37:41.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbcac06b3b0 msgr2=0x7fbcac06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:41.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbcac06b3b0 0x7fbcac06d870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbcc01e7020 tx=0x7fbcb4007e30 comp rx=0 tx=0).stop 2024-03-16T19:37:41.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcc00c3610 msgr2=0x7fbcc01509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:37:41.654 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcc00c3610 0x7fbcc01509b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0000c00 tx=0x7fbcb00046c0 comp rx=0 tx=0).stop 2024-03-16T19:37:41.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 shutdown_connections 2024-03-16T19:37:41.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7fbcac06b3b0 0x7fbcac06d870 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:41.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcc0152840 0x7fbcc0146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:41.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcc00cb2b0 0x7fbcc0146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:41.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 --2- 172.21.15.172:0/3027889782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbcc00c3610 0x7fbcc01509b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:37:41.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.675+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 >> 172.21.15.172:0/3027889782 conn(0x7fbcc0070860 msgr2=0x7fbcc014fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:37:41.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.676+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 shutdown_connections 2024-03-16T19:37:41.655 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:37:41.676+0000 7fbcc6313640 1 -- 172.21.15.172:0/3027889782 wait complete. 2024-03-16T19:37:41.669 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-03-16T19:37:40.297882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002436"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231608+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:33:38.231608+0000","last_peered":"2024-03-16T19:33:38.231608+0000","last_clean":"2024-03-16T19:33:38.231608+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:33:38.231608+0000","last_undegraded":"2024-03-16T19:33:38.231608+0000","last_fullsized":"2024-03-16T19:33:38.231608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241715+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:33:38.241715+0000","last_peered":"2024-03-16T19:33:38.241715+0000","last_clean":"2024-03-16T19:33:38.241715+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:33:38.241715+0000","last_undegraded":"2024-03-16T19:33:38.241715+0000","last_fullsized":"2024-03-16T19:33:38.241715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":114,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.235727+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:33:38.235727+0000","last_peered":"2024-03-16T19:33:38.235727+0000","last_clean":"2024-03-16T19:33:38.235727+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:33:38.235727+0000","last_undegraded":"2024-03-16T19:33:38.235727+0000","last_fullsized":"2024-03-16T19:33:38.235727+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.238944+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:35:28.238944+0000","last_peered":"2024-03-16T19:35:28.238944+0000","last_clean":"2024-03-16T19:35:28.238944+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:35:28.238944+0000","last_undegraded":"2024-03-16T19:35:28.238944+0000","last_fullsized":"2024-03-16T19:35:28.238944+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235065+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:34:28.235065+0000","last_peered":"2024-03-16T19:34:28.235065+0000","last_clean":"2024-03-16T19:34:28.235065+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:34:28.235065+0000","last_undegraded":"2024-03-16T19:34:28.235065+0000","last_fullsized":"2024-03-16T19:34:28.235065+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234972+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:34:28.234972+0000","last_peered":"2024-03-16T19:34:28.234972+0000","last_clean":"2024-03-16T19:34:28.234972+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:34:28.234972+0000","last_undegraded":"2024-03-16T19:34:28.234972+0000","last_fullsized":"2024-03-16T19:34:28.234972+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235046+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:34:28.235046+0000","last_peered":"2024-03-16T19:34:28.235046+0000","last_clean":"2024-03-16T19:34:28.235046+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:34:28.235046+0000","last_undegraded":"2024-03-16T19:34:28.235046+0000","last_fullsized":"2024-03-16T19:34:28.235046+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239229+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:33:58.239229+0000","last_peered":"2024-03-16T19:33:58.239229+0000","last_clean":"2024-03-16T19:33:58.239229+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:33:58.239229+0000","last_undegraded":"2024-03-16T19:33:58.239229+0000","last_fullsized":"2024-03-16T19:33:58.239229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234382+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:34:18.234382+0000","last_peered":"2024-03-16T19:34:18.234382+0000","last_clean":"2024-03-16T19:34:18.234382+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:34:18.234382+0000","last_undegraded":"2024-03-16T19:34:18.234382+0000","last_fullsized":"2024-03-16T19:34:18.234382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233484+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:34:18.233484+0000","last_peered":"2024-03-16T19:34:18.233484+0000","last_clean":"2024-03-16T19:34:18.233484+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:34:18.233484+0000","last_undegraded":"2024-03-16T19:34:18.233484+0000","last_fullsized":"2024-03-16T19:34:18.233484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233466+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:34:18.233466+0000","last_peered":"2024-03-16T19:34:18.233466+0000","last_clean":"2024-03-16T19:34:18.233466+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:34:18.233466+0000","last_undegraded":"2024-03-16T19:34:18.233466+0000","last_fullsized":"2024-03-16T19:34:18.233466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234110+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:34:18.234110+0000","last_peered":"2024-03-16T19:34:18.234110+0000","last_clean":"2024-03-16T19:34:18.234110+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:34:18.234110+0000","last_undegraded":"2024-03-16T19:34:18.234110+0000","last_fullsized":"2024-03-16T19:34:18.234110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238872+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:33:53.238872+0000","last_peered":"2024-03-16T19:33:53.238872+0000","last_clean":"2024-03-16T19:33:53.238872+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:33:53.238872+0000","last_undegraded":"2024-03-16T19:33:53.238872+0000","last_fullsized":"2024-03-16T19:33:53.238872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238806+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:34:13.238806+0000","last_peered":"2024-03-16T19:34:13.238806+0000","last_clean":"2024-03-16T19:34:13.238806+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:34:13.238806+0000","last_undegraded":"2024-03-16T19:34:13.238806+0000","last_fullsized":"2024-03-16T19:34:13.238806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.244177+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:34:08.244177+0000","last_peered":"2024-03-16T19:34:08.244177+0000","last_clean":"2024-03-16T19:34:08.244177+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:34:08.244177+0000","last_undegraded":"2024-03-16T19:34:08.244177+0000","last_fullsized":"2024-03-16T19:34:08.244177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233445+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:34:08.233445+0000","last_peered":"2024-03-16T19:34:08.233445+0000","last_clean":"2024-03-16T19:34:08.233445+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:34:08.233445+0000","last_undegraded":"2024-03-16T19:34:08.233445+0000","last_fullsized":"2024-03-16T19:34:08.233445+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233511+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:34:08.233511+0000","last_peered":"2024-03-16T19:34:08.233511+0000","last_clean":"2024-03-16T19:34:08.233511+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:34:08.233511+0000","last_undegraded":"2024-03-16T19:34:08.233511+0000","last_fullsized":"2024-03-16T19:34:08.233511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:07.990588+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:34:07.990588+0000","last_peered":"2024-03-16T19:34:07.990588+0000","last_clean":"2024-03-16T19:34:07.990588+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:34:07.990588+0000","last_undegraded":"2024-03-16T19:34:07.990588+0000","last_fullsized":"2024-03-16T19:34:07.990588+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233025+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:33:58.233025+0000","last_peered":"2024-03-16T19:33:58.233025+0000","last_clean":"2024-03-16T19:33:58.233025+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:33:58.233025+0000","last_undegraded":"2024-03-16T19:33:58.233025+0000","last_fullsized":"2024-03-16T19:33:58.233025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.231988+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:33:58.231988+0000","last_peered":"2024-03-16T19:33:58.231988+0000","last_clean":"2024-03-16T19:33:58.231988+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:33:58.231988+0000","last_undegraded":"2024-03-16T19:33:58.231988+0000","last_fullsized":"2024-03-16T19:33:58.231988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.238290+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:34:08.238290+0000","last_peered":"2024-03-16T19:34:08.238290+0000","last_clean":"2024-03-16T19:34:08.238290+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:34:08.238290+0000","last_undegraded":"2024-03-16T19:34:08.238290+0000","last_fullsized":"2024-03-16T19:34:08.238290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990165+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:34:02.990165+0000","last_peered":"2024-03-16T19:34:02.990165+0000","last_clean":"2024-03-16T19:34:02.990165+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:34:02.990165+0000","last_undegraded":"2024-03-16T19:34:02.990165+0000","last_fullsized":"2024-03-16T19:34:02.990165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237819+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:33:43.237819+0000","last_peered":"2024-03-16T19:33:43.237819+0000","last_clean":"2024-03-16T19:33:43.237819+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:33:43.237819+0000","last_undegraded":"2024-03-16T19:33:43.237819+0000","last_fullsized":"2024-03-16T19:33:43.237819+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231859+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:33:48.231859+0000","last_peered":"2024-03-16T19:33:48.231859+0000","last_clean":"2024-03-16T19:33:48.231859+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:33:48.231859+0000","last_undegraded":"2024-03-16T19:33:48.231859+0000","last_fullsized":"2024-03-16T19:33:48.231859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":173,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.238369+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:33:48.238369+0000","last_peered":"2024-03-16T19:33:48.238369+0000","last_clean":"2024-03-16T19:33:48.238369+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:33:48.238369+0000","last_undegraded":"2024-03-16T19:33:48.238369+0000","last_fullsized":"2024-03-16T19:33:48.238369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231247+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:33:48.231247+0000","last_peered":"2024-03-16T19:33:48.231247+0000","last_clean":"2024-03-16T19:33:48.231247+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:33:48.231247+0000","last_undegraded":"2024-03-16T19:33:48.231247+0000","last_fullsized":"2024-03-16T19:33:48.231247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242533+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:33:48.242533+0000","last_peered":"2024-03-16T19:33:48.242533+0000","last_clean":"2024-03-16T19:33:48.242533+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:33:48.242533+0000","last_undegraded":"2024-03-16T19:33:48.242533+0000","last_fullsized":"2024-03-16T19:33:48.242533+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":82,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237366+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:33:38.237366+0000","last_peered":"2024-03-16T19:33:38.237366+0000","last_clean":"2024-03-16T19:33:38.237366+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:33:38.237366+0000","last_undegraded":"2024-03-16T19:33:38.237366+0000","last_fullsized":"2024-03-16T19:33:38.237366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237508+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:33:58.237508+0000","last_peered":"2024-03-16T19:33:58.237508+0000","last_clean":"2024-03-16T19:33:58.237508+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:33:58.237508+0000","last_undegraded":"2024-03-16T19:33:58.237508+0000","last_fullsized":"2024-03-16T19:33:58.237508+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231143+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:33:38.231143+0000","last_peered":"2024-03-16T19:33:38.231143+0000","last_clean":"2024-03-16T19:33:38.231143+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:33:38.231143+0000","last_undegraded":"2024-03-16T19:33:38.231143+0000","last_fullsized":"2024-03-16T19:33:38.231143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":74,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989304+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:33:52.989304+0000","last_peered":"2024-03-16T19:33:52.989304+0000","last_clean":"2024-03-16T19:33:52.989304+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:33:52.989304+0000","last_undegraded":"2024-03-16T19:33:52.989304+0000","last_fullsized":"2024-03-16T19:33:52.989304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.230545+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:33:38.230545+0000","last_peered":"2024-03-16T19:33:38.230545+0000","last_clean":"2024-03-16T19:33:38.230545+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:33:38.230545+0000","last_undegraded":"2024-03-16T19:33:38.230545+0000","last_fullsized":"2024-03-16T19:33:38.230545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234753+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:34:28.234753+0000","last_peered":"2024-03-16T19:34:28.234753+0000","last_clean":"2024-03-16T19:34:28.234753+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:34:28.234753+0000","last_undegraded":"2024-03-16T19:34:28.234753+0000","last_fullsized":"2024-03-16T19:34:28.234753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239257+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:33:58.239257+0000","last_peered":"2024-03-16T19:33:58.239257+0000","last_clean":"2024-03-16T19:33:58.239257+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:33:58.239257+0000","last_undegraded":"2024-03-16T19:33:58.239257+0000","last_fullsized":"2024-03-16T19:33:58.239257+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:23.239683+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:34:23.239683+0000","last_peered":"2024-03-16T19:34:23.239683+0000","last_clean":"2024-03-16T19:34:23.239683+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:34:23.239683+0000","last_undegraded":"2024-03-16T19:34:23.239683+0000","last_fullsized":"2024-03-16T19:34:23.239683+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234192+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:34:28.234192+0000","last_peered":"2024-03-16T19:34:28.234192+0000","last_clean":"2024-03-16T19:34:28.234192+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:34:28.234192+0000","last_undegraded":"2024-03-16T19:34:28.234192+0000","last_fullsized":"2024-03-16T19:34:28.234192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238854+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:33:53.238854+0000","last_peered":"2024-03-16T19:33:53.238854+0000","last_clean":"2024-03-16T19:33:53.238854+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:33:53.238854+0000","last_undegraded":"2024-03-16T19:33:53.238854+0000","last_fullsized":"2024-03-16T19:33:53.238854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238825+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:33:53.238825+0000","last_peered":"2024-03-16T19:33:53.238825+0000","last_clean":"2024-03-16T19:33:53.238825+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:33:53.238825+0000","last_undegraded":"2024-03-16T19:33:53.238825+0000","last_fullsized":"2024-03-16T19:33:53.238825+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234171+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:34:18.234171+0000","last_peered":"2024-03-16T19:34:18.234171+0000","last_clean":"2024-03-16T19:34:18.234171+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:34:18.234171+0000","last_undegraded":"2024-03-16T19:34:18.234171+0000","last_fullsized":"2024-03-16T19:34:18.234171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.244981+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:34:18.244981+0000","last_peered":"2024-03-16T19:34:18.244981+0000","last_clean":"2024-03-16T19:34:18.244981+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:34:18.244981+0000","last_undegraded":"2024-03-16T19:34:18.244981+0000","last_fullsized":"2024-03-16T19:34:18.244981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233493+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:34:18.233493+0000","last_peered":"2024-03-16T19:34:18.233493+0000","last_clean":"2024-03-16T19:34:18.233493+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:34:18.233493+0000","last_undegraded":"2024-03-16T19:34:18.233493+0000","last_fullsized":"2024-03-16T19:34:18.233493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234208+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:34:18.234208+0000","last_peered":"2024-03-16T19:34:18.234208+0000","last_clean":"2024-03-16T19:34:18.234208+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:34:18.234208+0000","last_undegraded":"2024-03-16T19:34:18.234208+0000","last_fullsized":"2024-03-16T19:34:18.234208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234218+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:34:18.234218+0000","last_peered":"2024-03-16T19:34:18.234218+0000","last_clean":"2024-03-16T19:34:18.234218+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:34:18.234218+0000","last_undegraded":"2024-03-16T19:34:18.234218+0000","last_fullsized":"2024-03-16T19:34:18.234218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.239240+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:34:18.239240+0000","last_peered":"2024-03-16T19:34:18.239240+0000","last_clean":"2024-03-16T19:34:18.239240+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:34:18.239240+0000","last_undegraded":"2024-03-16T19:34:18.239240+0000","last_fullsized":"2024-03-16T19:34:18.239240+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.245012+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:34:18.245012+0000","last_peered":"2024-03-16T19:34:18.245012+0000","last_clean":"2024-03-16T19:34:18.245012+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:34:18.245012+0000","last_undegraded":"2024-03-16T19:34:18.245012+0000","last_fullsized":"2024-03-16T19:34:18.245012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":32,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:27.992368+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:34:27.992368+0000","last_peered":"2024-03-16T19:34:27.992368+0000","last_clean":"2024-03-16T19:34:27.992368+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:34:27.992368+0000","last_undegraded":"2024-03-16T19:34:27.992368+0000","last_fullsized":"2024-03-16T19:34:27.992368+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":248,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232779+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:34:08.232779+0000","last_peered":"2024-03-16T19:34:08.232779+0000","last_clean":"2024-03-16T19:34:08.232779+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:34:08.232779+0000","last_undegraded":"2024-03-16T19:34:08.232779+0000","last_fullsized":"2024-03-16T19:34:08.232779+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234360+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:34:18.234360+0000","last_peered":"2024-03-16T19:34:18.234360+0000","last_clean":"2024-03-16T19:34:18.234360+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:34:18.234360+0000","last_undegraded":"2024-03-16T19:34:18.234360+0000","last_fullsized":"2024-03-16T19:34:18.234360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238829+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:34:13.238829+0000","last_peered":"2024-03-16T19:34:13.238829+0000","last_clean":"2024-03-16T19:34:13.238829+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:34:13.238829+0000","last_undegraded":"2024-03-16T19:34:13.238829+0000","last_fullsized":"2024-03-16T19:34:13.238829+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238778+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:34:13.238778+0000","last_peered":"2024-03-16T19:34:13.238778+0000","last_clean":"2024-03-16T19:34:13.238778+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:34:13.238778+0000","last_undegraded":"2024-03-16T19:34:13.238778+0000","last_fullsized":"2024-03-16T19:34:13.238778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233485+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:34:08.233485+0000","last_peered":"2024-03-16T19:34:08.233485+0000","last_clean":"2024-03-16T19:34:08.233485+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:34:08.233485+0000","last_undegraded":"2024-03-16T19:34:08.233485+0000","last_fullsized":"2024-03-16T19:34:08.233485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233709+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:34:08.233709+0000","last_peered":"2024-03-16T19:34:08.233709+0000","last_clean":"2024-03-16T19:34:08.233709+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:34:08.233709+0000","last_undegraded":"2024-03-16T19:34:08.233709+0000","last_fullsized":"2024-03-16T19:34:08.233709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232766+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:34:08.232766+0000","last_peered":"2024-03-16T19:34:08.232766+0000","last_clean":"2024-03-16T19:34:08.232766+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:34:08.232766+0000","last_undegraded":"2024-03-16T19:34:08.232766+0000","last_fullsized":"2024-03-16T19:34:08.232766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232748+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:34:08.232748+0000","last_peered":"2024-03-16T19:34:08.232748+0000","last_clean":"2024-03-16T19:34:08.232748+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:34:08.232748+0000","last_undegraded":"2024-03-16T19:34:08.232748+0000","last_fullsized":"2024-03-16T19:34:08.232748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":85,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:37:18.008285+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:37:18.008285+0000","last_peered":"2024-03-16T19:37:18.008285+0000","last_clean":"2024-03-16T19:37:18.008285+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:37:18.008285+0000","last_undegraded":"2024-03-16T19:37:18.008285+0000","last_fullsized":"2024-03-16T19:37:18.008285+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 303s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233693+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:34:08.233693+0000","last_peered":"2024-03-16T19:34:08.233693+0000","last_clean":"2024-03-16T19:34:08.233693+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:34:08.233693+0000","last_undegraded":"2024-03-16T19:34:08.233693+0000","last_fullsized":"2024-03-16T19:34:08.233693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232760+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:33:58.232760+0000","last_peered":"2024-03-16T19:33:58.232760+0000","last_clean":"2024-03-16T19:33:58.232760+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:33:58.232760+0000","last_undegraded":"2024-03-16T19:33:58.232760+0000","last_fullsized":"2024-03-16T19:33:58.232760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233035+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:33:58.233035+0000","last_peered":"2024-03-16T19:33:58.233035+0000","last_clean":"2024-03-16T19:33:58.233035+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:33:58.233035+0000","last_undegraded":"2024-03-16T19:33:58.233035+0000","last_fullsized":"2024-03-16T19:33:58.233035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990184+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:34:02.990184+0000","last_peered":"2024-03-16T19:34:02.990184+0000","last_clean":"2024-03-16T19:34:02.990184+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:34:02.990184+0000","last_undegraded":"2024-03-16T19:34:02.990184+0000","last_fullsized":"2024-03-16T19:34:02.990184+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233045+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:33:58.233045+0000","last_peered":"2024-03-16T19:33:58.233045+0000","last_clean":"2024-03-16T19:33:58.233045+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:33:58.233045+0000","last_undegraded":"2024-03-16T19:33:58.233045+0000","last_fullsized":"2024-03-16T19:33:58.233045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.243408+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:33:58.243408+0000","last_peered":"2024-03-16T19:33:58.243408+0000","last_clean":"2024-03-16T19:33:58.243408+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:33:58.243408+0000","last_undegraded":"2024-03-16T19:33:58.243408+0000","last_fullsized":"2024-03-16T19:33:58.243408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232756+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:33:58.232756+0000","last_peered":"2024-03-16T19:33:58.232756+0000","last_clean":"2024-03-16T19:33:58.232756+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:33:58.232756+0000","last_undegraded":"2024-03-16T19:33:58.232756+0000","last_fullsized":"2024-03-16T19:33:58.232756+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990175+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:34:02.990175+0000","last_peered":"2024-03-16T19:34:02.990175+0000","last_clean":"2024-03-16T19:34:02.990175+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:34:02.990175+0000","last_undegraded":"2024-03-16T19:34:02.990175+0000","last_fullsized":"2024-03-16T19:34:02.990175+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":167,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:43.236034+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:34:43.236034+0000","last_peered":"2024-03-16T19:34:43.236034+0000","last_clean":"2024-03-16T19:34:43.236034+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:34:43.236034+0000","last_undegraded":"2024-03-16T19:34:43.236034+0000","last_fullsized":"2024-03-16T19:34:43.236034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":377,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:18.238443+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:35:18.238443+0000","last_peered":"2024-03-16T19:35:18.238443+0000","last_clean":"2024-03-16T19:35:18.238443+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:35:18.238443+0000","last_undegraded":"2024-03-16T19:35:18.238443+0000","last_fullsized":"2024-03-16T19:35:18.238443+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231177+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:33:38.231177+0000","last_peered":"2024-03-16T19:33:38.231177+0000","last_clean":"2024-03-16T19:33:38.231177+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:33:38.231177+0000","last_undegraded":"2024-03-16T19:33:38.231177+0000","last_fullsized":"2024-03-16T19:33:38.231177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:33.236883+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:33:33.236883+0000","last_peered":"2024-03-16T19:33:33.236883+0000","last_clean":"2024-03-16T19:33:33.236883+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:33:33.236883+0000","last_undegraded":"2024-03-16T19:33:33.236883+0000","last_fullsized":"2024-03-16T19:33:33.236883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.239079+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:35:28.239079+0000","last_peered":"2024-03-16T19:35:28.239079+0000","last_clean":"2024-03-16T19:35:28.239079+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:35:28.239079+0000","last_undegraded":"2024-03-16T19:35:28.239079+0000","last_fullsized":"2024-03-16T19:35:28.239079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237468+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:33:58.237468+0000","last_peered":"2024-03-16T19:33:58.237468+0000","last_clean":"2024-03-16T19:33:58.237468+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:33:58.237468+0000","last_undegraded":"2024-03-16T19:33:58.237468+0000","last_fullsized":"2024-03-16T19:33:58.237468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:52.989289+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:33:52.989289+0000","last_peered":"2024-03-16T19:33:52.989289+0000","last_clean":"2024-03-16T19:33:52.989289+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:33:52.989289+0000","last_undegraded":"2024-03-16T19:33:52.989289+0000","last_fullsized":"2024-03-16T19:33:52.989289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":212,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:37:23.260748+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:37:23.260748+0000","last_peered":"2024-03-16T19:37:23.260748+0000","last_clean":"2024-03-16T19:37:23.260748+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:37:23.260748+0000","last_undegraded":"2024-03-16T19:37:23.260748+0000","last_fullsized":"2024-03-16T19:37:23.260748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 302s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231164+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:33:38.231164+0000","last_peered":"2024-03-16T19:33:38.231164+0000","last_clean":"2024-03-16T19:33:38.231164+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:33:38.231164+0000","last_undegraded":"2024-03-16T19:33:38.231164+0000","last_fullsized":"2024-03-16T19:33:38.231164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231343+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:33:38.231343+0000","last_peered":"2024-03-16T19:33:38.231343+0000","last_clean":"2024-03-16T19:33:38.231343+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:33:38.231343+0000","last_undegraded":"2024-03-16T19:33:38.231343+0000","last_fullsized":"2024-03-16T19:33:38.231343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.241695+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:33:38.241695+0000","last_peered":"2024-03-16T19:33:38.241695+0000","last_clean":"2024-03-16T19:33:38.241695+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:33:38.241695+0000","last_undegraded":"2024-03-16T19:33:38.241695+0000","last_fullsized":"2024-03-16T19:33:38.241695+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231208+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:33:38.231208+0000","last_peered":"2024-03-16T19:33:38.231208+0000","last_clean":"2024-03-16T19:33:38.231208+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:33:38.231208+0000","last_undegraded":"2024-03-16T19:33:38.231208+0000","last_fullsized":"2024-03-16T19:33:38.231208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.231584+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:33:38.231584+0000","last_peered":"2024-03-16T19:33:38.231584+0000","last_clean":"2024-03-16T19:33:38.231584+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:33:38.231584+0000","last_undegraded":"2024-03-16T19:33:38.231584+0000","last_fullsized":"2024-03-16T19:33:38.231584+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":212,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:37:23.260783+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:37:23.260783+0000","last_peered":"2024-03-16T19:37:23.260783+0000","last_clean":"2024-03-16T19:37:23.260783+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:37:23.260783+0000","last_undegraded":"2024-03-16T19:37:23.260783+0000","last_fullsized":"2024-03-16T19:37:23.260783+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 303s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242558+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:33:48.242558+0000","last_peered":"2024-03-16T19:33:48.242558+0000","last_clean":"2024-03-16T19:33:48.242558+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:33:48.242558+0000","last_undegraded":"2024-03-16T19:33:48.242558+0000","last_fullsized":"2024-03-16T19:33:48.242558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:38.237393+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:33:38.237393+0000","last_peered":"2024-03-16T19:33:38.237393+0000","last_clean":"2024-03-16T19:33:38.237393+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:33:38.237393+0000","last_undegraded":"2024-03-16T19:33:38.237393+0000","last_fullsized":"2024-03-16T19:33:38.237393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231258+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:33:48.231258+0000","last_peered":"2024-03-16T19:33:48.231258+0000","last_clean":"2024-03-16T19:33:48.231258+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:33:48.231258+0000","last_undegraded":"2024-03-16T19:33:48.231258+0000","last_fullsized":"2024-03-16T19:33:48.231258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:03.237913+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:34:03.237913+0000","last_peered":"2024-03-16T19:34:03.237913+0000","last_clean":"2024-03-16T19:34:03.237913+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:34:03.237913+0000","last_undegraded":"2024-03-16T19:34:03.237913+0000","last_fullsized":"2024-03-16T19:34:03.237913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232268+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:33:48.232268+0000","last_peered":"2024-03-16T19:33:48.232268+0000","last_clean":"2024-03-16T19:33:48.232268+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:33:48.232268+0000","last_undegraded":"2024-03-16T19:33:48.232268+0000","last_fullsized":"2024-03-16T19:33:48.232268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232070+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:33:48.232070+0000","last_peered":"2024-03-16T19:33:48.232070+0000","last_clean":"2024-03-16T19:33:48.232070+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:33:48.232070+0000","last_undegraded":"2024-03-16T19:33:48.232070+0000","last_fullsized":"2024-03-16T19:33:48.232070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242493+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:33:48.242493+0000","last_peered":"2024-03-16T19:33:48.242493+0000","last_clean":"2024-03-16T19:33:48.242493+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:33:48.242493+0000","last_undegraded":"2024-03-16T19:33:48.242493+0000","last_fullsized":"2024-03-16T19:33:48.242493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":37,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:37:23.008681+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:37:23.008681+0000","last_peered":"2024-03-16T19:37:23.008681+0000","last_clean":"2024-03-16T19:37:23.008681+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:37:23.008681+0000","last_undegraded":"2024-03-16T19:37:23.008681+0000","last_fullsized":"2024-03-16T19:37:23.008681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 303s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242516+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:33:48.242516+0000","last_peered":"2024-03-16T19:33:48.242516+0000","last_clean":"2024-03-16T19:33:48.242516+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:33:48.242516+0000","last_undegraded":"2024-03-16T19:33:48.242516+0000","last_fullsized":"2024-03-16T19:33:48.242516+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237853+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:33:43.237853+0000","last_peered":"2024-03-16T19:33:43.237853+0000","last_clean":"2024-03-16T19:33:43.237853+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:33:43.237853+0000","last_undegraded":"2024-03-16T19:33:43.237853+0000","last_fullsized":"2024-03-16T19:33:43.237853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233004+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:33:58.233004+0000","last_peered":"2024-03-16T19:33:58.233004+0000","last_clean":"2024-03-16T19:33:58.233004+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:33:58.233004+0000","last_undegraded":"2024-03-16T19:33:58.233004+0000","last_fullsized":"2024-03-16T19:33:58.233004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231221+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:33:48.231221+0000","last_peered":"2024-03-16T19:33:48.231221+0000","last_clean":"2024-03-16T19:33:48.231221+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:33:48.231221+0000","last_undegraded":"2024-03-16T19:33:48.231221+0000","last_fullsized":"2024-03-16T19:33:48.231221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235008+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:34:28.235008+0000","last_peered":"2024-03-16T19:34:28.235008+0000","last_clean":"2024-03-16T19:34:28.235008+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:34:28.235008+0000","last_undegraded":"2024-03-16T19:34:28.235008+0000","last_fullsized":"2024-03-16T19:34:28.235008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235076+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:34:28.235076+0000","last_peered":"2024-03-16T19:34:28.235076+0000","last_clean":"2024-03-16T19:34:28.235076+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:34:28.235076+0000","last_undegraded":"2024-03-16T19:34:28.235076+0000","last_fullsized":"2024-03-16T19:34:28.235076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234995+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:34:28.234995+0000","last_peered":"2024-03-16T19:34:28.234995+0000","last_clean":"2024-03-16T19:34:28.234995+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:34:28.234995+0000","last_undegraded":"2024-03-16T19:34:28.234995+0000","last_fullsized":"2024-03-16T19:34:28.234995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235086+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:34:28.235086+0000","last_peered":"2024-03-16T19:34:28.235086+0000","last_clean":"2024-03-16T19:34:28.235086+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:34:28.235086+0000","last_undegraded":"2024-03-16T19:34:28.235086+0000","last_fullsized":"2024-03-16T19:34:28.235086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":182,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:38.241036+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:34:38.241036+0000","last_peered":"2024-03-16T19:34:38.241036+0000","last_clean":"2024-03-16T19:34:38.241036+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:34:38.241036+0000","last_undegraded":"2024-03-16T19:34:38.241036+0000","last_fullsized":"2024-03-16T19:34:38.241036+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.246603+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:35:28.246603+0000","last_peered":"2024-03-16T19:35:28.246603+0000","last_clean":"2024-03-16T19:35:28.246603+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:35:28.246603+0000","last_undegraded":"2024-03-16T19:35:28.246603+0000","last_fullsized":"2024-03-16T19:35:28.246603+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":177,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.246634+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:35:28.246634+0000","last_peered":"2024-03-16T19:35:28.246634+0000","last_clean":"2024-03-16T19:35:28.246634+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:35:28.246634+0000","last_undegraded":"2024-03-16T19:35:28.246634+0000","last_fullsized":"2024-03-16T19:35:28.246634+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237534+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:33:58.237534+0000","last_peered":"2024-03-16T19:33:58.237534+0000","last_clean":"2024-03-16T19:33:58.237534+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:33:58.237534+0000","last_undegraded":"2024-03-16T19:33:58.237534+0000","last_fullsized":"2024-03-16T19:33:58.237534+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574953,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574954,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574952,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574954,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122120,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574953,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:37:41.673 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:37:41.674 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:37:41.674 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:37:41.674 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:37:41.674 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:37:41.783 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:41.805+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:42.334 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:42.356+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:42.813 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:42.835+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:43.330 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:43.352+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:43.775 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:43.797+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:44.350 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:44.372+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:44.798 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:44.820+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:45.315 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:45.337+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:45.763 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:45.785+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:46.271 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:46.294+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:46.775 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:46.797+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:47.269 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:47.291+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:47.810 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:47.832+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:48.224 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:48.246+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:48.773 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:48.795+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:49.187 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:49.209+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:49.778 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:49.801+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:50.209 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:50.232+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:50.787 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:50.810+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:51.176 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:51.199+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:51.828 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:51.850+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:52.226 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:52.248+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:52.837 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:52.859+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:53.253 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:53.276+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:53.883 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:53.904+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:54.276 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:54.297+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:54.889 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:54.910+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:55.242 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:55.263+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:55.905 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:55.927+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:56.259 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:56.280+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:56.885 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:56.906+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:57.213 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:57.235+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:57.865 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:57.886+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:58.214 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:58.236+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:58.889 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:58.911+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:59.256 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:37:59.278+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:37:59.893 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:37:59.915+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:00.227 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:00.248+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:00.845 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:00.867+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:01.236 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:01.257+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:01.675 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:38:01.867 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:01.888+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:02.236 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:02.257+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:02.831 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:02.852+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:03.269 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:03.290+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:03.861 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:03.882+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:04.299 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:04.320+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:04.845 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:04.866+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:05.328 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:05.349+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:05.828 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:05.850+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:06.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.030+0000 7f46cbd3a640 1 Processor -- start 2024-03-16T19:38:06.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.030+0000 7f46cbd3a640 1 -- start start 2024-03-16T19:38:06.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.031+0000 7f46cbd3a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f46c40732d0 0x7f46c41648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:06.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.031+0000 7f46cbd3a640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f46c40c9650 0x7f46c4164e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:06.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.031+0000 7f46cbd3a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c406b630 con 0x7f46c416aeb0 2024-03-16T19:38:06.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.031+0000 7f46cbd3a640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c40c25b0 con 0x7f46c40c9650 2024-03-16T19:38:06.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.031+0000 7f46cbd3a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c40c1b70 con 0x7f46c40732d0 2024-03-16T19:38:06.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.032+0000 7f46c92d5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f46c40732d0 0x7f46c41648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:06.024 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.032+0000 7f46c9ad6640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f46c416aeb0 0x7f46c416b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.172:55826/0 (socket says 172.21.15.172:55826) 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.032+0000 7f46c9ad6640 1 -- 172.21.15.172:0/3528091581 learned_addr learned my addr 172.21.15.172:0/3528091581 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.032+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789835614 0 0) 0x7f46c406b630 con 0x7f46c416aeb0 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.032+0000 7f46ca2d7640 1 --2- 172.21.15.172:0/3528091581 >> v2:172.21.15.172:3300/0 conn(0x7f46c40c9650 0x7f46c4164e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.044+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b0003500 con 0x7f46c416aeb0 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.044+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3396772319 0 0) 0x7f46b0003500 con 0x7f46c416aeb0 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.044+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 >> v2:172.21.15.174:3300/0 conn(0x7f46c40732d0 msgr2=0x7f46c41648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.044+0000 7f46c8ad4640 1 --2- 172.21.15.172:0/3528091581 >> v2:172.21.15.174:3300/0 conn(0x7f46c40732d0 0x7f46c41648e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.044+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 >> v2:172.21.15.172:3300/0 conn(0x7f46c40c9650 msgr2=0x7f46c4164e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.044+0000 7f46c8ad4640 1 --2- 172.21.15.172:0/3528091581 >> v2:172.21.15.172:3300/0 conn(0x7f46c40c9650 0x7f46c4164e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.044+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c40c1e50 con 0x7f46c416aeb0 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.044+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 759959948 0 0) 0x7f46c0001670 con 0x7f46c416aeb0 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 >> v1:172.21.15.174:6789/0 conn(0x7f46c416aeb0 legacy=0x7f46c416b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:06.025 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c8ad4640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46b00082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c8ad4640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46b0008960 0x7f46b0008de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c8ad4640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0009450 0x7f46b000fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40c25b0 con 0x7f46b0008960 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40c1b70 con 0x7f46b0009450 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b0010330 con 0x7f46b0007eb0 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c92d5640 1 --2- 172.21.15.172:0/3528091581 >> v2:172.21.15.174:3300/0 conn(0x7f46c40732d0 0x7f46c41648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c9ad6640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46b00082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46ca2d7640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0009450 0x7f46b000fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c92d5640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46b0008960 0x7f46b0008de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c9ad6640 1 -- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0009450 msgr2=0x7f46b000fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c9ad6640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0009450 0x7f46b000fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c9ad6640 1 -- 172.21.15.172:0/3528091581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46b0008960 msgr2=0x7f46b0008de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c9ad6640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46b0008960 0x7f46b0008de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.026 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c9ad6640 1 -- 172.21.15.172:0/3528091581 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b0003500 con 0x7f46b0007eb0 2024-03-16T19:38:06.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46ca2d7640 1 -- 172.21.15.172:0/3528091581 reap_dead start 2024-03-16T19:38:06.027 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.045+0000 7f46c92d5640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46b0008960 0x7f46b0008de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T19:38:06.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.046+0000 7f46c9ad6640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46b00082f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c240 tx=0x7f46b00037b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=311acb8a67e76b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:38:06.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.046+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0005ad0 con 0x7f46b0007eb0 2024-03-16T19:38:06.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.046+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c0005c30 con 0x7f46b0007eb0 2024-03-16T19:38:06.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.046+0000 7f46c8ad4640 1 -- 172.21.15.172:0/3528091581 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c002ca90 con 0x7f46b0007eb0 2024-03-16T19:38:06.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.047+0000 7f46cbd3a640 1 -- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 msgr2=0x7f46b00082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:06.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.047+0000 7f46cbd3a640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46b00082f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c240 tx=0x7f46b00037b0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.047+0000 7f46cbd3a640 1 -- 172.21.15.172:0/3528091581 shutdown_connections 2024-03-16T19:38:06.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.047+0000 7f46cbd3a640 1 --2- 172.21.15.172:0/3528091581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46b00082f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.047+0000 7f46cbd3a640 1 -- 172.21.15.172:0/3528091581 >> 172.21.15.172:0/3528091581 conn(0x7f46c40c8b30 msgr2=0x7f46c40c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:38:06.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.048+0000 7f46cbd3a640 1 -- 172.21.15.172:0/3528091581 shutdown_connections 2024-03-16T19:38:06.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.048+0000 7f46cbd3a640 1 -- 172.21.15.172:0/3528091581 wait complete. 2024-03-16T19:38:06.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.049+0000 7f46cbd3a640 1 Processor -- start 2024-03-16T19:38:06.029 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.049+0000 7f46cbd3a640 1 -- start start 2024-03-16T19:38:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.050+0000 7f46cbd3a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46c415d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.050+0000 7f46cbd3a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46c40c75f0 0x7f46c4165a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.050+0000 7f46cbd3a640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46c4165f80 0x7f46c41fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.050+0000 7f46cbd3a640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c4165620 con 0x7f46c4165f80 2024-03-16T19:38:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.050+0000 7f46cbd3a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c415dad0 con 0x7f46c40c75f0 2024-03-16T19:38:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.050+0000 7f46cbd3a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c41673d0 con 0x7f46b0007eb0 2024-03-16T19:38:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c92d5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46c40c75f0 0x7f46c4165a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c92d5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46c40c75f0 0x7f46c4165a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.172:60456/0 (socket says 172.21.15.172:60456) 2024-03-16T19:38:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c9ad6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46c415d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c92d5640 1 -- 172.21.15.172:0/1086308721 learned_addr learned my addr 172.21.15.172:0/1086308721 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46ca2d7640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46c4165f80 0x7f46c41fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c9ad6640 1 -- 172.21.15.172:0/1086308721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46c40c75f0 msgr2=0x7f46c4165a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c9ad6640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46c40c75f0 0x7f46c4165a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c9ad6640 1 -- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46c4165f80 msgr2=0x7f46c41fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c9ad6640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46c4165f80 0x7f46c41fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c9ad6640 1 -- 172.21.15.172:0/1086308721 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c40c70a0 con 0x7f46b0007eb0 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46c9ad6640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46c415d590 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46b00104c0 tx=0x7f46c0001a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.051+0000 7f46ba7fc640 1 -- 172.21.15.172:0/1086308721 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c003c040 con 0x7f46b0007eb0 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.052+0000 7f46ba7fc640 1 -- 172.21.15.172:0/1086308721 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c002cd00 con 0x7f46b0007eb0 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.052+0000 7f46ba7fc640 1 -- 172.21.15.172:0/1086308721 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0007270 con 0x7f46b0007eb0 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.052+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c4166400 con 0x7f46b0007eb0 2024-03-16T19:38:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.052+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c40c8480 con 0x7f46b0007eb0 2024-03-16T19:38:06.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.053+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c40c82e0 con 0x7f46b0007eb0 2024-03-16T19:38:06.033 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.053+0000 7f46ba7fc640 1 -- 172.21.15.172:0/1086308721 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f46c0032070 con 0x7f46b0007eb0 2024-03-16T19:38:06.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.054+0000 7f46ba7fc640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f46a806b400 0x7f46a806d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:06.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.054+0000 7f46ba7fc640 1 -- 172.21.15.172:0/1086308721 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f46c00aaf80 con 0x7f46b0007eb0 2024-03-16T19:38:06.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.054+0000 7f46c92d5640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f46a806b400 0x7f46a806d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:06.034 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.055+0000 7f46c92d5640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f46a806b400 0x7f46a806d8c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f46c4163dd0 tx=0x7f46b400f040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:38:06.037 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.058+0000 7f46ba7fc640 1 -- 172.21.15.172:0/1086308721 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46c40c82e0 con 0x7f46b0007eb0 2024-03-16T19:38:06.283 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:06.305+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:06.364 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.385+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46c40c2130 con 0x7f46a806b400 2024-03-16T19:38:06.375 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.397+0000 7f46ba7fc640 1 -- 172.21.15.172:0/1086308721 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252938 (secure 0 0 0) 0x7f46b400ce40 con 0x7f46a806b400 2024-03-16T19:38:06.378 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:38:06.380 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:38:06.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f46a806b400 msgr2=0x7f46a806d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:06.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f46a806b400 0x7f46a806d8c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f46c4163dd0 tx=0x7f46b400f040 comp rx=0 tx=0).stop 2024-03-16T19:38:06.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 msgr2=0x7f46c415d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:06.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46c415d590 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46b00104c0 tx=0x7f46c0001a70 comp rx=0 tx=0).stop 2024-03-16T19:38:06.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 shutdown_connections 2024-03-16T19:38:06.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f46c4165f80 0x7f46c41fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46c40c75f0 0x7f46c4165a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.381 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f46a806b400 0x7f46a806d8c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:06.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 --2- 172.21.15.172:0/1086308721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46b0007eb0 0x7f46c415d590 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46b00104c0 tx=0x7f46c0001a70 comp rx=0 tx=0).stop 2024-03-16T19:38:06.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 >> 172.21.15.172:0/1086308721 conn(0x7f46c40c8b30 msgr2=0x7f46c4160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:38:06.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 shutdown_connections 2024-03-16T19:38:06.382 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:06.402+0000 7f46cbd3a640 1 -- 172.21.15.172:0/1086308721 wait complete. 2024-03-16T19:38:06.395 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-03-16T19:38:06.303254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002554"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240224+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:35:43.240224+0000","last_peered":"2024-03-16T19:35:43.240224+0000","last_clean":"2024-03-16T19:35:43.240224+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:35:43.240224+0000","last_undegraded":"2024-03-16T19:35:43.240224+0000","last_fullsized":"2024-03-16T19:35:43.240224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.252298+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:35:43.252298+0000","last_peered":"2024-03-16T19:35:43.252298+0000","last_clean":"2024-03-16T19:35:43.252298+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:35:43.252298+0000","last_undegraded":"2024-03-16T19:35:43.252298+0000","last_fullsized":"2024-03-16T19:35:43.252298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":115,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.246498+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:35:43.246498+0000","last_peered":"2024-03-16T19:35:43.246498+0000","last_clean":"2024-03-16T19:35:43.246498+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:35:43.246498+0000","last_undegraded":"2024-03-16T19:35:43.246498+0000","last_fullsized":"2024-03-16T19:35:43.246498+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.238944+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:35:28.238944+0000","last_peered":"2024-03-16T19:35:28.238944+0000","last_clean":"2024-03-16T19:35:28.238944+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:35:28.238944+0000","last_undegraded":"2024-03-16T19:35:28.238944+0000","last_fullsized":"2024-03-16T19:35:28.238944+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235065+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:34:28.235065+0000","last_peered":"2024-03-16T19:34:28.235065+0000","last_clean":"2024-03-16T19:34:28.235065+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:34:28.235065+0000","last_undegraded":"2024-03-16T19:34:28.235065+0000","last_fullsized":"2024-03-16T19:34:28.235065+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234972+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:34:28.234972+0000","last_peered":"2024-03-16T19:34:28.234972+0000","last_clean":"2024-03-16T19:34:28.234972+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:34:28.234972+0000","last_undegraded":"2024-03-16T19:34:28.234972+0000","last_fullsized":"2024-03-16T19:34:28.234972+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235046+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:34:28.235046+0000","last_peered":"2024-03-16T19:34:28.235046+0000","last_clean":"2024-03-16T19:34:28.235046+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:34:28.235046+0000","last_undegraded":"2024-03-16T19:34:28.235046+0000","last_fullsized":"2024-03-16T19:34:28.235046+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239229+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:33:58.239229+0000","last_peered":"2024-03-16T19:33:58.239229+0000","last_clean":"2024-03-16T19:33:58.239229+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:33:58.239229+0000","last_undegraded":"2024-03-16T19:33:58.239229+0000","last_fullsized":"2024-03-16T19:33:58.239229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234382+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:34:18.234382+0000","last_peered":"2024-03-16T19:34:18.234382+0000","last_clean":"2024-03-16T19:34:18.234382+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:34:18.234382+0000","last_undegraded":"2024-03-16T19:34:18.234382+0000","last_fullsized":"2024-03-16T19:34:18.234382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233484+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:34:18.233484+0000","last_peered":"2024-03-16T19:34:18.233484+0000","last_clean":"2024-03-16T19:34:18.233484+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:34:18.233484+0000","last_undegraded":"2024-03-16T19:34:18.233484+0000","last_fullsized":"2024-03-16T19:34:18.233484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233466+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:34:18.233466+0000","last_peered":"2024-03-16T19:34:18.233466+0000","last_clean":"2024-03-16T19:34:18.233466+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:34:18.233466+0000","last_undegraded":"2024-03-16T19:34:18.233466+0000","last_fullsized":"2024-03-16T19:34:18.233466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234110+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:34:18.234110+0000","last_peered":"2024-03-16T19:34:18.234110+0000","last_clean":"2024-03-16T19:34:18.234110+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:34:18.234110+0000","last_undegraded":"2024-03-16T19:34:18.234110+0000","last_fullsized":"2024-03-16T19:34:18.234110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238872+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:33:53.238872+0000","last_peered":"2024-03-16T19:33:53.238872+0000","last_clean":"2024-03-16T19:33:53.238872+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:33:53.238872+0000","last_undegraded":"2024-03-16T19:33:53.238872+0000","last_fullsized":"2024-03-16T19:33:53.238872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238806+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:34:13.238806+0000","last_peered":"2024-03-16T19:34:13.238806+0000","last_clean":"2024-03-16T19:34:13.238806+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:34:13.238806+0000","last_undegraded":"2024-03-16T19:34:13.238806+0000","last_fullsized":"2024-03-16T19:34:13.238806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.244177+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:34:08.244177+0000","last_peered":"2024-03-16T19:34:08.244177+0000","last_clean":"2024-03-16T19:34:08.244177+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:34:08.244177+0000","last_undegraded":"2024-03-16T19:34:08.244177+0000","last_fullsized":"2024-03-16T19:34:08.244177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233445+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:34:08.233445+0000","last_peered":"2024-03-16T19:34:08.233445+0000","last_clean":"2024-03-16T19:34:08.233445+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:34:08.233445+0000","last_undegraded":"2024-03-16T19:34:08.233445+0000","last_fullsized":"2024-03-16T19:34:08.233445+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233511+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:34:08.233511+0000","last_peered":"2024-03-16T19:34:08.233511+0000","last_clean":"2024-03-16T19:34:08.233511+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:34:08.233511+0000","last_undegraded":"2024-03-16T19:34:08.233511+0000","last_fullsized":"2024-03-16T19:34:08.233511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:07.990588+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:34:07.990588+0000","last_peered":"2024-03-16T19:34:07.990588+0000","last_clean":"2024-03-16T19:34:07.990588+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:34:07.990588+0000","last_undegraded":"2024-03-16T19:34:07.990588+0000","last_fullsized":"2024-03-16T19:34:07.990588+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233025+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:33:58.233025+0000","last_peered":"2024-03-16T19:33:58.233025+0000","last_clean":"2024-03-16T19:33:58.233025+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:33:58.233025+0000","last_undegraded":"2024-03-16T19:33:58.233025+0000","last_fullsized":"2024-03-16T19:33:58.233025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.231988+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:33:58.231988+0000","last_peered":"2024-03-16T19:33:58.231988+0000","last_clean":"2024-03-16T19:33:58.231988+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:33:58.231988+0000","last_undegraded":"2024-03-16T19:33:58.231988+0000","last_fullsized":"2024-03-16T19:33:58.231988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.238290+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:34:08.238290+0000","last_peered":"2024-03-16T19:34:08.238290+0000","last_clean":"2024-03-16T19:34:08.238290+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:34:08.238290+0000","last_undegraded":"2024-03-16T19:34:08.238290+0000","last_fullsized":"2024-03-16T19:34:08.238290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990165+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:34:02.990165+0000","last_peered":"2024-03-16T19:34:02.990165+0000","last_clean":"2024-03-16T19:34:02.990165+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:34:02.990165+0000","last_undegraded":"2024-03-16T19:34:02.990165+0000","last_fullsized":"2024-03-16T19:34:02.990165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":86,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237819+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:33:43.237819+0000","last_peered":"2024-03-16T19:33:43.237819+0000","last_clean":"2024-03-16T19:33:43.237819+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:33:43.237819+0000","last_undegraded":"2024-03-16T19:33:43.237819+0000","last_fullsized":"2024-03-16T19:33:43.237819+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231859+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:33:48.231859+0000","last_peered":"2024-03-16T19:33:48.231859+0000","last_clean":"2024-03-16T19:33:48.231859+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:33:48.231859+0000","last_undegraded":"2024-03-16T19:33:48.231859+0000","last_fullsized":"2024-03-16T19:33:48.231859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":173,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.238369+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:33:48.238369+0000","last_peered":"2024-03-16T19:33:48.238369+0000","last_clean":"2024-03-16T19:33:48.238369+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:33:48.238369+0000","last_undegraded":"2024-03-16T19:33:48.238369+0000","last_fullsized":"2024-03-16T19:33:48.238369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231247+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:33:48.231247+0000","last_peered":"2024-03-16T19:33:48.231247+0000","last_clean":"2024-03-16T19:33:48.231247+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:33:48.231247+0000","last_undegraded":"2024-03-16T19:33:48.231247+0000","last_fullsized":"2024-03-16T19:33:48.231247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242533+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:33:48.242533+0000","last_peered":"2024-03-16T19:33:48.242533+0000","last_clean":"2024-03-16T19:33:48.242533+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:33:48.242533+0000","last_undegraded":"2024-03-16T19:33:48.242533+0000","last_fullsized":"2024-03-16T19:33:48.242533+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:48.248360+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:35:48.248360+0000","last_peered":"2024-03-16T19:35:48.248360+0000","last_clean":"2024-03-16T19:35:48.248360+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:35:48.248360+0000","last_undegraded":"2024-03-16T19:35:48.248360+0000","last_fullsized":"2024-03-16T19:35:48.248360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237508+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:33:58.237508+0000","last_peered":"2024-03-16T19:33:58.237508+0000","last_clean":"2024-03-16T19:33:58.237508+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:33:58.237508+0000","last_undegraded":"2024-03-16T19:33:58.237508+0000","last_fullsized":"2024-03-16T19:33:58.237508+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239967+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:35:43.239967+0000","last_peered":"2024-03-16T19:35:43.239967+0000","last_clean":"2024-03-16T19:35:43.239967+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:35:43.239967+0000","last_undegraded":"2024-03-16T19:35:43.239967+0000","last_fullsized":"2024-03-16T19:35:43.239967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":75,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.000611+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:35:58.000611+0000","last_peered":"2024-03-16T19:35:58.000611+0000","last_clean":"2024-03-16T19:35:58.000611+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:35:58.000611+0000","last_undegraded":"2024-03-16T19:35:58.000611+0000","last_fullsized":"2024-03-16T19:35:58.000611+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239622+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:35:43.239622+0000","last_peered":"2024-03-16T19:35:43.239622+0000","last_clean":"2024-03-16T19:35:43.239622+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:35:43.239622+0000","last_undegraded":"2024-03-16T19:35:43.239622+0000","last_fullsized":"2024-03-16T19:35:43.239622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234753+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:34:28.234753+0000","last_peered":"2024-03-16T19:34:28.234753+0000","last_clean":"2024-03-16T19:34:28.234753+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:34:28.234753+0000","last_undegraded":"2024-03-16T19:34:28.234753+0000","last_fullsized":"2024-03-16T19:34:28.234753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239257+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:33:58.239257+0000","last_peered":"2024-03-16T19:33:58.239257+0000","last_clean":"2024-03-16T19:33:58.239257+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:33:58.239257+0000","last_undegraded":"2024-03-16T19:33:58.239257+0000","last_fullsized":"2024-03-16T19:33:58.239257+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:23.239683+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:34:23.239683+0000","last_peered":"2024-03-16T19:34:23.239683+0000","last_clean":"2024-03-16T19:34:23.239683+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:34:23.239683+0000","last_undegraded":"2024-03-16T19:34:23.239683+0000","last_fullsized":"2024-03-16T19:34:23.239683+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234192+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:34:28.234192+0000","last_peered":"2024-03-16T19:34:28.234192+0000","last_clean":"2024-03-16T19:34:28.234192+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:34:28.234192+0000","last_undegraded":"2024-03-16T19:34:28.234192+0000","last_fullsized":"2024-03-16T19:34:28.234192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238854+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:33:53.238854+0000","last_peered":"2024-03-16T19:33:53.238854+0000","last_clean":"2024-03-16T19:33:53.238854+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:33:53.238854+0000","last_undegraded":"2024-03-16T19:33:53.238854+0000","last_fullsized":"2024-03-16T19:33:53.238854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238825+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:33:53.238825+0000","last_peered":"2024-03-16T19:33:53.238825+0000","last_clean":"2024-03-16T19:33:53.238825+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:33:53.238825+0000","last_undegraded":"2024-03-16T19:33:53.238825+0000","last_fullsized":"2024-03-16T19:33:53.238825+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234171+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:34:18.234171+0000","last_peered":"2024-03-16T19:34:18.234171+0000","last_clean":"2024-03-16T19:34:18.234171+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:34:18.234171+0000","last_undegraded":"2024-03-16T19:34:18.234171+0000","last_fullsized":"2024-03-16T19:34:18.234171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.244981+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:34:18.244981+0000","last_peered":"2024-03-16T19:34:18.244981+0000","last_clean":"2024-03-16T19:34:18.244981+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:34:18.244981+0000","last_undegraded":"2024-03-16T19:34:18.244981+0000","last_fullsized":"2024-03-16T19:34:18.244981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233493+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:34:18.233493+0000","last_peered":"2024-03-16T19:34:18.233493+0000","last_clean":"2024-03-16T19:34:18.233493+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:34:18.233493+0000","last_undegraded":"2024-03-16T19:34:18.233493+0000","last_fullsized":"2024-03-16T19:34:18.233493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234208+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:34:18.234208+0000","last_peered":"2024-03-16T19:34:18.234208+0000","last_clean":"2024-03-16T19:34:18.234208+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:34:18.234208+0000","last_undegraded":"2024-03-16T19:34:18.234208+0000","last_fullsized":"2024-03-16T19:34:18.234208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234218+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:34:18.234218+0000","last_peered":"2024-03-16T19:34:18.234218+0000","last_clean":"2024-03-16T19:34:18.234218+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:34:18.234218+0000","last_undegraded":"2024-03-16T19:34:18.234218+0000","last_fullsized":"2024-03-16T19:34:18.234218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.239240+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:34:18.239240+0000","last_peered":"2024-03-16T19:34:18.239240+0000","last_clean":"2024-03-16T19:34:18.239240+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:34:18.239240+0000","last_undegraded":"2024-03-16T19:34:18.239240+0000","last_fullsized":"2024-03-16T19:34:18.239240+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.245012+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:34:18.245012+0000","last_peered":"2024-03-16T19:34:18.245012+0000","last_clean":"2024-03-16T19:34:18.245012+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:34:18.245012+0000","last_undegraded":"2024-03-16T19:34:18.245012+0000","last_fullsized":"2024-03-16T19:34:18.245012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":32,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:27.992368+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:34:27.992368+0000","last_peered":"2024-03-16T19:34:27.992368+0000","last_clean":"2024-03-16T19:34:27.992368+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:34:27.992368+0000","last_undegraded":"2024-03-16T19:34:27.992368+0000","last_fullsized":"2024-03-16T19:34:27.992368+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":248,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232779+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:34:08.232779+0000","last_peered":"2024-03-16T19:34:08.232779+0000","last_clean":"2024-03-16T19:34:08.232779+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:34:08.232779+0000","last_undegraded":"2024-03-16T19:34:08.232779+0000","last_fullsized":"2024-03-16T19:34:08.232779+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234360+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:34:18.234360+0000","last_peered":"2024-03-16T19:34:18.234360+0000","last_clean":"2024-03-16T19:34:18.234360+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:34:18.234360+0000","last_undegraded":"2024-03-16T19:34:18.234360+0000","last_fullsized":"2024-03-16T19:34:18.234360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238829+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:34:13.238829+0000","last_peered":"2024-03-16T19:34:13.238829+0000","last_clean":"2024-03-16T19:34:13.238829+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:34:13.238829+0000","last_undegraded":"2024-03-16T19:34:13.238829+0000","last_fullsized":"2024-03-16T19:34:13.238829+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238778+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:34:13.238778+0000","last_peered":"2024-03-16T19:34:13.238778+0000","last_clean":"2024-03-16T19:34:13.238778+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:34:13.238778+0000","last_undegraded":"2024-03-16T19:34:13.238778+0000","last_fullsized":"2024-03-16T19:34:13.238778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233485+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:34:08.233485+0000","last_peered":"2024-03-16T19:34:08.233485+0000","last_clean":"2024-03-16T19:34:08.233485+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:34:08.233485+0000","last_undegraded":"2024-03-16T19:34:08.233485+0000","last_fullsized":"2024-03-16T19:34:08.233485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233709+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:34:08.233709+0000","last_peered":"2024-03-16T19:34:08.233709+0000","last_clean":"2024-03-16T19:34:08.233709+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:34:08.233709+0000","last_undegraded":"2024-03-16T19:34:08.233709+0000","last_fullsized":"2024-03-16T19:34:08.233709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232766+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:34:08.232766+0000","last_peered":"2024-03-16T19:34:08.232766+0000","last_clean":"2024-03-16T19:34:08.232766+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:34:08.232766+0000","last_undegraded":"2024-03-16T19:34:08.232766+0000","last_fullsized":"2024-03-16T19:34:08.232766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232748+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:34:08.232748+0000","last_peered":"2024-03-16T19:34:08.232748+0000","last_clean":"2024-03-16T19:34:08.232748+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:34:08.232748+0000","last_undegraded":"2024-03-16T19:34:08.232748+0000","last_fullsized":"2024-03-16T19:34:08.232748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":87,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:37:48.010915+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:37:48.010915+0000","last_peered":"2024-03-16T19:37:48.010915+0000","last_clean":"2024-03-16T19:37:48.010915+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:37:48.010915+0000","last_undegraded":"2024-03-16T19:37:48.010915+0000","last_fullsized":"2024-03-16T19:37:48.010915+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 333s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233693+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:34:08.233693+0000","last_peered":"2024-03-16T19:34:08.233693+0000","last_clean":"2024-03-16T19:34:08.233693+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:34:08.233693+0000","last_undegraded":"2024-03-16T19:34:08.233693+0000","last_fullsized":"2024-03-16T19:34:08.233693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":216,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232760+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:33:58.232760+0000","last_peered":"2024-03-16T19:33:58.232760+0000","last_clean":"2024-03-16T19:33:58.232760+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:33:58.232760+0000","last_undegraded":"2024-03-16T19:33:58.232760+0000","last_fullsized":"2024-03-16T19:33:58.232760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233035+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:33:58.233035+0000","last_peered":"2024-03-16T19:33:58.233035+0000","last_clean":"2024-03-16T19:33:58.233035+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:33:58.233035+0000","last_undegraded":"2024-03-16T19:33:58.233035+0000","last_fullsized":"2024-03-16T19:33:58.233035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990184+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:34:02.990184+0000","last_peered":"2024-03-16T19:34:02.990184+0000","last_clean":"2024-03-16T19:34:02.990184+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:34:02.990184+0000","last_undegraded":"2024-03-16T19:34:02.990184+0000","last_fullsized":"2024-03-16T19:34:02.990184+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233045+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:33:58.233045+0000","last_peered":"2024-03-16T19:33:58.233045+0000","last_clean":"2024-03-16T19:33:58.233045+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:33:58.233045+0000","last_undegraded":"2024-03-16T19:33:58.233045+0000","last_fullsized":"2024-03-16T19:33:58.233045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.243408+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:33:58.243408+0000","last_peered":"2024-03-16T19:33:58.243408+0000","last_clean":"2024-03-16T19:33:58.243408+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:33:58.243408+0000","last_undegraded":"2024-03-16T19:33:58.243408+0000","last_fullsized":"2024-03-16T19:33:58.243408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.232756+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:33:58.232756+0000","last_peered":"2024-03-16T19:33:58.232756+0000","last_clean":"2024-03-16T19:33:58.232756+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:33:58.232756+0000","last_undegraded":"2024-03-16T19:33:58.232756+0000","last_fullsized":"2024-03-16T19:33:58.232756+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990175+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:34:02.990175+0000","last_peered":"2024-03-16T19:34:02.990175+0000","last_clean":"2024-03-16T19:34:02.990175+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:34:02.990175+0000","last_undegraded":"2024-03-16T19:34:02.990175+0000","last_fullsized":"2024-03-16T19:34:02.990175+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":167,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:43.236034+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:34:43.236034+0000","last_peered":"2024-03-16T19:34:43.236034+0000","last_clean":"2024-03-16T19:34:43.236034+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:34:43.236034+0000","last_undegraded":"2024-03-16T19:34:43.236034+0000","last_fullsized":"2024-03-16T19:34:43.236034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":377,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:18.238443+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:35:18.238443+0000","last_peered":"2024-03-16T19:35:18.238443+0000","last_clean":"2024-03-16T19:35:18.238443+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:35:18.238443+0000","last_undegraded":"2024-03-16T19:35:18.238443+0000","last_fullsized":"2024-03-16T19:35:18.238443+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240002+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:35:43.240002+0000","last_peered":"2024-03-16T19:35:43.240002+0000","last_clean":"2024-03-16T19:35:43.240002+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:35:43.240002+0000","last_undegraded":"2024-03-16T19:35:43.240002+0000","last_fullsized":"2024-03-16T19:35:43.240002+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:38.247538+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:35:38.247538+0000","last_peered":"2024-03-16T19:35:38.247538+0000","last_clean":"2024-03-16T19:35:38.247538+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:35:38.247538+0000","last_undegraded":"2024-03-16T19:35:38.247538+0000","last_fullsized":"2024-03-16T19:35:38.247538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.239079+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:35:28.239079+0000","last_peered":"2024-03-16T19:35:28.239079+0000","last_clean":"2024-03-16T19:35:28.239079+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:35:28.239079+0000","last_undegraded":"2024-03-16T19:35:28.239079+0000","last_fullsized":"2024-03-16T19:35:28.239079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237468+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:33:58.237468+0000","last_peered":"2024-03-16T19:33:58.237468+0000","last_clean":"2024-03-16T19:33:58.237468+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:33:58.237468+0000","last_undegraded":"2024-03-16T19:33:58.237468+0000","last_fullsized":"2024-03-16T19:33:58.237468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":65,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.000588+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:35:58.000588+0000","last_peered":"2024-03-16T19:35:58.000588+0000","last_clean":"2024-03-16T19:35:58.000588+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:35:58.000588+0000","last_undegraded":"2024-03-16T19:35:58.000588+0000","last_fullsized":"2024-03-16T19:35:58.000588+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":213,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:37:38.261952+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:37:38.261952+0000","last_peered":"2024-03-16T19:37:38.261952+0000","last_clean":"2024-03-16T19:37:38.261952+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:37:38.261952+0000","last_undegraded":"2024-03-16T19:37:38.261952+0000","last_fullsized":"2024-03-16T19:37:38.261952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 317s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239990+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:35:43.239990+0000","last_peered":"2024-03-16T19:35:43.239990+0000","last_clean":"2024-03-16T19:35:43.239990+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:35:43.239990+0000","last_undegraded":"2024-03-16T19:35:43.239990+0000","last_fullsized":"2024-03-16T19:35:43.239990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239673+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:35:43.239673+0000","last_peered":"2024-03-16T19:35:43.239673+0000","last_clean":"2024-03-16T19:35:43.239673+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:35:43.239673+0000","last_undegraded":"2024-03-16T19:35:43.239673+0000","last_fullsized":"2024-03-16T19:35:43.239673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.252272+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:35:43.252272+0000","last_peered":"2024-03-16T19:35:43.252272+0000","last_clean":"2024-03-16T19:35:43.252272+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:35:43.252272+0000","last_undegraded":"2024-03-16T19:35:43.252272+0000","last_fullsized":"2024-03-16T19:35:43.252272+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240023+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:35:43.240023+0000","last_peered":"2024-03-16T19:35:43.240023+0000","last_clean":"2024-03-16T19:35:43.240023+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:35:43.240023+0000","last_undegraded":"2024-03-16T19:35:43.240023+0000","last_fullsized":"2024-03-16T19:35:43.240023+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240209+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:35:43.240209+0000","last_peered":"2024-03-16T19:35:43.240209+0000","last_clean":"2024-03-16T19:35:43.240209+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:35:43.240209+0000","last_undegraded":"2024-03-16T19:35:43.240209+0000","last_fullsized":"2024-03-16T19:35:43.240209+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":213,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:37:38.262000+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:37:38.262000+0000","last_peered":"2024-03-16T19:37:38.262000+0000","last_clean":"2024-03-16T19:37:38.262000+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:37:38.262000+0000","last_undegraded":"2024-03-16T19:37:38.262000+0000","last_fullsized":"2024-03-16T19:37:38.262000+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 318s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242558+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:33:48.242558+0000","last_peered":"2024-03-16T19:33:48.242558+0000","last_clean":"2024-03-16T19:33:48.242558+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:33:48.242558+0000","last_undegraded":"2024-03-16T19:33:48.242558+0000","last_fullsized":"2024-03-16T19:33:48.242558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:48.248392+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:35:48.248392+0000","last_peered":"2024-03-16T19:35:48.248392+0000","last_clean":"2024-03-16T19:35:48.248392+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:35:48.248392+0000","last_undegraded":"2024-03-16T19:35:48.248392+0000","last_fullsized":"2024-03-16T19:35:48.248392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231258+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:33:48.231258+0000","last_peered":"2024-03-16T19:33:48.231258+0000","last_clean":"2024-03-16T19:33:48.231258+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:33:48.231258+0000","last_undegraded":"2024-03-16T19:33:48.231258+0000","last_fullsized":"2024-03-16T19:33:48.231258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:03.237913+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:34:03.237913+0000","last_peered":"2024-03-16T19:34:03.237913+0000","last_clean":"2024-03-16T19:34:03.237913+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:34:03.237913+0000","last_undegraded":"2024-03-16T19:34:03.237913+0000","last_fullsized":"2024-03-16T19:34:03.237913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232268+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:33:48.232268+0000","last_peered":"2024-03-16T19:33:48.232268+0000","last_clean":"2024-03-16T19:33:48.232268+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:33:48.232268+0000","last_undegraded":"2024-03-16T19:33:48.232268+0000","last_fullsized":"2024-03-16T19:33:48.232268+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.232070+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:33:48.232070+0000","last_peered":"2024-03-16T19:33:48.232070+0000","last_clean":"2024-03-16T19:33:48.232070+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:33:48.232070+0000","last_undegraded":"2024-03-16T19:33:48.232070+0000","last_fullsized":"2024-03-16T19:33:48.232070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242493+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:33:48.242493+0000","last_peered":"2024-03-16T19:33:48.242493+0000","last_clean":"2024-03-16T19:33:48.242493+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:33:48.242493+0000","last_undegraded":"2024-03-16T19:33:48.242493+0000","last_fullsized":"2024-03-16T19:33:48.242493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:37:38.010161+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:37:38.010161+0000","last_peered":"2024-03-16T19:37:38.010161+0000","last_clean":"2024-03-16T19:37:38.010161+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:37:38.010161+0000","last_undegraded":"2024-03-16T19:37:38.010161+0000","last_fullsized":"2024-03-16T19:37:38.010161+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 318s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.242516+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:33:48.242516+0000","last_peered":"2024-03-16T19:33:48.242516+0000","last_clean":"2024-03-16T19:33:48.242516+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:33:48.242516+0000","last_undegraded":"2024-03-16T19:33:48.242516+0000","last_fullsized":"2024-03-16T19:33:48.242516+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:43.237853+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:33:43.237853+0000","last_peered":"2024-03-16T19:33:43.237853+0000","last_clean":"2024-03-16T19:33:43.237853+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:33:43.237853+0000","last_undegraded":"2024-03-16T19:33:43.237853+0000","last_fullsized":"2024-03-16T19:33:43.237853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.233004+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:33:58.233004+0000","last_peered":"2024-03-16T19:33:58.233004+0000","last_clean":"2024-03-16T19:33:58.233004+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:33:58.233004+0000","last_undegraded":"2024-03-16T19:33:58.233004+0000","last_fullsized":"2024-03-16T19:33:58.233004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:48.231221+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:33:48.231221+0000","last_peered":"2024-03-16T19:33:48.231221+0000","last_clean":"2024-03-16T19:33:48.231221+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:33:48.231221+0000","last_undegraded":"2024-03-16T19:33:48.231221+0000","last_fullsized":"2024-03-16T19:33:48.231221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235008+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:34:28.235008+0000","last_peered":"2024-03-16T19:34:28.235008+0000","last_clean":"2024-03-16T19:34:28.235008+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:34:28.235008+0000","last_undegraded":"2024-03-16T19:34:28.235008+0000","last_fullsized":"2024-03-16T19:34:28.235008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235076+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:34:28.235076+0000","last_peered":"2024-03-16T19:34:28.235076+0000","last_clean":"2024-03-16T19:34:28.235076+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:34:28.235076+0000","last_undegraded":"2024-03-16T19:34:28.235076+0000","last_fullsized":"2024-03-16T19:34:28.235076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234995+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:34:28.234995+0000","last_peered":"2024-03-16T19:34:28.234995+0000","last_clean":"2024-03-16T19:34:28.234995+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:34:28.234995+0000","last_undegraded":"2024-03-16T19:34:28.234995+0000","last_fullsized":"2024-03-16T19:34:28.234995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235086+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:34:28.235086+0000","last_peered":"2024-03-16T19:34:28.235086+0000","last_clean":"2024-03-16T19:34:28.235086+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:34:28.235086+0000","last_undegraded":"2024-03-16T19:34:28.235086+0000","last_fullsized":"2024-03-16T19:34:28.235086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":182,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:38.241036+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:34:38.241036+0000","last_peered":"2024-03-16T19:34:38.241036+0000","last_clean":"2024-03-16T19:34:38.241036+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:34:38.241036+0000","last_undegraded":"2024-03-16T19:34:38.241036+0000","last_fullsized":"2024-03-16T19:34:38.241036+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.246603+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:35:28.246603+0000","last_peered":"2024-03-16T19:35:28.246603+0000","last_clean":"2024-03-16T19:35:28.246603+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:35:28.246603+0000","last_undegraded":"2024-03-16T19:35:28.246603+0000","last_fullsized":"2024-03-16T19:35:28.246603+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":177,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.246634+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:35:28.246634+0000","last_peered":"2024-03-16T19:35:28.246634+0000","last_clean":"2024-03-16T19:35:28.246634+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:35:28.246634+0000","last_undegraded":"2024-03-16T19:35:28.246634+0000","last_fullsized":"2024-03-16T19:35:28.246634+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.237534+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:33:58.237534+0000","last_peered":"2024-03-16T19:33:58.237534+0000","last_clean":"2024-03-16T19:33:58.237534+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:33:58.237534+0000","last_undegraded":"2024-03-16T19:33:58.237534+0000","last_fullsized":"2024-03-16T19:33:58.237534+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574958,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574959,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574957,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574958,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574959,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574958,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:38:06.401 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:06.402 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:06.402 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:06.402 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:06.402 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:38:06.869 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:06.891+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:07.312 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:07.333+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:07.905 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:07.927+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:08.361 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:08.383+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:08.949 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:08.970+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:09.399 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:09.420+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:09.915 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:09.937+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:10.400 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:10.422+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:10.952 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:10.973+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:11.387 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:11.407+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:11.963 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:11.984+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:12.408 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:12.430+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:12.988 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:13.010+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:13.435 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:13.457+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:14.024 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:14.046+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:14.475 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:14.497+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:14.995 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:15.017+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:15.496 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:15.519+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:16.043 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:16.065+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:16.466 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:16.488+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:17.091 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:17.113+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:17.469 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:17.491+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:18.089 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:18.112+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:18.421 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:18.443+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:19.129 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:19.152+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:19.386 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:19.409+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:20.166 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:20.188+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:20.425 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:20.448+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:21.199 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:21.222+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:21.408 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:21.430+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:22.153 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:22.175+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:22.364 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:22.387+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:23.156 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:23.178+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:23.375 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:23.397+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:24.184 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:24.206+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:24.379 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:24.402+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:25.192 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:25.214+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:25.344 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:25.367+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:26.205 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:26.227+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:26.363 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:26.386+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:26.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:38:27.199 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:27.221+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:27.403 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:27.426+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:28.245 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:28.268+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:28.393 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:28.415+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:29.228 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:29.251+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:29.428 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:29.451+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:29.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.804+0000 7f86ae9e2640 1 Processor -- start 2024-03-16T19:38:29.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.804+0000 7f86ae9e2640 1 -- start start 2024-03-16T19:38:29.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.805+0000 7f86ae9e2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f86a816aeb0 0x7f86a816b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:29.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.806+0000 7f86ae9e2640 1 -- --> v1:172.21.15.172:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a806b630 con 0x7f86a80732d0 2024-03-16T19:38:29.796 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.806+0000 7f86ae9e2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a80c1b70 con 0x7f86a80c9650 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.806+0000 7f86ae9e2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a80c25b0 con 0x7f86a816aeb0 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.806+0000 7f86a77fe640 1 --1- >> v1:172.21.15.172:6789/0 conn(0x7f86a80732d0 0x7f86a81648e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.172:6789/0 says I am v1:172.21.15.172:46570/0 (socket says 172.21.15.172:46570) 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.806+0000 7f86a77fe640 1 -- 172.21.15.172:0/3663233996 learned_addr learned my addr 172.21.15.172:0/3663233996 (peer_addr_for_me v1:172.21.15.172:0/0) 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.806+0000 7f86a7fff640 1 --2- 172.21.15.172:0/3663233996 >> v2:172.21.15.174:3300/0 conn(0x7f86a816aeb0 0x7f86a816b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.806+0000 7f86a6ffd640 1 -- 172.21.15.172:0/3663233996 <== mon.0 v1:172.21.15.172:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154800335 0 0) 0x7f86a806b630 con 0x7f86a80732d0 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.818+0000 7f86a6ffd640 1 -- 172.21.15.172:0/3663233996 --> v1:172.21.15.172:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8690003500 con 0x7f86a80732d0 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.818+0000 7f86a6ffd640 1 -- 172.21.15.172:0/3663233996 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792169149 0 0) 0x7f86a80c1b70 con 0x7f86a80c9650 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.818+0000 7f86a6ffd640 1 -- 172.21.15.172:0/3663233996 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a806b630 con 0x7f86a80c9650 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.818+0000 7f86a7fff640 1 -- 172.21.15.172:0/3663233996 >> v1:172.21.15.174:6789/0 conn(0x7f86a80c9650 legacy=0x7f86a806bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.818+0000 7f86a7fff640 1 -- 172.21.15.172:0/3663233996 >> v1:172.21.15.172:6789/0 conn(0x7f86a80732d0 legacy=0x7f86a81648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.819+0000 7f86a7fff640 1 -- 172.21.15.172:0/3663233996 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a80c1e50 con 0x7f86a816aeb0 2024-03-16T19:38:29.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.819+0000 7f86a7fff640 1 --2- 172.21.15.172:0/3663233996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a816aeb0 0x7f86a816b290 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f86980097d0 tx=0x7f869802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77966a4909dfd703 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:38:29.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.820+0000 7f86a6ffd640 1 -- 172.21.15.172:0/3663233996 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c001670 con 0x7f86a816aeb0 2024-03-16T19:38:29.798 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.820+0000 7f86a6ffd640 1 -- 172.21.15.172:0/3663233996 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86980092e0 con 0x7f86a816aeb0 2024-03-16T19:38:29.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.820+0000 7f86a6ffd640 1 -- 172.21.15.172:0/3663233996 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0001970 con 0x7f86a816aeb0 2024-03-16T19:38:29.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.820+0000 7f86ae9e2640 1 -- 172.21.15.172:0/3663233996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a816aeb0 msgr2=0x7f86a816b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:29.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.820+0000 7f86ae9e2640 1 --2- 172.21.15.172:0/3663233996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a816aeb0 0x7f86a816b290 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f86980097d0 tx=0x7f869802b7a0 comp rx=0 tx=0).stop 2024-03-16T19:38:29.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.821+0000 7f86ae9e2640 1 -- 172.21.15.172:0/3663233996 shutdown_connections 2024-03-16T19:38:29.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.821+0000 7f86ae9e2640 1 --2- 172.21.15.172:0/3663233996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a816aeb0 0x7f86a816b290 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:29.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.821+0000 7f86ae9e2640 1 -- 172.21.15.172:0/3663233996 >> 172.21.15.172:0/3663233996 conn(0x7f86a80c8b30 msgr2=0x7f86a80c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:38:29.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.821+0000 7f86ae9e2640 1 -- 172.21.15.172:0/3663233996 shutdown_connections 2024-03-16T19:38:29.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.821+0000 7f86ae9e2640 1 -- 172.21.15.172:0/3663233996 wait complete. 2024-03-16T19:38:29.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.822+0000 7f86ae9e2640 1 Processor -- start 2024-03-16T19:38:29.800 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.822+0000 7f86ae9e2640 1 -- start start 2024-03-16T19:38:29.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.823+0000 7f86ae9e2640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86a80732d0 0x7f86a8160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:29.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.823+0000 7f86ae9e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a80c9650 0x7f86a81f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:29.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.823+0000 7f86ae9e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a816aeb0 0x7f86a81f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:29.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.823+0000 7f86ae9e2640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a8167b50 con 0x7f86a80732d0 2024-03-16T19:38:29.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.823+0000 7f86ae9e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80c7b80 con 0x7f86a80c9650 2024-03-16T19:38:29.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.823+0000 7f86ae9e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80c8350 con 0x7f86a816aeb0 2024-03-16T19:38:29.801 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.823+0000 7f86a7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86a80732d0 0x7f86a8160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:29.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.823+0000 7f86a7fff640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86a80732d0 0x7f86a8160350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:45432/0 (socket says 172.21.15.172:45432) 2024-03-16T19:38:29.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.823+0000 7f86a7fff640 1 -- 172.21.15.172:0/1195210148 learned_addr learned my addr 172.21.15.172:0/1195210148 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:38:29.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a77fe640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a80c9650 0x7f86a81f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:29.802 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86acf7f640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a816aeb0 0x7f86a81f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a77fe640 1 -- 172.21.15.172:0/1195210148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a816aeb0 msgr2=0x7f86a81f3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a77fe640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a816aeb0 0x7f86a81f3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a77fe640 1 -- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86a80732d0 msgr2=0x7f86a8160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a77fe640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86a80732d0 0x7f86a8160350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a77fe640 1 -- 172.21.15.172:0/1195210148 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86980094b0 con 0x7f86a80c9650 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a7fff640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86a80732d0 0x7f86a8160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a77fe640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a80c9650 0x7f86a81f3950 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f869c00c780 tx=0x7f869c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a4ff9640 1 -- 172.21.15.172:0/1195210148 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c0072d0 con 0x7f86a80c9650 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.824+0000 7f86a4ff9640 1 -- 172.21.15.172:0/1195210148 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c007430 con 0x7f86a80c9650 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.825+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a8204ec0 con 0x7f86a80c9650 2024-03-16T19:38:29.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.825+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a80c7d90 con 0x7f86a80c9650 2024-03-16T19:38:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.825+0000 7f86a4ff9640 1 -- 172.21.15.172:0/1195210148 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c0047e0 con 0x7f86a80c9650 2024-03-16T19:38:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.826+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a80767c0 con 0x7f86a80c9650 2024-03-16T19:38:29.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.826+0000 7f86a4ff9640 1 -- 172.21.15.172:0/1195210148 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f869c004940 con 0x7f86a80c9650 2024-03-16T19:38:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.827+0000 7f86a4ff9640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f869406b3b0 0x7f869406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.827+0000 7f86a4ff9640 1 -- 172.21.15.172:0/1195210148 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f869c089370 con 0x7f86a80c9650 2024-03-16T19:38:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.827+0000 7f86a7fff640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f869406b3b0 0x7f869406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:29.805 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.828+0000 7f86a7fff640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f869406b3b0 0x7f869406d870 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f869802b7e0 tx=0x7f869802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:38:29.809 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:29.831+0000 7f86a4ff9640 1 -- 172.21.15.172:0/1195210148 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86a80767c0 con 0x7f86a80c9650 2024-03-16T19:38:30.123 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.145+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f86a8164ff0 con 0x7f869406b3b0 2024-03-16T19:38:30.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.156+0000 7f86a4ff9640 1 -- 172.21.15.172:0/1195210148 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252939 (secure 0 0 0) 0x7f8698048450 con 0x7f869406b3b0 2024-03-16T19:38:30.136 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:38:30.137 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:38:30.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f869406b3b0 msgr2=0x7f869406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:30.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f869406b3b0 0x7f869406d870 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f869802b7e0 tx=0x7f869802f000 comp rx=0 tx=0).stop 2024-03-16T19:38:30.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a80c9650 msgr2=0x7f86a81f3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:30.139 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a80c9650 0x7f86a81f3950 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f869c00c780 tx=0x7f869c00cc40 comp rx=0 tx=0).stop 2024-03-16T19:38:30.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 shutdown_connections 2024-03-16T19:38:30.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f869406b3b0 0x7f869406d870 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:30.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a816aeb0 0x7f86a81f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:30.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a80c9650 0x7f86a81f3950 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:30.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 --2- 172.21.15.172:0/1195210148 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86a80732d0 0x7f86a8160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:30.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 >> 172.21.15.172:0/1195210148 conn(0x7f86a80c8b30 msgr2=0x7f86a8071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:38:30.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 shutdown_connections 2024-03-16T19:38:30.140 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:30.160+0000 7f86ae9e2640 1 -- 172.21.15.172:0/1195210148 wait complete. 2024-03-16T19:38:30.153 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-03-16T19:38:28.307961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002579"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240224+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:35:43.240224+0000","last_peered":"2024-03-16T19:35:43.240224+0000","last_clean":"2024-03-16T19:35:43.240224+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:35:43.240224+0000","last_undegraded":"2024-03-16T19:35:43.240224+0000","last_fullsized":"2024-03-16T19:35:43.240224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.252298+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:35:43.252298+0000","last_peered":"2024-03-16T19:35:43.252298+0000","last_clean":"2024-03-16T19:35:43.252298+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:35:43.252298+0000","last_undegraded":"2024-03-16T19:35:43.252298+0000","last_fullsized":"2024-03-16T19:35:43.252298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":115,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.246498+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:35:43.246498+0000","last_peered":"2024-03-16T19:35:43.246498+0000","last_clean":"2024-03-16T19:35:43.246498+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:35:43.246498+0000","last_undegraded":"2024-03-16T19:35:43.246498+0000","last_fullsized":"2024-03-16T19:35:43.246498+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.238944+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:35:28.238944+0000","last_peered":"2024-03-16T19:35:28.238944+0000","last_clean":"2024-03-16T19:35:28.238944+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:35:28.238944+0000","last_undegraded":"2024-03-16T19:35:28.238944+0000","last_fullsized":"2024-03-16T19:35:28.238944+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235065+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:34:28.235065+0000","last_peered":"2024-03-16T19:34:28.235065+0000","last_clean":"2024-03-16T19:34:28.235065+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:34:28.235065+0000","last_undegraded":"2024-03-16T19:34:28.235065+0000","last_fullsized":"2024-03-16T19:34:28.235065+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234972+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:34:28.234972+0000","last_peered":"2024-03-16T19:34:28.234972+0000","last_clean":"2024-03-16T19:34:28.234972+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:34:28.234972+0000","last_undegraded":"2024-03-16T19:34:28.234972+0000","last_fullsized":"2024-03-16T19:34:28.234972+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235046+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:34:28.235046+0000","last_peered":"2024-03-16T19:34:28.235046+0000","last_clean":"2024-03-16T19:34:28.235046+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:34:28.235046+0000","last_undegraded":"2024-03-16T19:34:28.235046+0000","last_fullsized":"2024-03-16T19:34:28.235046+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239229+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:33:58.239229+0000","last_peered":"2024-03-16T19:33:58.239229+0000","last_clean":"2024-03-16T19:33:58.239229+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:33:58.239229+0000","last_undegraded":"2024-03-16T19:33:58.239229+0000","last_fullsized":"2024-03-16T19:33:58.239229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234382+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:34:18.234382+0000","last_peered":"2024-03-16T19:34:18.234382+0000","last_clean":"2024-03-16T19:34:18.234382+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:34:18.234382+0000","last_undegraded":"2024-03-16T19:34:18.234382+0000","last_fullsized":"2024-03-16T19:34:18.234382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233484+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:34:18.233484+0000","last_peered":"2024-03-16T19:34:18.233484+0000","last_clean":"2024-03-16T19:34:18.233484+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:34:18.233484+0000","last_undegraded":"2024-03-16T19:34:18.233484+0000","last_fullsized":"2024-03-16T19:34:18.233484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233466+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:34:18.233466+0000","last_peered":"2024-03-16T19:34:18.233466+0000","last_clean":"2024-03-16T19:34:18.233466+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:34:18.233466+0000","last_undegraded":"2024-03-16T19:34:18.233466+0000","last_fullsized":"2024-03-16T19:34:18.233466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234110+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:34:18.234110+0000","last_peered":"2024-03-16T19:34:18.234110+0000","last_clean":"2024-03-16T19:34:18.234110+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:34:18.234110+0000","last_undegraded":"2024-03-16T19:34:18.234110+0000","last_fullsized":"2024-03-16T19:34:18.234110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238872+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:33:53.238872+0000","last_peered":"2024-03-16T19:33:53.238872+0000","last_clean":"2024-03-16T19:33:53.238872+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:33:53.238872+0000","last_undegraded":"2024-03-16T19:33:53.238872+0000","last_fullsized":"2024-03-16T19:33:53.238872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238806+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:34:13.238806+0000","last_peered":"2024-03-16T19:34:13.238806+0000","last_clean":"2024-03-16T19:34:13.238806+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:34:13.238806+0000","last_undegraded":"2024-03-16T19:34:13.238806+0000","last_fullsized":"2024-03-16T19:34:13.238806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.244177+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:34:08.244177+0000","last_peered":"2024-03-16T19:34:08.244177+0000","last_clean":"2024-03-16T19:34:08.244177+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:34:08.244177+0000","last_undegraded":"2024-03-16T19:34:08.244177+0000","last_fullsized":"2024-03-16T19:34:08.244177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233445+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:34:08.233445+0000","last_peered":"2024-03-16T19:34:08.233445+0000","last_clean":"2024-03-16T19:34:08.233445+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:34:08.233445+0000","last_undegraded":"2024-03-16T19:34:08.233445+0000","last_fullsized":"2024-03-16T19:34:08.233445+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233511+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:34:08.233511+0000","last_peered":"2024-03-16T19:34:08.233511+0000","last_clean":"2024-03-16T19:34:08.233511+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:34:08.233511+0000","last_undegraded":"2024-03-16T19:34:08.233511+0000","last_fullsized":"2024-03-16T19:34:08.233511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:07.990588+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:34:07.990588+0000","last_peered":"2024-03-16T19:34:07.990588+0000","last_clean":"2024-03-16T19:34:07.990588+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:34:07.990588+0000","last_undegraded":"2024-03-16T19:34:07.990588+0000","last_fullsized":"2024-03-16T19:34:07.990588+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242410+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:36:13.242410+0000","last_peered":"2024-03-16T19:36:13.242410+0000","last_clean":"2024-03-16T19:36:13.242410+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:36:13.242410+0000","last_undegraded":"2024-03-16T19:36:13.242410+0000","last_fullsized":"2024-03-16T19:36:13.242410+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242025+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:36:13.242025+0000","last_peered":"2024-03-16T19:36:13.242025+0000","last_clean":"2024-03-16T19:36:13.242025+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:36:13.242025+0000","last_undegraded":"2024-03-16T19:36:13.242025+0000","last_fullsized":"2024-03-16T19:36:13.242025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.238290+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:34:08.238290+0000","last_peered":"2024-03-16T19:34:08.238290+0000","last_clean":"2024-03-16T19:34:08.238290+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:34:08.238290+0000","last_undegraded":"2024-03-16T19:34:08.238290+0000","last_fullsized":"2024-03-16T19:34:08.238290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990165+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:34:02.990165+0000","last_peered":"2024-03-16T19:34:02.990165+0000","last_clean":"2024-03-16T19:34:02.990165+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:34:02.990165+0000","last_undegraded":"2024-03-16T19:34:02.990165+0000","last_fullsized":"2024-03-16T19:34:02.990165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.249270+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:35:58.249270+0000","last_peered":"2024-03-16T19:35:58.249270+0000","last_clean":"2024-03-16T19:35:58.249270+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:35:58.249270+0000","last_undegraded":"2024-03-16T19:35:58.249270+0000","last_fullsized":"2024-03-16T19:35:58.249270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.241018+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:35:58.241018+0000","last_peered":"2024-03-16T19:35:58.241018+0000","last_clean":"2024-03-16T19:35:58.241018+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:35:58.241018+0000","last_undegraded":"2024-03-16T19:35:58.241018+0000","last_fullsized":"2024-03-16T19:35:58.241018+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":174,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.249329+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:35:58.249329+0000","last_peered":"2024-03-16T19:35:58.249329+0000","last_clean":"2024-03-16T19:35:58.249329+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:35:58.249329+0000","last_undegraded":"2024-03-16T19:35:58.249329+0000","last_fullsized":"2024-03-16T19:35:58.249329+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.240836+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:35:58.240836+0000","last_peered":"2024-03-16T19:35:58.240836+0000","last_clean":"2024-03-16T19:35:58.240836+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:35:58.240836+0000","last_undegraded":"2024-03-16T19:35:58.240836+0000","last_fullsized":"2024-03-16T19:35:58.240836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.253579+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:35:58.253579+0000","last_peered":"2024-03-16T19:35:58.253579+0000","last_clean":"2024-03-16T19:35:58.253579+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:35:58.253579+0000","last_undegraded":"2024-03-16T19:35:58.253579+0000","last_fullsized":"2024-03-16T19:35:58.253579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:48.248360+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:35:48.248360+0000","last_peered":"2024-03-16T19:35:48.248360+0000","last_clean":"2024-03-16T19:35:48.248360+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:35:48.248360+0000","last_undegraded":"2024-03-16T19:35:48.248360+0000","last_fullsized":"2024-03-16T19:35:48.248360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":100,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:03.248241+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:36:03.248241+0000","last_peered":"2024-03-16T19:36:03.248241+0000","last_clean":"2024-03-16T19:36:03.248241+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:36:03.248241+0000","last_undegraded":"2024-03-16T19:36:03.248241+0000","last_fullsized":"2024-03-16T19:36:03.248241+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239967+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:35:43.239967+0000","last_peered":"2024-03-16T19:35:43.239967+0000","last_clean":"2024-03-16T19:35:43.239967+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:35:43.239967+0000","last_undegraded":"2024-03-16T19:35:43.239967+0000","last_fullsized":"2024-03-16T19:35:43.239967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":75,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.000611+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:35:58.000611+0000","last_peered":"2024-03-16T19:35:58.000611+0000","last_clean":"2024-03-16T19:35:58.000611+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:35:58.000611+0000","last_undegraded":"2024-03-16T19:35:58.000611+0000","last_fullsized":"2024-03-16T19:35:58.000611+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239622+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:35:43.239622+0000","last_peered":"2024-03-16T19:35:43.239622+0000","last_clean":"2024-03-16T19:35:43.239622+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:35:43.239622+0000","last_undegraded":"2024-03-16T19:35:43.239622+0000","last_fullsized":"2024-03-16T19:35:43.239622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234753+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:34:28.234753+0000","last_peered":"2024-03-16T19:34:28.234753+0000","last_clean":"2024-03-16T19:34:28.234753+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:34:28.234753+0000","last_undegraded":"2024-03-16T19:34:28.234753+0000","last_fullsized":"2024-03-16T19:34:28.234753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239257+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:33:58.239257+0000","last_peered":"2024-03-16T19:33:58.239257+0000","last_clean":"2024-03-16T19:33:58.239257+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:33:58.239257+0000","last_undegraded":"2024-03-16T19:33:58.239257+0000","last_fullsized":"2024-03-16T19:33:58.239257+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:23.239683+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:34:23.239683+0000","last_peered":"2024-03-16T19:34:23.239683+0000","last_clean":"2024-03-16T19:34:23.239683+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:34:23.239683+0000","last_undegraded":"2024-03-16T19:34:23.239683+0000","last_fullsized":"2024-03-16T19:34:23.239683+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234192+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:34:28.234192+0000","last_peered":"2024-03-16T19:34:28.234192+0000","last_clean":"2024-03-16T19:34:28.234192+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:34:28.234192+0000","last_undegraded":"2024-03-16T19:34:28.234192+0000","last_fullsized":"2024-03-16T19:34:28.234192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238854+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:33:53.238854+0000","last_peered":"2024-03-16T19:33:53.238854+0000","last_clean":"2024-03-16T19:33:53.238854+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:33:53.238854+0000","last_undegraded":"2024-03-16T19:33:53.238854+0000","last_fullsized":"2024-03-16T19:33:53.238854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":96,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:53.238825+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:33:53.238825+0000","last_peered":"2024-03-16T19:33:53.238825+0000","last_clean":"2024-03-16T19:33:53.238825+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:33:53.238825+0000","last_undegraded":"2024-03-16T19:33:53.238825+0000","last_fullsized":"2024-03-16T19:33:53.238825+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234171+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:34:18.234171+0000","last_peered":"2024-03-16T19:34:18.234171+0000","last_clean":"2024-03-16T19:34:18.234171+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:34:18.234171+0000","last_undegraded":"2024-03-16T19:34:18.234171+0000","last_fullsized":"2024-03-16T19:34:18.234171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.244981+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:34:18.244981+0000","last_peered":"2024-03-16T19:34:18.244981+0000","last_clean":"2024-03-16T19:34:18.244981+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:34:18.244981+0000","last_undegraded":"2024-03-16T19:34:18.244981+0000","last_fullsized":"2024-03-16T19:34:18.244981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233493+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:34:18.233493+0000","last_peered":"2024-03-16T19:34:18.233493+0000","last_clean":"2024-03-16T19:34:18.233493+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:34:18.233493+0000","last_undegraded":"2024-03-16T19:34:18.233493+0000","last_fullsized":"2024-03-16T19:34:18.233493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234208+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:34:18.234208+0000","last_peered":"2024-03-16T19:34:18.234208+0000","last_clean":"2024-03-16T19:34:18.234208+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:34:18.234208+0000","last_undegraded":"2024-03-16T19:34:18.234208+0000","last_fullsized":"2024-03-16T19:34:18.234208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234218+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:34:18.234218+0000","last_peered":"2024-03-16T19:34:18.234218+0000","last_clean":"2024-03-16T19:34:18.234218+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:34:18.234218+0000","last_undegraded":"2024-03-16T19:34:18.234218+0000","last_fullsized":"2024-03-16T19:34:18.234218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.239240+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:34:18.239240+0000","last_peered":"2024-03-16T19:34:18.239240+0000","last_clean":"2024-03-16T19:34:18.239240+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:34:18.239240+0000","last_undegraded":"2024-03-16T19:34:18.239240+0000","last_fullsized":"2024-03-16T19:34:18.239240+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.245012+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:34:18.245012+0000","last_peered":"2024-03-16T19:34:18.245012+0000","last_clean":"2024-03-16T19:34:18.245012+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:34:18.245012+0000","last_undegraded":"2024-03-16T19:34:18.245012+0000","last_fullsized":"2024-03-16T19:34:18.245012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":32,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:27.992368+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:34:27.992368+0000","last_peered":"2024-03-16T19:34:27.992368+0000","last_clean":"2024-03-16T19:34:27.992368+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:34:27.992368+0000","last_undegraded":"2024-03-16T19:34:27.992368+0000","last_fullsized":"2024-03-16T19:34:27.992368+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":248,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232779+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:34:08.232779+0000","last_peered":"2024-03-16T19:34:08.232779+0000","last_clean":"2024-03-16T19:34:08.232779+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:34:08.232779+0000","last_undegraded":"2024-03-16T19:34:08.232779+0000","last_fullsized":"2024-03-16T19:34:08.232779+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234360+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:34:18.234360+0000","last_peered":"2024-03-16T19:34:18.234360+0000","last_clean":"2024-03-16T19:34:18.234360+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:34:18.234360+0000","last_undegraded":"2024-03-16T19:34:18.234360+0000","last_fullsized":"2024-03-16T19:34:18.234360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":218,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238829+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:34:13.238829+0000","last_peered":"2024-03-16T19:34:13.238829+0000","last_clean":"2024-03-16T19:34:13.238829+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:34:13.238829+0000","last_undegraded":"2024-03-16T19:34:13.238829+0000","last_fullsized":"2024-03-16T19:34:13.238829+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:13.238778+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:34:13.238778+0000","last_peered":"2024-03-16T19:34:13.238778+0000","last_clean":"2024-03-16T19:34:13.238778+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:34:13.238778+0000","last_undegraded":"2024-03-16T19:34:13.238778+0000","last_fullsized":"2024-03-16T19:34:13.238778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233485+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:34:08.233485+0000","last_peered":"2024-03-16T19:34:08.233485+0000","last_clean":"2024-03-16T19:34:08.233485+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:34:08.233485+0000","last_undegraded":"2024-03-16T19:34:08.233485+0000","last_fullsized":"2024-03-16T19:34:08.233485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233709+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:34:08.233709+0000","last_peered":"2024-03-16T19:34:08.233709+0000","last_clean":"2024-03-16T19:34:08.233709+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:34:08.233709+0000","last_undegraded":"2024-03-16T19:34:08.233709+0000","last_fullsized":"2024-03-16T19:34:08.233709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232766+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:34:08.232766+0000","last_peered":"2024-03-16T19:34:08.232766+0000","last_clean":"2024-03-16T19:34:08.232766+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:34:08.232766+0000","last_undegraded":"2024-03-16T19:34:08.232766+0000","last_fullsized":"2024-03-16T19:34:08.232766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.232748+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:34:08.232748+0000","last_peered":"2024-03-16T19:34:08.232748+0000","last_clean":"2024-03-16T19:34:08.232748+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:34:08.232748+0000","last_undegraded":"2024-03-16T19:34:08.232748+0000","last_fullsized":"2024-03-16T19:34:08.232748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":88,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:38:03.012363+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:38:03.012363+0000","last_peered":"2024-03-16T19:38:03.012363+0000","last_clean":"2024-03-16T19:38:03.012363+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:38:03.012363+0000","last_undegraded":"2024-03-16T19:38:03.012363+0000","last_fullsized":"2024-03-16T19:38:03.012363+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 348s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:08.233693+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:34:08.233693+0000","last_peered":"2024-03-16T19:34:08.233693+0000","last_clean":"2024-03-16T19:34:08.233693+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:34:08.233693+0000","last_undegraded":"2024-03-16T19:34:08.233693+0000","last_fullsized":"2024-03-16T19:34:08.233693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242067+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:36:13.242067+0000","last_peered":"2024-03-16T19:36:13.242067+0000","last_clean":"2024-03-16T19:36:13.242067+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:36:13.242067+0000","last_undegraded":"2024-03-16T19:36:13.242067+0000","last_fullsized":"2024-03-16T19:36:13.242067+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242425+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:36:13.242425+0000","last_peered":"2024-03-16T19:36:13.242425+0000","last_clean":"2024-03-16T19:36:13.242425+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:36:13.242425+0000","last_undegraded":"2024-03-16T19:36:13.242425+0000","last_fullsized":"2024-03-16T19:36:13.242425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990184+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:34:02.990184+0000","last_peered":"2024-03-16T19:34:02.990184+0000","last_clean":"2024-03-16T19:34:02.990184+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:34:02.990184+0000","last_undegraded":"2024-03-16T19:34:02.990184+0000","last_fullsized":"2024-03-16T19:34:02.990184+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242441+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:36:13.242441+0000","last_peered":"2024-03-16T19:36:13.242441+0000","last_clean":"2024-03-16T19:36:13.242441+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:36:13.242441+0000","last_undegraded":"2024-03-16T19:36:13.242441+0000","last_fullsized":"2024-03-16T19:36:13.242441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.254718+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:36:13.254718+0000","last_peered":"2024-03-16T19:36:13.254718+0000","last_clean":"2024-03-16T19:36:13.254718+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:36:13.254718+0000","last_undegraded":"2024-03-16T19:36:13.254718+0000","last_fullsized":"2024-03-16T19:36:13.254718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242270+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:36:13.242270+0000","last_peered":"2024-03-16T19:36:13.242270+0000","last_clean":"2024-03-16T19:36:13.242270+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:36:13.242270+0000","last_undegraded":"2024-03-16T19:36:13.242270+0000","last_fullsized":"2024-03-16T19:36:13.242270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:02.990175+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:34:02.990175+0000","last_peered":"2024-03-16T19:34:02.990175+0000","last_clean":"2024-03-16T19:34:02.990175+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:34:02.990175+0000","last_undegraded":"2024-03-16T19:34:02.990175+0000","last_fullsized":"2024-03-16T19:34:02.990175+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":167,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:43.236034+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:34:43.236034+0000","last_peered":"2024-03-16T19:34:43.236034+0000","last_clean":"2024-03-16T19:34:43.236034+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:34:43.236034+0000","last_undegraded":"2024-03-16T19:34:43.236034+0000","last_fullsized":"2024-03-16T19:34:43.236034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":377,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:18.238443+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:35:18.238443+0000","last_peered":"2024-03-16T19:35:18.238443+0000","last_clean":"2024-03-16T19:35:18.238443+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:35:18.238443+0000","last_undegraded":"2024-03-16T19:35:18.238443+0000","last_fullsized":"2024-03-16T19:35:18.238443+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240002+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:35:43.240002+0000","last_peered":"2024-03-16T19:35:43.240002+0000","last_clean":"2024-03-16T19:35:43.240002+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:35:43.240002+0000","last_undegraded":"2024-03-16T19:35:43.240002+0000","last_fullsized":"2024-03-16T19:35:43.240002+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:38.247538+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:35:38.247538+0000","last_peered":"2024-03-16T19:35:38.247538+0000","last_clean":"2024-03-16T19:35:38.247538+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:35:38.247538+0000","last_undegraded":"2024-03-16T19:35:38.247538+0000","last_fullsized":"2024-03-16T19:35:38.247538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.239079+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:35:28.239079+0000","last_peered":"2024-03-16T19:35:28.239079+0000","last_clean":"2024-03-16T19:35:28.239079+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:35:28.239079+0000","last_undegraded":"2024-03-16T19:35:28.239079+0000","last_fullsized":"2024-03-16T19:35:28.239079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:03.248206+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:36:03.248206+0000","last_peered":"2024-03-16T19:36:03.248206+0000","last_clean":"2024-03-16T19:36:03.248206+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:36:03.248206+0000","last_undegraded":"2024-03-16T19:36:03.248206+0000","last_fullsized":"2024-03-16T19:36:03.248206+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":65,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.000588+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:35:58.000588+0000","last_peered":"2024-03-16T19:35:58.000588+0000","last_clean":"2024-03-16T19:35:58.000588+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:35:58.000588+0000","last_undegraded":"2024-03-16T19:35:58.000588+0000","last_fullsized":"2024-03-16T19:35:58.000588+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":215,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:38:08.264461+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:38:08.264461+0000","last_peered":"2024-03-16T19:38:08.264461+0000","last_clean":"2024-03-16T19:38:08.264461+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:38:08.264461+0000","last_undegraded":"2024-03-16T19:38:08.264461+0000","last_fullsized":"2024-03-16T19:38:08.264461+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 347s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239990+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:35:43.239990+0000","last_peered":"2024-03-16T19:35:43.239990+0000","last_clean":"2024-03-16T19:35:43.239990+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:35:43.239990+0000","last_undegraded":"2024-03-16T19:35:43.239990+0000","last_fullsized":"2024-03-16T19:35:43.239990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239673+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:35:43.239673+0000","last_peered":"2024-03-16T19:35:43.239673+0000","last_clean":"2024-03-16T19:35:43.239673+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:35:43.239673+0000","last_undegraded":"2024-03-16T19:35:43.239673+0000","last_fullsized":"2024-03-16T19:35:43.239673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.252272+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:35:43.252272+0000","last_peered":"2024-03-16T19:35:43.252272+0000","last_clean":"2024-03-16T19:35:43.252272+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:35:43.252272+0000","last_undegraded":"2024-03-16T19:35:43.252272+0000","last_fullsized":"2024-03-16T19:35:43.252272+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240023+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:35:43.240023+0000","last_peered":"2024-03-16T19:35:43.240023+0000","last_clean":"2024-03-16T19:35:43.240023+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:35:43.240023+0000","last_undegraded":"2024-03-16T19:35:43.240023+0000","last_fullsized":"2024-03-16T19:35:43.240023+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240209+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:35:43.240209+0000","last_peered":"2024-03-16T19:35:43.240209+0000","last_clean":"2024-03-16T19:35:43.240209+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:35:43.240209+0000","last_undegraded":"2024-03-16T19:35:43.240209+0000","last_fullsized":"2024-03-16T19:35:43.240209+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":215,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:38:08.264525+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:38:08.264525+0000","last_peered":"2024-03-16T19:38:08.264525+0000","last_clean":"2024-03-16T19:38:08.264525+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:38:08.264525+0000","last_undegraded":"2024-03-16T19:38:08.264525+0000","last_fullsized":"2024-03-16T19:38:08.264525+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 348s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.253604+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:35:58.253604+0000","last_peered":"2024-03-16T19:35:58.253604+0000","last_clean":"2024-03-16T19:35:58.253604+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:35:58.253604+0000","last_undegraded":"2024-03-16T19:35:58.253604+0000","last_fullsized":"2024-03-16T19:35:58.253604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:48.248392+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:35:48.248392+0000","last_peered":"2024-03-16T19:35:48.248392+0000","last_clean":"2024-03-16T19:35:48.248392+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:35:48.248392+0000","last_undegraded":"2024-03-16T19:35:48.248392+0000","last_fullsized":"2024-03-16T19:35:48.248392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.240855+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:35:58.240855+0000","last_peered":"2024-03-16T19:35:58.240855+0000","last_clean":"2024-03-16T19:35:58.240855+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:35:58.240855+0000","last_undegraded":"2024-03-16T19:35:58.240855+0000","last_fullsized":"2024-03-16T19:35:58.240855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.249116+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:36:13.249116+0000","last_peered":"2024-03-16T19:36:13.249116+0000","last_clean":"2024-03-16T19:36:13.249116+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:36:13.249116+0000","last_undegraded":"2024-03-16T19:36:13.249116+0000","last_fullsized":"2024-03-16T19:36:13.249116+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.241280+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:35:58.241280+0000","last_peered":"2024-03-16T19:35:58.241280+0000","last_clean":"2024-03-16T19:35:58.241280+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:35:58.241280+0000","last_undegraded":"2024-03-16T19:35:58.241280+0000","last_fullsized":"2024-03-16T19:35:58.241280+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.240837+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:35:58.240837+0000","last_peered":"2024-03-16T19:35:58.240837+0000","last_clean":"2024-03-16T19:35:58.240837+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:35:58.240837+0000","last_undegraded":"2024-03-16T19:35:58.240837+0000","last_fullsized":"2024-03-16T19:35:58.240837+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.253534+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:35:58.253534+0000","last_peered":"2024-03-16T19:35:58.253534+0000","last_clean":"2024-03-16T19:35:58.253534+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:35:58.253534+0000","last_undegraded":"2024-03-16T19:35:58.253534+0000","last_fullsized":"2024-03-16T19:35:58.253534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":40,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:38:08.012777+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:38:08.012777+0000","last_peered":"2024-03-16T19:38:08.012777+0000","last_clean":"2024-03-16T19:38:08.012777+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:38:08.012777+0000","last_undegraded":"2024-03-16T19:38:08.012777+0000","last_fullsized":"2024-03-16T19:38:08.012777+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 348s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.253561+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:35:58.253561+0000","last_peered":"2024-03-16T19:35:58.253561+0000","last_clean":"2024-03-16T19:35:58.253561+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:35:58.253561+0000","last_undegraded":"2024-03-16T19:35:58.253561+0000","last_fullsized":"2024-03-16T19:35:58.253561+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":97,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.249305+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:35:58.249305+0000","last_peered":"2024-03-16T19:35:58.249305+0000","last_clean":"2024-03-16T19:35:58.249305+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:35:58.249305+0000","last_undegraded":"2024-03-16T19:35:58.249305+0000","last_fullsized":"2024-03-16T19:35:58.249305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242384+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:36:13.242384+0000","last_peered":"2024-03-16T19:36:13.242384+0000","last_clean":"2024-03-16T19:36:13.242384+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:36:13.242384+0000","last_undegraded":"2024-03-16T19:36:13.242384+0000","last_fullsized":"2024-03-16T19:36:13.242384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.240799+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:35:58.240799+0000","last_peered":"2024-03-16T19:35:58.240799+0000","last_clean":"2024-03-16T19:35:58.240799+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:35:58.240799+0000","last_undegraded":"2024-03-16T19:35:58.240799+0000","last_fullsized":"2024-03-16T19:35:58.240799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235008+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:34:28.235008+0000","last_peered":"2024-03-16T19:34:28.235008+0000","last_clean":"2024-03-16T19:34:28.235008+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:34:28.235008+0000","last_undegraded":"2024-03-16T19:34:28.235008+0000","last_fullsized":"2024-03-16T19:34:28.235008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235076+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:34:28.235076+0000","last_peered":"2024-03-16T19:34:28.235076+0000","last_clean":"2024-03-16T19:34:28.235076+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:34:28.235076+0000","last_undegraded":"2024-03-16T19:34:28.235076+0000","last_fullsized":"2024-03-16T19:34:28.235076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234995+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:34:28.234995+0000","last_peered":"2024-03-16T19:34:28.234995+0000","last_clean":"2024-03-16T19:34:28.234995+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:34:28.234995+0000","last_undegraded":"2024-03-16T19:34:28.234995+0000","last_fullsized":"2024-03-16T19:34:28.234995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235086+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:34:28.235086+0000","last_peered":"2024-03-16T19:34:28.235086+0000","last_clean":"2024-03-16T19:34:28.235086+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:34:28.235086+0000","last_undegraded":"2024-03-16T19:34:28.235086+0000","last_fullsized":"2024-03-16T19:34:28.235086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":182,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:38.241036+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:34:38.241036+0000","last_peered":"2024-03-16T19:34:38.241036+0000","last_clean":"2024-03-16T19:34:38.241036+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:34:38.241036+0000","last_undegraded":"2024-03-16T19:34:38.241036+0000","last_fullsized":"2024-03-16T19:34:38.241036+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.246603+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:35:28.246603+0000","last_peered":"2024-03-16T19:35:28.246603+0000","last_clean":"2024-03-16T19:35:28.246603+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:35:28.246603+0000","last_undegraded":"2024-03-16T19:35:28.246603+0000","last_fullsized":"2024-03-16T19:35:28.246603+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":177,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.246634+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:35:28.246634+0000","last_peered":"2024-03-16T19:35:28.246634+0000","last_clean":"2024-03-16T19:35:28.246634+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:35:28.246634+0000","last_undegraded":"2024-03-16T19:35:28.246634+0000","last_fullsized":"2024-03-16T19:35:28.246634+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":219,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:03.248271+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:36:03.248271+0000","last_peered":"2024-03-16T19:36:03.248271+0000","last_clean":"2024-03-16T19:36:03.248271+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:36:03.248271+0000","last_undegraded":"2024-03-16T19:36:03.248271+0000","last_fullsized":"2024-03-16T19:36:03.248271+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574963,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574964,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574963,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574964,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,17],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574963,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:38:30.158 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:30.159 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:30.159 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:30.159 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:30.159 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:38:30.218 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:30.240+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:30.460 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:30.483+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:31.215 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:31.238+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:31.448 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:31.469+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:32.170 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:32.192+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:32.401 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:32.423+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:33.216 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:33.239+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:33.430 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:33.453+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:34.253 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:34.276+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:161 3.0 3:03de5254:::10000000054.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:34.384 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:34.406+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:35.278 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:38:35.301+0000 7fda7ff29640 -1 osd.1 106 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:328 3.0 3:00b77ae3:::100000000fb.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:35.341 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:35.363+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:36.342 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:36.365+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:37.304 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:37.327+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:38.263 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:38.285+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:39.305 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:39.327+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:40.278 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:40.300+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:41.291 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:41.312+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:42.245 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:42.267+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:43.247 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:43.268+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:44.270 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:44.292+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:45.285 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:45.307+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:46.267 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:46.289+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:47.222 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:47.244+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:48.236 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:48.258+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:49.266 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:49.273+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:50.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:38:50.252 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:50.274+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:51.209 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:51.231+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:52.236 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:52.257+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:53.234 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:53.256+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:54.258 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:54.279+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:54.351 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.363+0000 7f54e12fa640 1 Processor -- start 2024-03-16T19:38:54.351 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.363+0000 7f54e12fa640 1 -- start start 2024-03-16T19:38:54.351 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54e12fa640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54dc0732d0 0x7f54dc1648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54e12fa640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f54dc0c9650 0x7f54dc164e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54e12fa640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc06b630 con 0x7f54dc16aeb0 2024-03-16T19:38:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54e12fa640 1 -- --> v2:172.21.15.172:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0c25b0 con 0x7f54dc0732d0 2024-03-16T19:38:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54e12fa640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0c1b70 con 0x7f54dc0c9650 2024-03-16T19:38:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54da7fc640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54dc0732d0 0x7f54dc1648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54da7fc640 1 --2- >> v2:172.21.15.172:3300/0 conn(0x7f54dc0732d0 0x7f54dc1648e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:53910/0 (socket says 172.21.15.172:53910) 2024-03-16T19:38:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54da7fc640 1 -- 172.21.15.172:0/2375027632 learned_addr learned my addr 172.21.15.172:0/2375027632 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:38:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54db7fe640 1 --2- 172.21.15.172:0/2375027632 >> v2:172.21.15.174:3300/0 conn(0x7f54dc0c9650 0x7f54dc164e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.364+0000 7f54d9ffb640 1 -- 172.21.15.172:0/2375027632 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019588179 0 0) 0x7f54dc06b630 con 0x7f54dc16aeb0 2024-03-16T19:38:54.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.371+0000 7f54d9ffb640 1 -- 172.21.15.172:0/2375027632 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c4003520 con 0x7f54dc16aeb0 2024-03-16T19:38:54.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.371+0000 7f54db7fe640 1 -- 172.21.15.172:0/2375027632 >> v2:172.21.15.172:3300/0 conn(0x7f54dc0732d0 msgr2=0x7f54dc1648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T19:38:54.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.371+0000 7f54db7fe640 1 --2- 172.21.15.172:0/2375027632 >> v2:172.21.15.172:3300/0 conn(0x7f54dc0732d0 0x7f54dc1648e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.372+0000 7f54db7fe640 1 -- 172.21.15.172:0/2375027632 >> v1:172.21.15.174:6789/0 conn(0x7f54dc16aeb0 legacy=0x7f54dc16b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:54.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.372+0000 7f54db7fe640 1 -- 172.21.15.172:0/2375027632 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0c1e50 con 0x7f54dc0c9650 2024-03-16T19:38:54.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.372+0000 7f54db7fe640 1 --2- 172.21.15.172:0/2375027632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc0c9650 0x7f54dc164e20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f54dc1687e0 tx=0x7f54d402fef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bbb33558f635e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:38:54.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.373+0000 7f54d9ffb640 1 -- 172.21.15.172:0/2375027632 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc001670 con 0x7f54dc0c9650 2024-03-16T19:38:54.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.373+0000 7f54d9ffb640 1 -- 172.21.15.172:0/2375027632 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d400e2e0 con 0x7f54dc0c9650 2024-03-16T19:38:54.353 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.373+0000 7f54d9ffb640 1 -- 172.21.15.172:0/2375027632 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4007220 con 0x7f54dc0c9650 2024-03-16T19:38:54.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.373+0000 7f54e12fa640 1 -- 172.21.15.172:0/2375027632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc0c9650 msgr2=0x7f54dc164e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:54.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.373+0000 7f54e12fa640 1 --2- 172.21.15.172:0/2375027632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc0c9650 0x7f54dc164e20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f54dc1687e0 tx=0x7f54d402fef0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.374+0000 7f54e12fa640 1 -- 172.21.15.172:0/2375027632 shutdown_connections 2024-03-16T19:38:54.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.374+0000 7f54e12fa640 1 --2- 172.21.15.172:0/2375027632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc0c9650 0x7f54dc164e20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.357 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.374+0000 7f54e12fa640 1 --2- 172.21.15.172:0/2375027632 >> v2:172.21.15.172:3300/0 conn(0x7f54dc0732d0 0x7f54dc1648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.374+0000 7f54e12fa640 1 -- 172.21.15.172:0/2375027632 >> 172.21.15.172:0/2375027632 conn(0x7f54dc0c8b30 msgr2=0x7f54dc0c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:38:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.374+0000 7f54e12fa640 1 -- 172.21.15.172:0/2375027632 shutdown_connections 2024-03-16T19:38:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.374+0000 7f54e12fa640 1 -- 172.21.15.172:0/2375027632 wait complete. 2024-03-16T19:38:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.375+0000 7f54e12fa640 1 Processor -- start 2024-03-16T19:38:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.375+0000 7f54e12fa640 1 -- start start 2024-03-16T19:38:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54e12fa640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0732d0 0x7f54dc15bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54e12fa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54dc0c9650 0x7f54dc06ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54e12fa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc16aeb0 0x7f54dc200190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54e12fa640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc15b360 con 0x7f54dc0732d0 2024-03-16T19:38:54.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54e12fa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc1668b0 con 0x7f54dc16aeb0 2024-03-16T19:38:54.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54e12fa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc15c2d0 con 0x7f54dc0c9650 2024-03-16T19:38:54.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54daffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0732d0 0x7f54dc15bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:54.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54db7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc16aeb0 0x7f54dc200190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:54.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54daffd640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0732d0 0x7f54dc15bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:53926/0 (socket says 172.21.15.172:53926) 2024-03-16T19:38:54.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54daffd640 1 -- 172.21.15.172:0/2216638939 learned_addr learned my addr 172.21.15.172:0/2216638939 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-16T19:38:54.359 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.376+0000 7f54da7fc640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54dc0c9650 0x7f54dc06ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:54.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54db7fe640 1 -- 172.21.15.172:0/2216638939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54dc0c9650 msgr2=0x7f54dc06ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:54.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54db7fe640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54dc0c9650 0x7f54dc06ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54db7fe640 1 -- 172.21.15.172:0/2216638939 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0732d0 msgr2=0x7f54dc15bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:54.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54db7fe640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0732d0 0x7f54dc15bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54db7fe640 1 -- 172.21.15.172:0/2216638939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d40077b0 con 0x7f54dc16aeb0 2024-03-16T19:38:54.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54db7fe640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc16aeb0 0x7f54dc200190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f54d4036000 tx=0x7f54d4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:38:54.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54c37fe640 1 -- 172.21.15.172:0/2216638939 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4046070 con 0x7f54dc16aeb0 2024-03-16T19:38:54.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54c37fe640 1 -- 172.21.15.172:0/2216638939 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d4006060 con 0x7f54dc16aeb0 2024-03-16T19:38:54.360 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54c37fe640 1 -- 172.21.15.172:0/2216638939 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4030dd0 con 0x7f54dc16aeb0 2024-03-16T19:38:54.361 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc15c540 con 0x7f54dc16aeb0 2024-03-16T19:38:54.361 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.377+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc15ad10 con 0x7f54dc16aeb0 2024-03-16T19:38:54.361 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.379+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc15b510 con 0x7f54dc16aeb0 2024-03-16T19:38:54.361 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.379+0000 7f54c37fe640 1 -- 172.21.15.172:0/2216638939 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f54d4006580 con 0x7f54dc16aeb0 2024-03-16T19:38:54.362 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.379+0000 7f54c37fe640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f54c806b3b0 0x7f54c806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T19:38:54.362 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.380+0000 7f54c37fe640 1 -- 172.21.15.172:0/2216638939 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5857+0+0 (secure 0 0 0) 0x7f54d40af620 con 0x7f54dc16aeb0 2024-03-16T19:38:54.362 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.380+0000 7f54daffd640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f54c806b3b0 0x7f54c806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T19:38:54.362 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.380+0000 7f54daffd640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f54c806b3b0 0x7f54c806d870 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0017a0 tx=0x7f54cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T19:38:54.362 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.384+0000 7f54c37fe640 1 -- 172.21.15.172:0/2216638939 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54dc15b510 con 0x7f54dc16aeb0 2024-03-16T19:38:54.596 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.618+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 --> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54dc06b2c0 con 0x7f54c806b3b0 2024-03-16T19:38:54.605 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.627+0000 7f54c37fe640 1 -- 172.21.15.172:0/2216638939 <== mgr.4107 v2:172.21.15.172:6832/1161375053 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252892 (secure 0 0 0) 0x7f54cc007df0 con 0x7f54c806b3b0 2024-03-16T19:38:54.608 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-16T19:38:54.609 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f54c806b3b0 msgr2=0x7f54c806d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f54c806b3b0 0x7f54c806d870 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0017a0 tx=0x7f54cc009040 comp rx=0 tx=0).stop 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc16aeb0 msgr2=0x7f54dc200190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc16aeb0 0x7f54dc200190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f54d4036000 tx=0x7f54d4004a40 comp rx=0 tx=0).stop 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 shutdown_connections 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54dc16aeb0 0x7f54dc200190 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.172:6832/1161375053,v1:172.21.15.172:6833/1161375053] conn(0x7f54c806b3b0 0x7f54c806d870 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54dc0c9650 0x7f54dc06ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 --2- 172.21.15.172:0/2216638939 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54dc0732d0 0x7f54dc15bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 >> 172.21.15.172:0/2216638939 conn(0x7f54dc0c8b30 msgr2=0x7f54dc071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 shutdown_connections 2024-03-16T19:38:54.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-16T19:38:54.630+0000 7f54e12fa640 1 -- 172.21.15.172:0/2216638939 wait complete. 2024-03-16T19:38:54.683 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-03-16T19:38:54.313360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631035,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1195128,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236760,"kb_used_data":11160,"kb_used_omap":16,"kb_used_meta":225519,"kb_avail":754737960,"statfs":{"total":773094113280,"available":772851671040,"internally_reserved":0,"allocated":11427840,"data_stored":6023254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16981,"internal_metadata":230931883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002447"},"pg_stats":[{"pgid":"3.3f","version":"106'8","reported_seq":199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240224+0000","last_change":"2024-03-16T19:32:36.221338+0000","last_active":"2024-03-16T19:35:43.240224+0000","last_peered":"2024-03-16T19:35:43.240224+0000","last_clean":"2024-03-16T19:35:43.240224+0000","last_became_active":"2024-03-16T19:29:26.359741+0000","last_became_peered":"2024-03-16T19:29:26.359741+0000","last_unstale":"2024-03-16T19:35:43.240224+0000","last_undegraded":"2024-03-16T19:35:43.240224+0000","last_fullsized":"2024-03-16T19:35:43.240224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:36.221195+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.221195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:29.899050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"106'4","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.252298+0000","last_change":"2024-03-16T19:32:36.681944+0000","last_active":"2024-03-16T19:35:43.252298+0000","last_peered":"2024-03-16T19:35:43.252298+0000","last_clean":"2024-03-16T19:35:43.252298+0000","last_became_active":"2024-03-16T19:31:49.979213+0000","last_became_peered":"2024-03-16T19:31:49.979213+0000","last_unstale":"2024-03-16T19:35:43.252298+0000","last_undegraded":"2024-03-16T19:35:43.252298+0000","last_fullsized":"2024-03-16T19:35:43.252298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.681794+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.681794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:15.866132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"106'7","reported_seq":115,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.246498+0000","last_change":"2024-03-16T19:32:28.236924+0000","last_active":"2024-03-16T19:35:43.246498+0000","last_peered":"2024-03-16T19:35:43.246498+0000","last_clean":"2024-03-16T19:35:43.246498+0000","last_became_active":"2024-03-16T19:31:51.807621+0000","last_became_peered":"2024-03-16T19:31:51.807621+0000","last_unstale":"2024-03-16T19:35:43.246498+0000","last_undegraded":"2024-03-16T19:35:43.246498+0000","last_fullsized":"2024-03-16T19:35:43.246498+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.236788+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.236788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:45:01.895596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"106'8","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.238944+0000","last_change":"2024-03-16T19:32:35.687123+0000","last_active":"2024-03-16T19:35:28.238944+0000","last_peered":"2024-03-16T19:35:28.238944+0000","last_clean":"2024-03-16T19:35:28.238944+0000","last_became_active":"2024-03-16T19:31:26.808336+0000","last_became_peered":"2024-03-16T19:31:26.808336+0000","last_unstale":"2024-03-16T19:35:28.238944+0000","last_undegraded":"2024-03-16T19:35:28.238944+0000","last_fullsized":"2024-03-16T19:35:28.238944+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.687031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.687031+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:41:01.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"106'4","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235065+0000","last_change":"2024-03-16T19:32:42.193204+0000","last_active":"2024-03-16T19:34:28.235065+0000","last_peered":"2024-03-16T19:34:28.235065+0000","last_clean":"2024-03-16T19:34:28.235065+0000","last_became_active":"2024-03-16T19:31:49.976557+0000","last_became_peered":"2024-03-16T19:31:49.976557+0000","last_unstale":"2024-03-16T19:34:28.235065+0000","last_undegraded":"2024-03-16T19:34:28.235065+0000","last_fullsized":"2024-03-16T19:34:28.235065+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:42.193031+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.193031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:54:32.825600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234972+0000","last_change":"2024-03-16T19:32:43.501752+0000","last_active":"2024-03-16T19:34:28.234972+0000","last_peered":"2024-03-16T19:34:28.234972+0000","last_clean":"2024-03-16T19:34:28.234972+0000","last_became_active":"2024-03-16T19:31:49.981937+0000","last_became_peered":"2024-03-16T19:31:49.981937+0000","last_unstale":"2024-03-16T19:34:28.234972+0000","last_undegraded":"2024-03-16T19:34:28.234972+0000","last_fullsized":"2024-03-16T19:34:28.234972+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:43.501654+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.501654+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:35:17.894611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"106'5","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235046+0000","last_change":"2024-03-16T19:32:45.180916+0000","last_active":"2024-03-16T19:34:28.235046+0000","last_peered":"2024-03-16T19:34:28.235046+0000","last_clean":"2024-03-16T19:34:28.235046+0000","last_became_active":"2024-03-16T19:31:49.980849+0000","last_became_peered":"2024-03-16T19:31:49.980849+0000","last_unstale":"2024-03-16T19:34:28.235046+0000","last_undegraded":"2024-03-16T19:34:28.235046+0000","last_fullsized":"2024-03-16T19:34:28.235046+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:45.180818+0000","last_clean_scrub_stamp":"2024-03-16T19:32:45.180818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:09:33.054857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"106'9","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239229+0000","last_change":"2024-03-16T19:32:31.142618+0000","last_active":"2024-03-16T19:33:58.239229+0000","last_peered":"2024-03-16T19:33:58.239229+0000","last_clean":"2024-03-16T19:33:58.239229+0000","last_became_active":"2024-03-16T19:31:26.808908+0000","last_became_peered":"2024-03-16T19:31:26.808908+0000","last_unstale":"2024-03-16T19:33:58.239229+0000","last_undegraded":"2024-03-16T19:33:58.239229+0000","last_fullsized":"2024-03-16T19:33:58.239229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.142461+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.142461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:19:17.328112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"106'11","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234382+0000","last_change":"2024-03-16T19:32:30.050037+0000","last_active":"2024-03-16T19:34:18.234382+0000","last_peered":"2024-03-16T19:34:18.234382+0000","last_clean":"2024-03-16T19:34:18.234382+0000","last_became_active":"2024-03-16T19:29:26.363239+0000","last_became_peered":"2024-03-16T19:29:26.363239+0000","last_unstale":"2024-03-16T19:34:18.234382+0000","last_undegraded":"2024-03-16T19:34:18.234382+0000","last_fullsized":"2024-03-16T19:34:18.234382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:30.049942+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.049942+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:26.927106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"106'16","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233484+0000","last_change":"2024-03-16T19:32:36.906526+0000","last_active":"2024-03-16T19:34:18.233484+0000","last_peered":"2024-03-16T19:34:18.233484+0000","last_clean":"2024-03-16T19:34:18.233484+0000","last_became_active":"2024-03-16T19:29:26.366590+0000","last_became_peered":"2024-03-16T19:29:26.366590+0000","last_unstale":"2024-03-16T19:34:18.233484+0000","last_undegraded":"2024-03-16T19:34:18.233484+0000","last_fullsized":"2024-03-16T19:34:18.233484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'16","last_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_deep_scrub":"106'16","last_deep_scrub_stamp":"2024-03-16T19:32:36.906354+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.906354+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:56:50.347031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"106'6","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233466+0000","last_change":"2024-03-16T19:32:37.946474+0000","last_active":"2024-03-16T19:34:18.233466+0000","last_peered":"2024-03-16T19:34:18.233466+0000","last_clean":"2024-03-16T19:34:18.233466+0000","last_became_active":"2024-03-16T19:29:26.366612+0000","last_became_peered":"2024-03-16T19:29:26.366612+0000","last_unstale":"2024-03-16T19:34:18.233466+0000","last_undegraded":"2024-03-16T19:34:18.233466+0000","last_fullsized":"2024-03-16T19:34:18.233466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.946374+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.946374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:28:45.380440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234110+0000","last_change":"2024-03-16T19:32:29.012365+0000","last_active":"2024-03-16T19:34:18.234110+0000","last_peered":"2024-03-16T19:34:18.234110+0000","last_clean":"2024-03-16T19:34:18.234110+0000","last_became_active":"2024-03-16T19:29:26.361149+0000","last_became_peered":"2024-03-16T19:29:26.361149+0000","last_unstale":"2024-03-16T19:34:18.234110+0000","last_undegraded":"2024-03-16T19:34:18.234110+0000","last_fullsized":"2024-03-16T19:34:18.234110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:29.012262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.012262+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:43:02.186677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"106'10","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:18.251016+0000","last_change":"2024-03-16T19:32:34.085970+0000","last_active":"2024-03-16T19:36:18.251016+0000","last_peered":"2024-03-16T19:36:18.251016+0000","last_clean":"2024-03-16T19:36:18.251016+0000","last_became_active":"2024-03-16T19:31:26.814412+0000","last_became_peered":"2024-03-16T19:31:26.814412+0000","last_unstale":"2024-03-16T19:36:18.251016+0000","last_undegraded":"2024-03-16T19:36:18.251016+0000","last_fullsized":"2024-03-16T19:36:18.251016+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.085865+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.085865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:36.310091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"106'10","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:33.250827+0000","last_change":"2024-03-16T19:32:31.194818+0000","last_active":"2024-03-16T19:36:33.250827+0000","last_peered":"2024-03-16T19:36:33.250827+0000","last_clean":"2024-03-16T19:36:33.250827+0000","last_became_active":"2024-03-16T19:31:51.809803+0000","last_became_peered":"2024-03-16T19:31:51.809803+0000","last_unstale":"2024-03-16T19:36:33.250827+0000","last_undegraded":"2024-03-16T19:36:33.250827+0000","last_fullsized":"2024-03-16T19:36:33.250827+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.194669+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.194669+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:14:00.095465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"106'7","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.256097+0000","last_change":"2024-03-16T19:32:30.622169+0000","last_active":"2024-03-16T19:36:28.256097+0000","last_peered":"2024-03-16T19:36:28.256097+0000","last_clean":"2024-03-16T19:36:28.256097+0000","last_became_active":"2024-03-16T19:29:26.351269+0000","last_became_peered":"2024-03-16T19:29:26.351269+0000","last_unstale":"2024-03-16T19:36:28.256097+0000","last_undegraded":"2024-03-16T19:36:28.256097+0000","last_fullsized":"2024-03-16T19:36:28.256097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.622029+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.622029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:52:18.127719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"106'9","reported_seq":214,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.243498+0000","last_change":"2024-03-16T19:32:41.528006+0000","last_active":"2024-03-16T19:36:28.243498+0000","last_peered":"2024-03-16T19:36:28.243498+0000","last_clean":"2024-03-16T19:36:28.243498+0000","last_became_active":"2024-03-16T19:31:49.972859+0000","last_became_peered":"2024-03-16T19:31:49.972859+0000","last_unstale":"2024-03-16T19:36:28.243498+0000","last_undegraded":"2024-03-16T19:36:28.243498+0000","last_fullsized":"2024-03-16T19:36:28.243498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:41.527915+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.527915+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:09.276076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"106'8","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.243560+0000","last_change":"2024-03-16T19:32:33.668910+0000","last_active":"2024-03-16T19:36:28.243560+0000","last_peered":"2024-03-16T19:36:28.243560+0000","last_clean":"2024-03-16T19:36:28.243560+0000","last_became_active":"2024-03-16T19:29:26.358688+0000","last_became_peered":"2024-03-16T19:29:26.358688+0000","last_unstale":"2024-03-16T19:36:28.243560+0000","last_undegraded":"2024-03-16T19:36:28.243560+0000","last_fullsized":"2024-03-16T19:36:28.243560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:33.668758+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.668758+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:06:04.914091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"106'10","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.003600+0000","last_change":"2024-03-16T19:32:30.158111+0000","last_active":"2024-03-16T19:36:28.003600+0000","last_peered":"2024-03-16T19:36:28.003600+0000","last_clean":"2024-03-16T19:36:28.003600+0000","last_became_active":"2024-03-16T19:31:49.980739+0000","last_became_peered":"2024-03-16T19:31:49.980739+0000","last_unstale":"2024-03-16T19:36:28.003600+0000","last_undegraded":"2024-03-16T19:36:28.003600+0000","last_fullsized":"2024-03-16T19:36:28.003600+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:30.157990+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.157990+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:23:55.416226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"106'7","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242410+0000","last_change":"2024-03-16T19:32:43.198985+0000","last_active":"2024-03-16T19:36:13.242410+0000","last_peered":"2024-03-16T19:36:13.242410+0000","last_clean":"2024-03-16T19:36:13.242410+0000","last_became_active":"2024-03-16T19:31:49.976588+0000","last_became_peered":"2024-03-16T19:31:49.976588+0000","last_unstale":"2024-03-16T19:36:13.242410+0000","last_undegraded":"2024-03-16T19:36:13.242410+0000","last_fullsized":"2024-03-16T19:36:13.242410+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:43.198883+0000","last_clean_scrub_stamp":"2024-03-16T19:32:43.198883+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:15.234528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"106'8","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242025+0000","last_change":"2024-03-16T19:32:34.874927+0000","last_active":"2024-03-16T19:36:13.242025+0000","last_peered":"2024-03-16T19:36:13.242025+0000","last_clean":"2024-03-16T19:36:13.242025+0000","last_became_active":"2024-03-16T19:29:26.359125+0000","last_became_peered":"2024-03-16T19:29:26.359125+0000","last_unstale":"2024-03-16T19:36:13.242025+0000","last_undegraded":"2024-03-16T19:36:13.242025+0000","last_fullsized":"2024-03-16T19:36:13.242025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:34.874844+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.874844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:25:47.219461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"106'11","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:23.250023+0000","last_change":"2024-03-16T19:32:26.202050+0000","last_active":"2024-03-16T19:36:23.250023+0000","last_peered":"2024-03-16T19:36:23.250023+0000","last_clean":"2024-03-16T19:36:23.250023+0000","last_became_active":"2024-03-16T19:31:51.807094+0000","last_became_peered":"2024-03-16T19:31:51.807094+0000","last_unstale":"2024-03-16T19:36:23.250023+0000","last_undegraded":"2024-03-16T19:36:23.250023+0000","last_fullsized":"2024-03-16T19:36:23.250023+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:26.201905+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.201905+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:49:05.899320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"106'10","reported_seq":74,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:18.002660+0000","last_change":"2024-03-16T19:32:31.113826+0000","last_active":"2024-03-16T19:36:18.002660+0000","last_peered":"2024-03-16T19:36:18.002660+0000","last_clean":"2024-03-16T19:36:18.002660+0000","last_became_active":"2024-03-16T19:31:49.980798+0000","last_became_peered":"2024-03-16T19:31:49.980798+0000","last_unstale":"2024-03-16T19:36:18.002660+0000","last_undegraded":"2024-03-16T19:36:18.002660+0000","last_fullsized":"2024-03-16T19:36:18.002660+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:31.113694+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.113694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:43:19.635053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"106'8","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.249270+0000","last_change":"2024-03-16T19:32:30.189280+0000","last_active":"2024-03-16T19:35:58.249270+0000","last_peered":"2024-03-16T19:35:58.249270+0000","last_clean":"2024-03-16T19:35:58.249270+0000","last_became_active":"2024-03-16T19:31:26.808512+0000","last_became_peered":"2024-03-16T19:31:26.808512+0000","last_unstale":"2024-03-16T19:35:58.249270+0000","last_undegraded":"2024-03-16T19:35:58.249270+0000","last_fullsized":"2024-03-16T19:35:58.249270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:30.189153+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.189153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:32:42.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"106'6","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.241018+0000","last_change":"2024-03-16T19:32:38.633708+0000","last_active":"2024-03-16T19:35:58.241018+0000","last_peered":"2024-03-16T19:35:58.241018+0000","last_clean":"2024-03-16T19:35:58.241018+0000","last_became_active":"2024-03-16T19:31:26.808536+0000","last_became_peered":"2024-03-16T19:31:26.808536+0000","last_unstale":"2024-03-16T19:35:58.241018+0000","last_undegraded":"2024-03-16T19:35:58.241018+0000","last_fullsized":"2024-03-16T19:35:58.241018+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.633601+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.633601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:33:58.418688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"106'2","reported_seq":174,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.249329+0000","last_change":"2024-03-16T19:32:36.030046+0000","last_active":"2024-03-16T19:35:58.249329+0000","last_peered":"2024-03-16T19:35:58.249329+0000","last_clean":"2024-03-16T19:35:58.249329+0000","last_became_active":"2024-03-16T19:31:27.802008+0000","last_became_peered":"2024-03-16T19:31:27.802008+0000","last_unstale":"2024-03-16T19:35:58.249329+0000","last_undegraded":"2024-03-16T19:35:58.249329+0000","last_fullsized":"2024-03-16T19:35:58.249329+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:36.029912+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.029912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:21.544581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"106'15","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.240836+0000","last_change":"2024-03-16T19:32:35.881976+0000","last_active":"2024-03-16T19:35:58.240836+0000","last_peered":"2024-03-16T19:35:58.240836+0000","last_clean":"2024-03-16T19:35:58.240836+0000","last_became_active":"2024-03-16T19:29:26.363186+0000","last_became_peered":"2024-03-16T19:29:26.363186+0000","last_unstale":"2024-03-16T19:35:58.240836+0000","last_undegraded":"2024-03-16T19:35:58.240836+0000","last_fullsized":"2024-03-16T19:35:58.240836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:35.881867+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.881867+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:13.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"106'6","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.253579+0000","last_change":"2024-03-16T19:32:31.651765+0000","last_active":"2024-03-16T19:35:58.253579+0000","last_peered":"2024-03-16T19:35:58.253579+0000","last_clean":"2024-03-16T19:35:58.253579+0000","last_became_active":"2024-03-16T19:31:26.811823+0000","last_became_peered":"2024-03-16T19:31:26.811823+0000","last_unstale":"2024-03-16T19:35:58.253579+0000","last_undegraded":"2024-03-16T19:35:58.253579+0000","last_fullsized":"2024-03-16T19:35:58.253579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:31.651620+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.651620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:21:16.929179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"106'4","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:48.248360+0000","last_change":"2024-03-16T19:32:35.051224+0000","last_active":"2024-03-16T19:35:48.248360+0000","last_peered":"2024-03-16T19:35:48.248360+0000","last_clean":"2024-03-16T19:35:48.248360+0000","last_became_active":"2024-03-16T19:31:26.814561+0000","last_became_peered":"2024-03-16T19:31:26.814561+0000","last_unstale":"2024-03-16T19:35:48.248360+0000","last_undegraded":"2024-03-16T19:35:48.248360+0000","last_fullsized":"2024-03-16T19:35:48.248360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.051115+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.051115+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:38.718713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"106'7","reported_seq":100,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:03.248241+0000","last_change":"2024-03-16T19:32:37.209568+0000","last_active":"2024-03-16T19:36:03.248241+0000","last_peered":"2024-03-16T19:36:03.248241+0000","last_clean":"2024-03-16T19:36:03.248241+0000","last_became_active":"2024-03-16T19:31:52.262032+0000","last_became_peered":"2024-03-16T19:31:52.262032+0000","last_unstale":"2024-03-16T19:36:03.248241+0000","last_undegraded":"2024-03-16T19:36:03.248241+0000","last_fullsized":"2024-03-16T19:36:03.248241+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:37.209440+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.209440+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:02.468219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"106'8","reported_seq":213,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239967+0000","last_change":"2024-03-16T19:32:40.540714+0000","last_active":"2024-03-16T19:35:43.239967+0000","last_peered":"2024-03-16T19:35:43.239967+0000","last_clean":"2024-03-16T19:35:43.239967+0000","last_became_active":"2024-03-16T19:31:26.813523+0000","last_became_peered":"2024-03-16T19:31:26.813523+0000","last_unstale":"2024-03-16T19:35:43.239967+0000","last_undegraded":"2024-03-16T19:35:43.239967+0000","last_fullsized":"2024-03-16T19:35:43.239967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:40.540621+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.540621+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:13:58.828250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"106'12","reported_seq":75,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.000611+0000","last_change":"2024-03-16T19:32:27.216691+0000","last_active":"2024-03-16T19:35:58.000611+0000","last_peered":"2024-03-16T19:35:58.000611+0000","last_clean":"2024-03-16T19:35:58.000611+0000","last_became_active":"2024-03-16T19:31:49.973790+0000","last_became_peered":"2024-03-16T19:31:49.973790+0000","last_unstale":"2024-03-16T19:35:58.000611+0000","last_undegraded":"2024-03-16T19:35:58.000611+0000","last_fullsized":"2024-03-16T19:35:58.000611+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:27.216573+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.216573+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:43:40.364857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"106'9","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239622+0000","last_change":"2024-03-16T19:32:40.897672+0000","last_active":"2024-03-16T19:35:43.239622+0000","last_peered":"2024-03-16T19:35:43.239622+0000","last_clean":"2024-03-16T19:35:43.239622+0000","last_became_active":"2024-03-16T19:31:49.971908+0000","last_became_peered":"2024-03-16T19:31:49.971908+0000","last_unstale":"2024-03-16T19:35:43.239622+0000","last_undegraded":"2024-03-16T19:35:43.239622+0000","last_fullsized":"2024-03-16T19:35:43.239622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:40.897582+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.897582+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:52.743461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"106'7","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234753+0000","last_change":"2024-03-16T19:32:28.002778+0000","last_active":"2024-03-16T19:34:28.234753+0000","last_peered":"2024-03-16T19:34:28.234753+0000","last_clean":"2024-03-16T19:34:28.234753+0000","last_became_active":"2024-03-16T19:29:26.355390+0000","last_became_peered":"2024-03-16T19:29:26.355390+0000","last_unstale":"2024-03-16T19:34:28.234753+0000","last_undegraded":"2024-03-16T19:34:28.234753+0000","last_fullsized":"2024-03-16T19:34:28.234753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:28.002660+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.002660+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:39:48.405282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:33:58.239257+0000","last_change":"2024-03-16T19:32:28.207579+0000","last_active":"2024-03-16T19:33:58.239257+0000","last_peered":"2024-03-16T19:33:58.239257+0000","last_clean":"2024-03-16T19:33:58.239257+0000","last_became_active":"2024-03-16T19:31:26.807689+0000","last_became_peered":"2024-03-16T19:31:26.807689+0000","last_unstale":"2024-03-16T19:33:58.239257+0000","last_undegraded":"2024-03-16T19:33:58.239257+0000","last_fullsized":"2024-03-16T19:33:58.239257+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:28.207433+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.207433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:37.667405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"106'8","reported_seq":205,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:23.239683+0000","last_change":"2024-03-16T19:32:35.265410+0000","last_active":"2024-03-16T19:34:23.239683+0000","last_peered":"2024-03-16T19:34:23.239683+0000","last_clean":"2024-03-16T19:34:23.239683+0000","last_became_active":"2024-03-16T19:31:51.812476+0000","last_became_peered":"2024-03-16T19:31:51.812476+0000","last_unstale":"2024-03-16T19:34:23.239683+0000","last_undegraded":"2024-03-16T19:34:23.239683+0000","last_fullsized":"2024-03-16T19:34:23.239683+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:35.265262+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.265262+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:01.455136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234192+0000","last_change":"2024-03-16T19:32:32.913659+0000","last_active":"2024-03-16T19:34:28.234192+0000","last_peered":"2024-03-16T19:34:28.234192+0000","last_clean":"2024-03-16T19:34:28.234192+0000","last_became_active":"2024-03-16T19:29:25.347200+0000","last_became_peered":"2024-03-16T19:29:25.347200+0000","last_unstale":"2024-03-16T19:34:28.234192+0000","last_undegraded":"2024-03-16T19:34:28.234192+0000","last_fullsized":"2024-03-16T19:34:28.234192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:32.913569+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.913569+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:16:32.647103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":79,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:18.250998+0000","last_change":"2024-03-16T19:32:32.105281+0000","last_active":"2024-03-16T19:36:18.250998+0000","last_peered":"2024-03-16T19:36:18.250998+0000","last_clean":"2024-03-16T19:36:18.250998+0000","last_became_active":"2024-03-16T19:31:26.810243+0000","last_became_peered":"2024-03-16T19:31:26.810243+0000","last_unstale":"2024-03-16T19:36:18.250998+0000","last_undegraded":"2024-03-16T19:36:18.250998+0000","last_fullsized":"2024-03-16T19:36:18.250998+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:32.105164+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.105164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:35:45.684293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"106'8","reported_seq":97,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:18.250975+0000","last_change":"2024-03-16T19:32:38.062784+0000","last_active":"2024-03-16T19:36:18.250975+0000","last_peered":"2024-03-16T19:36:18.250975+0000","last_clean":"2024-03-16T19:36:18.250975+0000","last_became_active":"2024-03-16T19:31:49.971979+0000","last_became_peered":"2024-03-16T19:31:49.971979+0000","last_unstale":"2024-03-16T19:36:18.250975+0000","last_undegraded":"2024-03-16T19:36:18.250975+0000","last_fullsized":"2024-03-16T19:36:18.250975+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:38.062636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.062636+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:13:32.926082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234171+0000","last_change":"2024-03-16T19:32:28.686005+0000","last_active":"2024-03-16T19:34:18.234171+0000","last_peered":"2024-03-16T19:34:18.234171+0000","last_clean":"2024-03-16T19:34:18.234171+0000","last_became_active":"2024-03-16T19:29:25.338016+0000","last_became_peered":"2024-03-16T19:29:25.338016+0000","last_unstale":"2024-03-16T19:34:18.234171+0000","last_undegraded":"2024-03-16T19:34:18.234171+0000","last_fullsized":"2024-03-16T19:34:18.234171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.685903+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.685903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:14:11.578775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"106'10","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.244981+0000","last_change":"2024-03-16T19:32:32.691258+0000","last_active":"2024-03-16T19:34:18.244981+0000","last_peered":"2024-03-16T19:34:18.244981+0000","last_clean":"2024-03-16T19:34:18.244981+0000","last_became_active":"2024-03-16T19:31:26.812081+0000","last_became_peered":"2024-03-16T19:31:26.812081+0000","last_unstale":"2024-03-16T19:34:18.244981+0000","last_undegraded":"2024-03-16T19:34:18.244981+0000","last_fullsized":"2024-03-16T19:34:18.244981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:32.691117+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.691117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:42:51.730144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'1","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.233493+0000","last_change":"2024-03-16T19:32:30.944590+0000","last_active":"2024-03-16T19:34:18.233493+0000","last_peered":"2024-03-16T19:34:18.233493+0000","last_clean":"2024-03-16T19:34:18.233493+0000","last_became_active":"2024-03-16T19:29:25.343539+0000","last_became_peered":"2024-03-16T19:29:25.343539+0000","last_unstale":"2024-03-16T19:34:18.233493+0000","last_undegraded":"2024-03-16T19:34:18.233493+0000","last_fullsized":"2024-03-16T19:34:18.233493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:30.944489+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.944489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:34.550044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"106'10","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234208+0000","last_change":"2024-03-16T19:32:34.706805+0000","last_active":"2024-03-16T19:34:18.234208+0000","last_peered":"2024-03-16T19:34:18.234208+0000","last_clean":"2024-03-16T19:34:18.234208+0000","last_became_active":"2024-03-16T19:29:26.361043+0000","last_became_peered":"2024-03-16T19:29:26.361043+0000","last_unstale":"2024-03-16T19:34:18.234208+0000","last_undegraded":"2024-03-16T19:34:18.234208+0000","last_fullsized":"2024-03-16T19:34:18.234208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'10","last_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_deep_scrub":"106'10","last_deep_scrub_stamp":"2024-03-16T19:32:34.706688+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.706688+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:20:07.593009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234218+0000","last_change":"2024-03-16T19:32:29.646648+0000","last_active":"2024-03-16T19:34:18.234218+0000","last_peered":"2024-03-16T19:34:18.234218+0000","last_clean":"2024-03-16T19:34:18.234218+0000","last_became_active":"2024-03-16T19:29:25.340706+0000","last_became_peered":"2024-03-16T19:29:25.340706+0000","last_unstale":"2024-03-16T19:34:18.234218+0000","last_undegraded":"2024-03-16T19:34:18.234218+0000","last_fullsized":"2024-03-16T19:34:18.234218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.646553+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.646553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:49:01.659122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"106'9","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.239240+0000","last_change":"2024-03-16T19:32:30.223858+0000","last_active":"2024-03-16T19:34:18.239240+0000","last_peered":"2024-03-16T19:34:18.239240+0000","last_clean":"2024-03-16T19:34:18.239240+0000","last_became_active":"2024-03-16T19:31:51.808856+0000","last_became_peered":"2024-03-16T19:31:51.808856+0000","last_unstale":"2024-03-16T19:34:18.239240+0000","last_undegraded":"2024-03-16T19:34:18.239240+0000","last_fullsized":"2024-03-16T19:34:18.239240+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:30.223720+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.223720+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:53:42.385041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"106'7","reported_seq":198,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.245012+0000","last_change":"2024-03-16T19:32:26.538424+0000","last_active":"2024-03-16T19:34:18.245012+0000","last_peered":"2024-03-16T19:34:18.245012+0000","last_clean":"2024-03-16T19:34:18.245012+0000","last_became_active":"2024-03-16T19:29:25.339252+0000","last_became_peered":"2024-03-16T19:29:25.339252+0000","last_unstale":"2024-03-16T19:34:18.245012+0000","last_undegraded":"2024-03-16T19:34:18.245012+0000","last_fullsized":"2024-03-16T19:34:18.245012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:26.538251+0000","last_clean_scrub_stamp":"2024-03-16T19:32:26.538251+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:09.854028+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"106'11","reported_seq":32,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:27.992368+0000","last_change":"2024-03-16T19:32:35.076164+0000","last_active":"2024-03-16T19:34:27.992368+0000","last_peered":"2024-03-16T19:34:27.992368+0000","last_clean":"2024-03-16T19:34:27.992368+0000","last_became_active":"2024-03-16T19:32:08.938486+0000","last_became_peered":"2024-03-16T19:32:08.938486+0000","last_unstale":"2024-03-16T19:34:27.992368+0000","last_undegraded":"2024-03-16T19:34:27.992368+0000","last_fullsized":"2024-03-16T19:34:27.992368+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'11","last_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_deep_scrub":"106'11","last_deep_scrub_stamp":"2024-03-16T19:32:35.076009+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.076009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:38.991520+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"106'27","reported_seq":249,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.243283+0000","last_change":"2024-03-16T19:32:38.954039+0000","last_active":"2024-03-16T19:36:28.243283+0000","last_peered":"2024-03-16T19:36:28.243283+0000","last_clean":"2024-03-16T19:36:28.243283+0000","last_became_active":"2024-03-16T19:31:26.815423+0000","last_became_peered":"2024-03-16T19:31:26.815423+0000","last_unstale":"2024-03-16T19:36:28.243283+0000","last_undegraded":"2024-03-16T19:36:28.243283+0000","last_fullsized":"2024-03-16T19:36:28.243283+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_deep_scrub":"106'27","last_deep_scrub_stamp":"2024-03-16T19:32:38.953939+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.953939+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:56.933085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13324,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"106'6","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:18.234360+0000","last_change":"2024-03-16T19:32:41.179941+0000","last_active":"2024-03-16T19:34:18.234360+0000","last_peered":"2024-03-16T19:34:18.234360+0000","last_clean":"2024-03-16T19:34:18.234360+0000","last_became_active":"2024-03-16T19:31:26.810160+0000","last_became_peered":"2024-03-16T19:31:26.810160+0000","last_unstale":"2024-03-16T19:34:18.234360+0000","last_undegraded":"2024-03-16T19:34:18.234360+0000","last_fullsized":"2024-03-16T19:34:18.234360+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:41.179841+0000","last_clean_scrub_stamp":"2024-03-16T19:32:41.179841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:10:01.279170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"106'4","reported_seq":219,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:33.250844+0000","last_change":"2024-03-16T19:32:36.245702+0000","last_active":"2024-03-16T19:36:33.250844+0000","last_peered":"2024-03-16T19:36:33.250844+0000","last_clean":"2024-03-16T19:36:33.250844+0000","last_became_active":"2024-03-16T19:31:51.812611+0000","last_became_peered":"2024-03-16T19:31:51.812611+0000","last_unstale":"2024-03-16T19:36:33.250844+0000","last_undegraded":"2024-03-16T19:36:33.250844+0000","last_fullsized":"2024-03-16T19:36:33.250844+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:36.245565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.245565+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:22.902653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"106'9","reported_seq":207,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:33.250804+0000","last_change":"2024-03-16T19:32:27.197908+0000","last_active":"2024-03-16T19:36:33.250804+0000","last_peered":"2024-03-16T19:36:33.250804+0000","last_clean":"2024-03-16T19:36:33.250804+0000","last_became_active":"2024-03-16T19:31:51.807604+0000","last_became_peered":"2024-03-16T19:31:51.807604+0000","last_unstale":"2024-03-16T19:36:33.250804+0000","last_undegraded":"2024-03-16T19:36:33.250804+0000","last_fullsized":"2024-03-16T19:36:33.250804+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:27.197749+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.197749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:55.872269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"84'1","reported_seq":208,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.243522+0000","last_change":"2024-03-16T19:32:36.696958+0000","last_active":"2024-03-16T19:36:28.243522+0000","last_peered":"2024-03-16T19:36:28.243522+0000","last_clean":"2024-03-16T19:36:28.243522+0000","last_became_active":"2024-03-16T19:31:26.808450+0000","last_became_peered":"2024-03-16T19:31:26.808450+0000","last_unstale":"2024-03-16T19:36:28.243522+0000","last_undegraded":"2024-03-16T19:36:28.243522+0000","last_fullsized":"2024-03-16T19:36:28.243522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'1","last_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_deep_scrub":"84'1","last_deep_scrub_stamp":"2024-03-16T19:32:36.696860+0000","last_clean_scrub_stamp":"2024-03-16T19:32:36.696860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:11:40.661836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"106'15","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.243593+0000","last_change":"2024-03-16T19:32:37.190754+0000","last_active":"2024-03-16T19:36:28.243593+0000","last_peered":"2024-03-16T19:36:28.243593+0000","last_clean":"2024-03-16T19:36:28.243593+0000","last_became_active":"2024-03-16T19:29:26.360393+0000","last_became_peered":"2024-03-16T19:29:26.360393+0000","last_unstale":"2024-03-16T19:36:28.243593+0000","last_undegraded":"2024-03-16T19:36:28.243593+0000","last_fullsized":"2024-03-16T19:36:28.243593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:37.190612+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.190612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:35:24.325830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.243265+0000","last_change":"2024-03-16T19:32:29.938129+0000","last_active":"2024-03-16T19:36:28.243265+0000","last_peered":"2024-03-16T19:36:28.243265+0000","last_clean":"2024-03-16T19:36:28.243265+0000","last_became_active":"2024-03-16T19:29:25.342228+0000","last_became_peered":"2024-03-16T19:29:25.342228+0000","last_unstale":"2024-03-16T19:36:28.243265+0000","last_undegraded":"2024-03-16T19:36:28.243265+0000","last_fullsized":"2024-03-16T19:36:28.243265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:29.937972+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.937972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:25:04.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"106'6","reported_seq":206,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.243236+0000","last_change":"2024-03-16T19:32:39.932726+0000","last_active":"2024-03-16T19:36:28.243236+0000","last_peered":"2024-03-16T19:36:28.243236+0000","last_clean":"2024-03-16T19:36:28.243236+0000","last_became_active":"2024-03-16T19:31:49.971552+0000","last_became_peered":"2024-03-16T19:31:49.971552+0000","last_unstale":"2024-03-16T19:36:28.243236+0000","last_undegraded":"2024-03-16T19:36:28.243236+0000","last_fullsized":"2024-03-16T19:36:28.243236+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:39.932636+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.932636+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:21.334630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'9","reported_seq":90,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:38:33.015088+0000","last_change":"2024-03-16T19:32:29.203243+0000","last_active":"2024-03-16T19:38:33.015088+0000","last_peered":"2024-03-16T19:38:33.015088+0000","last_clean":"2024-03-16T19:38:33.015088+0000","last_became_active":"2024-03-16T19:31:49.975256+0000","last_became_peered":"2024-03-16T19:31:49.975256+0000","last_unstale":"2024-03-16T19:38:33.015088+0000","last_undegraded":"2024-03-16T19:38:33.015088+0000","last_fullsized":"2024-03-16T19:38:33.015088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:24.317376+0000","last_clean_scrub_stamp":"2024-03-16T19:29:24.317376+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 378s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"106'6","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:28.243567+0000","last_change":"2024-03-16T19:32:38.176466+0000","last_active":"2024-03-16T19:36:28.243567+0000","last_peered":"2024-03-16T19:36:28.243567+0000","last_clean":"2024-03-16T19:36:28.243567+0000","last_became_active":"2024-03-16T19:29:26.360811+0000","last_became_peered":"2024-03-16T19:29:26.360811+0000","last_unstale":"2024-03-16T19:36:28.243567+0000","last_undegraded":"2024-03-16T19:36:28.243567+0000","last_fullsized":"2024-03-16T19:36:28.243567+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:38.176327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:38.176327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:52:21.601292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"106'2","reported_seq":217,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242067+0000","last_change":"2024-03-16T19:32:31.982088+0000","last_active":"2024-03-16T19:36:13.242067+0000","last_peered":"2024-03-16T19:36:13.242067+0000","last_clean":"2024-03-16T19:36:13.242067+0000","last_became_active":"2024-03-16T19:31:49.978258+0000","last_became_peered":"2024-03-16T19:31:49.978258+0000","last_unstale":"2024-03-16T19:36:13.242067+0000","last_undegraded":"2024-03-16T19:36:13.242067+0000","last_fullsized":"2024-03-16T19:36:13.242067+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:31.981938+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.981938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:50:25.584497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"106'5","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242425+0000","last_change":"2024-03-16T19:32:39.149110+0000","last_active":"2024-03-16T19:36:13.242425+0000","last_peered":"2024-03-16T19:36:13.242425+0000","last_clean":"2024-03-16T19:36:13.242425+0000","last_became_active":"2024-03-16T19:29:26.361081+0000","last_became_peered":"2024-03-16T19:29:26.361081+0000","last_unstale":"2024-03-16T19:36:13.242425+0000","last_undegraded":"2024-03-16T19:36:13.242425+0000","last_fullsized":"2024-03-16T19:36:13.242425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.148940+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.148940+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:58:01.404353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"106'3","reported_seq":68,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:18.002692+0000","last_change":"2024-03-16T19:32:32.108585+0000","last_active":"2024-03-16T19:36:18.002692+0000","last_peered":"2024-03-16T19:36:18.002692+0000","last_clean":"2024-03-16T19:36:18.002692+0000","last_became_active":"2024-03-16T19:31:50.984956+0000","last_became_peered":"2024-03-16T19:31:50.984956+0000","last_unstale":"2024-03-16T19:36:18.002692+0000","last_undegraded":"2024-03-16T19:36:18.002692+0000","last_fullsized":"2024-03-16T19:36:18.002692+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.108463+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.108463+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:53:12.244019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"106'9","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242441+0000","last_change":"2024-03-16T19:32:31.059668+0000","last_active":"2024-03-16T19:36:13.242441+0000","last_peered":"2024-03-16T19:36:13.242441+0000","last_clean":"2024-03-16T19:36:13.242441+0000","last_became_active":"2024-03-16T19:29:26.362229+0000","last_became_peered":"2024-03-16T19:29:26.362229+0000","last_unstale":"2024-03-16T19:36:13.242441+0000","last_undegraded":"2024-03-16T19:36:13.242441+0000","last_fullsized":"2024-03-16T19:36:13.242441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:31.059566+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.059566+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:15.853159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"106'2","reported_seq":194,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.254718+0000","last_change":"2024-03-16T19:32:27.568574+0000","last_active":"2024-03-16T19:36:13.254718+0000","last_peered":"2024-03-16T19:36:13.254718+0000","last_clean":"2024-03-16T19:36:13.254718+0000","last_became_active":"2024-03-16T19:29:25.346388+0000","last_became_peered":"2024-03-16T19:29:25.346388+0000","last_unstale":"2024-03-16T19:36:13.254718+0000","last_undegraded":"2024-03-16T19:36:13.254718+0000","last_fullsized":"2024-03-16T19:36:13.254718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:27.568383+0000","last_clean_scrub_stamp":"2024-03-16T19:32:27.568383+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:37.500568+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'6","reported_seq":211,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242270+0000","last_change":"2024-03-16T19:32:37.651659+0000","last_active":"2024-03-16T19:36:13.242270+0000","last_peered":"2024-03-16T19:36:13.242270+0000","last_clean":"2024-03-16T19:36:13.242270+0000","last_became_active":"2024-03-16T19:31:26.808481+0000","last_became_peered":"2024-03-16T19:31:26.808481+0000","last_unstale":"2024-03-16T19:36:13.242270+0000","last_undegraded":"2024-03-16T19:36:13.242270+0000","last_fullsized":"2024-03-16T19:36:13.242270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:37.651539+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.651539+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:47.087456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:18.002679+0000","last_change":"2024-03-16T19:32:28.205835+0000","last_active":"2024-03-16T19:36:18.002679+0000","last_peered":"2024-03-16T19:36:18.002679+0000","last_clean":"2024-03-16T19:36:18.002679+0000","last_became_active":"2024-03-16T19:31:49.974330+0000","last_became_peered":"2024-03-16T19:31:49.974330+0000","last_unstale":"2024-03-16T19:36:18.002679+0000","last_undegraded":"2024-03-16T19:36:18.002679+0000","last_fullsized":"2024-03-16T19:36:18.002679+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.205721+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.205721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:31.393420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"106'7","reported_seq":167,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:43.236034+0000","last_change":"2024-03-16T19:32:40.166603+0000","last_active":"2024-03-16T19:34:43.236034+0000","last_peered":"2024-03-16T19:34:43.236034+0000","last_clean":"2024-03-16T19:34:43.236034+0000","last_became_active":"2024-03-16T19:30:13.514115+0000","last_became_peered":"2024-03-16T19:30:13.514115+0000","last_unstale":"2024-03-16T19:34:43.236034+0000","last_undegraded":"2024-03-16T19:34:43.236034+0000","last_fullsized":"2024-03-16T19:34:43.236034+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:40.166511+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.166511+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:20:29.474345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"106'119","reported_seq":377,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:18.238443+0000","last_change":"2024-03-16T19:32:47.115779+0000","last_active":"2024-03-16T19:35:18.238443+0000","last_peered":"2024-03-16T19:35:18.238443+0000","last_clean":"2024-03-16T19:35:18.238443+0000","last_became_active":"2024-03-16T19:31:52.807484+0000","last_became_peered":"2024-03-16T19:31:52.807484+0000","last_unstale":"2024-03-16T19:35:18.238443+0000","last_undegraded":"2024-03-16T19:35:18.238443+0000","last_fullsized":"2024-03-16T19:35:18.238443+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'119","last_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_deep_scrub":"106'119","last_deep_scrub_stamp":"2024-03-16T19:32:47.115614+0000","last_clean_scrub_stamp":"2024-03-16T19:32:47.115614+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:31.970905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240002+0000","last_change":"2024-03-16T19:32:42.532575+0000","last_active":"2024-03-16T19:35:43.240002+0000","last_peered":"2024-03-16T19:35:43.240002+0000","last_clean":"2024-03-16T19:35:43.240002+0000","last_became_active":"2024-03-16T19:31:49.975015+0000","last_became_peered":"2024-03-16T19:31:49.975015+0000","last_unstale":"2024-03-16T19:35:43.240002+0000","last_undegraded":"2024-03-16T19:35:43.240002+0000","last_fullsized":"2024-03-16T19:35:43.240002+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:42.532477+0000","last_clean_scrub_stamp":"2024-03-16T19:32:42.532477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:42:28.515720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'8","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:38.247538+0000","last_change":"2024-03-16T19:32:29.219294+0000","last_active":"2024-03-16T19:35:38.247538+0000","last_peered":"2024-03-16T19:35:38.247538+0000","last_clean":"2024-03-16T19:35:38.247538+0000","last_became_active":"2024-03-16T19:31:26.808839+0000","last_became_peered":"2024-03-16T19:31:26.808839+0000","last_unstale":"2024-03-16T19:35:38.247538+0000","last_undegraded":"2024-03-16T19:35:38.247538+0000","last_fullsized":"2024-03-16T19:35:38.247538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:29.219149+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.219149+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:53:54.258510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":193,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.239079+0000","last_change":"2024-03-16T19:32:34.162302+0000","last_active":"2024-03-16T19:35:28.239079+0000","last_peered":"2024-03-16T19:35:28.239079+0000","last_clean":"2024-03-16T19:35:28.239079+0000","last_became_active":"2024-03-16T19:29:25.347822+0000","last_became_peered":"2024-03-16T19:29:25.347822+0000","last_unstale":"2024-03-16T19:35:28.239079+0000","last_undegraded":"2024-03-16T19:35:28.239079+0000","last_fullsized":"2024-03-16T19:35:28.239079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:34.162212+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.162212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:38:21.959473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"106'5","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:03.248206+0000","last_change":"2024-03-16T19:32:34.240521+0000","last_active":"2024-03-16T19:36:03.248206+0000","last_peered":"2024-03-16T19:36:03.248206+0000","last_clean":"2024-03-16T19:36:03.248206+0000","last_became_active":"2024-03-16T19:31:51.811948+0000","last_became_peered":"2024-03-16T19:31:51.811948+0000","last_unstale":"2024-03-16T19:36:03.248206+0000","last_undegraded":"2024-03-16T19:36:03.248206+0000","last_fullsized":"2024-03-16T19:36:03.248206+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:34.240371+0000","last_clean_scrub_stamp":"2024-03-16T19:32:34.240371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:39:55.575559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"106'3","reported_seq":65,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.000588+0000","last_change":"2024-03-16T19:32:33.145030+0000","last_active":"2024-03-16T19:35:58.000588+0000","last_peered":"2024-03-16T19:35:58.000588+0000","last_clean":"2024-03-16T19:35:58.000588+0000","last_became_active":"2024-03-16T19:31:50.984993+0000","last_became_peered":"2024-03-16T19:31:50.984993+0000","last_unstale":"2024-03-16T19:35:58.000588+0000","last_undegraded":"2024-03-16T19:35:58.000588+0000","last_fullsized":"2024-03-16T19:35:58.000588+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:33.144916+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.144916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:37.574156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"106'8","reported_seq":225,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:38:38.267229+0000","last_change":"2024-03-16T19:32:35.702860+0000","last_active":"2024-03-16T19:38:38.267229+0000","last_peered":"2024-03-16T19:38:38.267229+0000","last_clean":"2024-03-16T19:38:38.267229+0000","last_became_active":"2024-03-16T19:31:49.976210+0000","last_became_peered":"2024-03-16T19:31:49.976210+0000","last_unstale":"2024-03-16T19:38:38.267229+0000","last_undegraded":"2024-03-16T19:38:38.267229+0000","last_fullsized":"2024-03-16T19:38:38.267229+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 377s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239990+0000","last_change":"2024-03-16T19:32:30.692412+0000","last_active":"2024-03-16T19:35:43.239990+0000","last_peered":"2024-03-16T19:35:43.239990+0000","last_clean":"2024-03-16T19:35:43.239990+0000","last_became_active":"2024-03-16T19:29:25.341468+0000","last_became_peered":"2024-03-16T19:29:25.341468+0000","last_unstale":"2024-03-16T19:35:43.239990+0000","last_undegraded":"2024-03-16T19:35:43.239990+0000","last_fullsized":"2024-03-16T19:35:43.239990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:30.692274+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.692274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:19:34.107959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'7","reported_seq":212,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.239673+0000","last_change":"2024-03-16T19:32:30.998707+0000","last_active":"2024-03-16T19:35:43.239673+0000","last_peered":"2024-03-16T19:35:43.239673+0000","last_clean":"2024-03-16T19:35:43.239673+0000","last_became_active":"2024-03-16T19:31:49.980615+0000","last_became_peered":"2024-03-16T19:31:49.980615+0000","last_unstale":"2024-03-16T19:35:43.239673+0000","last_undegraded":"2024-03-16T19:35:43.239673+0000","last_fullsized":"2024-03-16T19:35:43.239673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:30.998550+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.998550+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:24:36.554010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.252272+0000","last_change":"2024-03-16T19:32:37.721534+0000","last_active":"2024-03-16T19:35:43.252272+0000","last_peered":"2024-03-16T19:35:43.252272+0000","last_clean":"2024-03-16T19:35:43.252272+0000","last_became_active":"2024-03-16T19:31:49.979658+0000","last_became_peered":"2024-03-16T19:31:49.979658+0000","last_unstale":"2024-03-16T19:35:43.252272+0000","last_undegraded":"2024-03-16T19:35:43.252272+0000","last_fullsized":"2024-03-16T19:35:43.252272+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:37.721393+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.721393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:27:06.914919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'5","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240023+0000","last_change":"2024-03-16T19:32:39.589670+0000","last_active":"2024-03-16T19:35:43.240023+0000","last_peered":"2024-03-16T19:35:43.240023+0000","last_clean":"2024-03-16T19:35:43.240023+0000","last_became_active":"2024-03-16T19:31:26.808408+0000","last_became_peered":"2024-03-16T19:31:26.808408+0000","last_unstale":"2024-03-16T19:35:43.240023+0000","last_undegraded":"2024-03-16T19:35:43.240023+0000","last_fullsized":"2024-03-16T19:35:43.240023+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:39.589565+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.589565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:49:50.752557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:43.240209+0000","last_change":"2024-03-16T19:32:46.142320+0000","last_active":"2024-03-16T19:35:43.240209+0000","last_peered":"2024-03-16T19:35:43.240209+0000","last_clean":"2024-03-16T19:35:43.240209+0000","last_became_active":"2024-03-16T19:31:49.982111+0000","last_became_peered":"2024-03-16T19:31:49.982111+0000","last_unstale":"2024-03-16T19:35:43.240209+0000","last_undegraded":"2024-03-16T19:35:43.240209+0000","last_fullsized":"2024-03-16T19:35:43.240209+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:46.142157+0000","last_clean_scrub_stamp":"2024-03-16T19:32:46.142157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:11:57.334808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"106'9","reported_seq":226,"reported_epoch":106,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-16T19:38:38.267271+0000","last_change":"2024-03-16T19:32:34.733084+0000","last_active":"2024-03-16T19:38:38.267271+0000","last_peered":"2024-03-16T19:38:38.267271+0000","last_clean":"2024-03-16T19:38:38.267271+0000","last_became_active":"2024-03-16T19:31:49.975466+0000","last_became_peered":"2024-03-16T19:31:49.975466+0000","last_unstale":"2024-03-16T19:38:38.267271+0000","last_undegraded":"2024-03-16T19:38:38.267271+0000","last_fullsized":"2024-03-16T19:38:38.267271+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 378s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"106'12","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.253604+0000","last_change":"2024-03-16T19:32:29.586903+0000","last_active":"2024-03-16T19:35:58.253604+0000","last_peered":"2024-03-16T19:35:58.253604+0000","last_clean":"2024-03-16T19:35:58.253604+0000","last_became_active":"2024-03-16T19:29:25.349420+0000","last_became_peered":"2024-03-16T19:29:25.349420+0000","last_unstale":"2024-03-16T19:35:58.253604+0000","last_undegraded":"2024-03-16T19:35:58.253604+0000","last_fullsized":"2024-03-16T19:35:58.253604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'12","last_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_deep_scrub":"106'12","last_deep_scrub_stamp":"2024-03-16T19:32:29.586743+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.586743+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:43:59.735669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"106'5","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:48.248392+0000","last_change":"2024-03-16T19:32:33.125506+0000","last_active":"2024-03-16T19:35:48.248392+0000","last_peered":"2024-03-16T19:35:48.248392+0000","last_clean":"2024-03-16T19:35:48.248392+0000","last_became_active":"2024-03-16T19:31:26.810248+0000","last_became_peered":"2024-03-16T19:31:26.810248+0000","last_unstale":"2024-03-16T19:35:48.248392+0000","last_undegraded":"2024-03-16T19:35:48.248392+0000","last_fullsized":"2024-03-16T19:35:48.248392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.125376+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.125376+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:42.580504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.240855+0000","last_change":"2024-03-16T19:32:31.910678+0000","last_active":"2024-03-16T19:35:58.240855+0000","last_peered":"2024-03-16T19:35:58.240855+0000","last_clean":"2024-03-16T19:35:58.240855+0000","last_became_active":"2024-03-16T19:29:25.346820+0000","last_became_peered":"2024-03-16T19:29:25.346820+0000","last_unstale":"2024-03-16T19:35:58.240855+0000","last_undegraded":"2024-03-16T19:35:58.240855+0000","last_fullsized":"2024-03-16T19:35:58.240855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:31.910534+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.910534+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:37:06.645156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"106'7","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.249116+0000","last_change":"2024-03-16T19:32:32.188030+0000","last_active":"2024-03-16T19:36:13.249116+0000","last_peered":"2024-03-16T19:36:13.249116+0000","last_clean":"2024-03-16T19:36:13.249116+0000","last_became_active":"2024-03-16T19:31:51.809862+0000","last_became_peered":"2024-03-16T19:31:51.809862+0000","last_unstale":"2024-03-16T19:36:13.249116+0000","last_undegraded":"2024-03-16T19:36:13.249116+0000","last_fullsized":"2024-03-16T19:36:13.249116+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'7","last_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_deep_scrub":"106'7","last_deep_scrub_stamp":"2024-03-16T19:32:32.187918+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.187918+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:28:17.331289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.241280+0000","last_change":"2024-03-16T19:32:44.190121+0000","last_active":"2024-03-16T19:35:58.241280+0000","last_peered":"2024-03-16T19:35:58.241280+0000","last_clean":"2024-03-16T19:35:58.241280+0000","last_became_active":"2024-03-16T19:31:49.980655+0000","last_became_peered":"2024-03-16T19:31:49.980655+0000","last_unstale":"2024-03-16T19:35:58.241280+0000","last_undegraded":"2024-03-16T19:35:58.241280+0000","last_fullsized":"2024-03-16T19:35:58.241280+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:44.190028+0000","last_clean_scrub_stamp":"2024-03-16T19:32:44.190028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:17:12.004606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"106'2","reported_seq":204,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.240837+0000","last_change":"2024-03-16T19:32:30.026167+0000","last_active":"2024-03-16T19:35:58.240837+0000","last_peered":"2024-03-16T19:35:58.240837+0000","last_clean":"2024-03-16T19:35:58.240837+0000","last_became_active":"2024-03-16T19:31:49.973851+0000","last_became_peered":"2024-03-16T19:31:49.973851+0000","last_unstale":"2024-03-16T19:35:58.240837+0000","last_undegraded":"2024-03-16T19:35:58.240837+0000","last_fullsized":"2024-03-16T19:35:58.240837+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'2","last_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_deep_scrub":"106'2","last_deep_scrub_stamp":"2024-03-16T19:32:30.026030+0000","last_clean_scrub_stamp":"2024-03-16T19:32:30.026030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:00.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":192,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.253534+0000","last_change":"2024-03-16T19:32:28.604791+0000","last_active":"2024-03-16T19:35:58.253534+0000","last_peered":"2024-03-16T19:35:58.253534+0000","last_clean":"2024-03-16T19:35:58.253534+0000","last_became_active":"2024-03-16T19:29:25.346556+0000","last_became_peered":"2024-03-16T19:29:25.346556+0000","last_unstale":"2024-03-16T19:35:58.253534+0000","last_undegraded":"2024-03-16T19:35:58.253534+0000","last_fullsized":"2024-03-16T19:35:58.253534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:28.604677+0000","last_clean_scrub_stamp":"2024-03-16T19:32:28.604677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:51.854202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":42,"reported_epoch":106,"state":"active+clean+scrubbing+deep+laggy","last_fresh":"2024-03-16T19:38:38.015558+0000","last_change":"2024-03-16T19:32:34.100406+0000","last_active":"2024-03-16T19:38:38.015558+0000","last_peered":"2024-03-16T19:38:38.015558+0000","last_clean":"2024-03-16T19:38:38.015558+0000","last_became_active":"2024-03-16T19:32:08.100914+0000","last_became_peered":"2024-03-16T19:32:08.100914+0000","last_unstale":"2024-03-16T19:38:38.015558+0000","last_undegraded":"2024-03-16T19:38:38.015558+0000","last_fullsized":"2024-03-16T19:38:38.015558+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:29:25.321521+0000","last_clean_scrub_stamp":"2024-03-16T19:29:25.321521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 378s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":202,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.253561+0000","last_change":"2024-03-16T19:32:33.699900+0000","last_active":"2024-03-16T19:35:58.253561+0000","last_peered":"2024-03-16T19:35:58.253561+0000","last_clean":"2024-03-16T19:35:58.253561+0000","last_became_active":"2024-03-16T19:31:26.814716+0000","last_became_peered":"2024-03-16T19:31:26.814716+0000","last_unstale":"2024-03-16T19:35:58.253561+0000","last_undegraded":"2024-03-16T19:35:58.253561+0000","last_fullsized":"2024-03-16T19:35:58.253561+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T19:32:33.699763+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.699763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:03.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"106'8","reported_seq":97,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.249305+0000","last_change":"2024-03-16T19:32:39.030458+0000","last_active":"2024-03-16T19:35:58.249305+0000","last_peered":"2024-03-16T19:35:58.249305+0000","last_clean":"2024-03-16T19:35:58.249305+0000","last_became_active":"2024-03-16T19:31:49.984315+0000","last_became_peered":"2024-03-16T19:31:49.984315+0000","last_unstale":"2024-03-16T19:35:58.249305+0000","last_undegraded":"2024-03-16T19:35:58.249305+0000","last_fullsized":"2024-03-16T19:35:58.249305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'8","last_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_deep_scrub":"106'8","last_deep_scrub_stamp":"2024-03-16T19:32:39.030303+0000","last_clean_scrub_stamp":"2024-03-16T19:32:39.030303+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:43:12.769684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"106'4","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:13.242384+0000","last_change":"2024-03-16T19:32:33.152425+0000","last_active":"2024-03-16T19:36:13.242384+0000","last_peered":"2024-03-16T19:36:13.242384+0000","last_clean":"2024-03-16T19:36:13.242384+0000","last_became_active":"2024-03-16T19:29:25.347808+0000","last_became_peered":"2024-03-16T19:29:25.347808+0000","last_unstale":"2024-03-16T19:36:13.242384+0000","last_undegraded":"2024-03-16T19:36:13.242384+0000","last_fullsized":"2024-03-16T19:36:13.242384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:33.152341+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.152341+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:34:18.570168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"106'5","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:58.240799+0000","last_change":"2024-03-16T19:32:33.889937+0000","last_active":"2024-03-16T19:35:58.240799+0000","last_peered":"2024-03-16T19:35:58.240799+0000","last_clean":"2024-03-16T19:35:58.240799+0000","last_became_active":"2024-03-16T19:29:26.358986+0000","last_became_peered":"2024-03-16T19:29:26.358986+0000","last_unstale":"2024-03-16T19:35:58.240799+0000","last_undegraded":"2024-03-16T19:35:58.240799+0000","last_fullsized":"2024-03-16T19:35:58.240799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:33.889778+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.889778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:13:46.705019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"106'15","reported_seq":209,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235008+0000","last_change":"2024-03-16T19:32:31.695513+0000","last_active":"2024-03-16T19:34:28.235008+0000","last_peered":"2024-03-16T19:34:28.235008+0000","last_clean":"2024-03-16T19:34:28.235008+0000","last_became_active":"2024-03-16T19:29:26.355195+0000","last_became_peered":"2024-03-16T19:29:26.355195+0000","last_unstale":"2024-03-16T19:34:28.235008+0000","last_undegraded":"2024-03-16T19:34:28.235008+0000","last_fullsized":"2024-03-16T19:34:28.235008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'15","last_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_deep_scrub":"106'15","last_deep_scrub_stamp":"2024-03-16T19:32:31.695342+0000","last_clean_scrub_stamp":"2024-03-16T19:32:31.695342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:06:09.672330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"106'3","reported_seq":195,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235076+0000","last_change":"2024-03-16T19:32:32.102649+0000","last_active":"2024-03-16T19:34:28.235076+0000","last_peered":"2024-03-16T19:34:28.235076+0000","last_clean":"2024-03-16T19:34:28.235076+0000","last_became_active":"2024-03-16T19:29:25.346383+0000","last_became_peered":"2024-03-16T19:29:25.346383+0000","last_unstale":"2024-03-16T19:34:28.235076+0000","last_undegraded":"2024-03-16T19:34:28.235076+0000","last_fullsized":"2024-03-16T19:34:28.235076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'3","last_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_deep_scrub":"106'3","last_deep_scrub_stamp":"2024-03-16T19:32:32.102526+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.102526+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:55.803998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"106'9","reported_seq":203,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.234995+0000","last_change":"2024-03-16T19:32:32.696352+0000","last_active":"2024-03-16T19:34:28.234995+0000","last_peered":"2024-03-16T19:34:28.234995+0000","last_clean":"2024-03-16T19:34:28.234995+0000","last_became_active":"2024-03-16T19:29:26.356936+0000","last_became_peered":"2024-03-16T19:29:26.356936+0000","last_unstale":"2024-03-16T19:34:28.234995+0000","last_undegraded":"2024-03-16T19:34:28.234995+0000","last_fullsized":"2024-03-16T19:34:28.234995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'9","last_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_deep_scrub":"106'9","last_deep_scrub_stamp":"2024-03-16T19:32:32.696255+0000","last_clean_scrub_stamp":"2024-03-16T19:32:32.696255+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:13:03.845081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"106'4","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:34:28.235086+0000","last_change":"2024-03-16T19:32:35.177567+0000","last_active":"2024-03-16T19:34:28.235086+0000","last_peered":"2024-03-16T19:34:28.235086+0000","last_clean":"2024-03-16T19:34:28.235086+0000","last_became_active":"2024-03-16T19:29:25.349525+0000","last_became_peered":"2024-03-16T19:29:25.349525+0000","last_unstale":"2024-03-16T19:34:28.235086+0000","last_undegraded":"2024-03-16T19:34:28.235086+0000","last_fullsized":"2024-03-16T19:34:28.235086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'4","last_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_deep_scrub":"106'4","last_deep_scrub_stamp":"2024-03-16T19:32:35.177456+0000","last_clean_scrub_stamp":"2024-03-16T19:32:35.177456+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:31:17.616177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"106'6","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:43.251827+0000","last_change":"2024-03-16T19:32:33.230775+0000","last_active":"2024-03-16T19:36:43.251827+0000","last_peered":"2024-03-16T19:36:43.251827+0000","last_clean":"2024-03-16T19:36:43.251827+0000","last_became_active":"2024-03-16T19:31:51.809781+0000","last_became_peered":"2024-03-16T19:31:51.809781+0000","last_unstale":"2024-03-16T19:36:43.251827+0000","last_undegraded":"2024-03-16T19:36:43.251827+0000","last_fullsized":"2024-03-16T19:36:43.251827+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:33.230681+0000","last_clean_scrub_stamp":"2024-03-16T19:32:33.230681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:08:22.432674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.246603+0000","last_change":"2024-03-16T19:32:40.046537+0000","last_active":"2024-03-16T19:35:28.246603+0000","last_peered":"2024-03-16T19:35:28.246603+0000","last_clean":"2024-03-16T19:35:28.246603+0000","last_became_active":"2024-03-16T19:31:49.984391+0000","last_became_peered":"2024-03-16T19:31:49.984391+0000","last_unstale":"2024-03-16T19:35:28.246603+0000","last_undegraded":"2024-03-16T19:35:28.246603+0000","last_fullsized":"2024-03-16T19:35:28.246603+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-16T19:32:40.046375+0000","last_clean_scrub_stamp":"2024-03-16T19:32:40.046375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:43:02.973781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"106'5","reported_seq":177,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:35:28.246634+0000","last_change":"2024-03-16T19:32:37.063436+0000","last_active":"2024-03-16T19:35:28.246634+0000","last_peered":"2024-03-16T19:35:28.246634+0000","last_clean":"2024-03-16T19:35:28.246634+0000","last_became_active":"2024-03-16T19:31:27.858889+0000","last_became_peered":"2024-03-16T19:31:27.858889+0000","last_unstale":"2024-03-16T19:35:28.246634+0000","last_undegraded":"2024-03-16T19:35:28.246634+0000","last_fullsized":"2024-03-16T19:35:28.246634+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'5","last_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_deep_scrub":"106'5","last_deep_scrub_stamp":"2024-03-16T19:32:37.063327+0000","last_clean_scrub_stamp":"2024-03-16T19:32:37.063327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:38.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"106'6","reported_seq":219,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-16T19:36:03.248271+0000","last_change":"2024-03-16T19:32:29.272673+0000","last_active":"2024-03-16T19:36:03.248271+0000","last_peered":"2024-03-16T19:36:03.248271+0000","last_clean":"2024-03-16T19:36:03.248271+0000","last_became_active":"2024-03-16T19:31:51.808525+0000","last_became_peered":"2024-03-16T19:31:51.808525+0000","last_unstale":"2024-03-16T19:36:03.248271+0000","last_undegraded":"2024-03-16T19:36:03.248271+0000","last_fullsized":"2024-03-16T19:36:03.248271+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'6","last_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_deep_scrub":"106'6","last_deep_scrub_stamp":"2024-03-16T19:32:29.272516+0000","last_clean_scrub_stamp":"2024-03-16T19:32:29.272516+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:13:28.283465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":494,"ondisk_log_size":494,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":40667,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14388,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":81334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574968,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1859584,"data_stored":1186098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574969,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":634613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574967,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1265664,"data_stored":595238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574968,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343580,"statfs":{"total":96636764160,"available":96607825920,"internally_reserved":0,"allocated":1273856,"data_stored":595274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574969,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1314816,"data_stored":635017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":88,"seq":377957122135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29404,"kb_used_data":1236,"kb_used_omap":5,"kb_used_meta":28154,"kb_avail":94342436,"statfs":{"total":96636764160,"available":96606654464,"internally_reserved":0,"allocated":1265664,"data_stored":595552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5825,"internal_metadata":28830015},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1269760,"data_stored":595308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574968,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1867776,"data_stored":1186154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4230,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":39869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T19:38:54.688 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-16T19:29:24.317376+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:54.688 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:54.688 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:54.689 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-16T19:29:25.321521+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=76, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=16, tm_hour=19, tm_min=32, tm_sec=23, tm_wday=5, tm_yday=76, tm_isdst=0) 2024-03-16T19:38:54.689 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-16T19:38:55.301 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:55.322+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:56.297 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:56.319+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:57.316 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:57.338+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:58.364 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:58.386+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:38:59.340 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:38:59.362+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:00.312 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:00.334+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:01.356 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:01.376+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:02.399 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:02.421+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:03.389 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:03.412+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:04.438 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:04.461+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:05.403 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:05.425+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:06.391 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:06.414+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:07.381 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:07.403+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:08.667 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:08.435+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:09.605 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:09.480+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:11.837 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:11.105+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:13.118 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:12.689+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:14.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T19:39:15.295 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:15.125+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:22.748 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:19.473+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:35.702 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:32.841+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:39:41.427 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:39:41.450+0000 7ff536b89640 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.172:6804 osd.0 since back 2024-03-16T19:39:19.436230+0000 front 2024-03-16T19:39:19.436086+0000 (oldest deadline 2024-03-16T19:39:41.238547+0000) 2024-03-16T19:39:41.427 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:39:41.450+0000 7ff536b89640 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.172:6828 osd.2 since back 2024-03-16T19:39:26.563770+0000 front 2024-03-16T19:39:19.436147+0000 (oldest deadline 2024-03-16T19:39:41.238547+0000) 2024-03-16T19:39:41.427 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:39:41.450+0000 7ff536b89640 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.172:6820 osd.3 since back 2024-03-16T19:39:19.436046+0000 front 2024-03-16T19:39:19.465053+0000 (oldest deadline 2024-03-16T19:39:41.238547+0000) 2024-03-16T19:39:42.385 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:39:42.408+0000 7ff536b89640 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.172:6804 osd.0 since back 2024-03-16T19:39:19.436230+0000 front 2024-03-16T19:39:19.436086+0000 (oldest deadline 2024-03-16T19:39:41.238547+0000) 2024-03-16T19:39:42.386 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:39:42.408+0000 7ff536b89640 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.172:6828 osd.2 since back 2024-03-16T19:39:26.563770+0000 front 2024-03-16T19:39:19.436147+0000 (oldest deadline 2024-03-16T19:39:41.238547+0000) 2024-03-16T19:39:42.386 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:39:42.408+0000 7ff536b89640 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.172:6820 osd.3 since back 2024-03-16T19:39:19.436046+0000 front 2024-03-16T19:39:19.465053+0000 (oldest deadline 2024-03-16T19:39:41.238547+0000) 2024-03-16T19:39:43.342 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:39:43.365+0000 7ff51a8c9640 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.172:6804 osd.0 since back 2024-03-16T19:39:38.927492+0000 front 2024-03-16T19:39:18.068724+0000 (oldest deadline 2024-03-16T19:39:42.729223+0000) 2024-03-16T19:39:43.342 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:39:43.365+0000 7ff51a8c9640 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:26.563670+0000 front 2024-03-16T19:39:18.068810+0000 (oldest deadline 2024-03-16T19:39:42.729223+0000) 2024-03-16T19:39:43.342 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:39:43.365+0000 7ff51a8c9640 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.172:6828 osd.2 since back 2024-03-16T19:39:18.611643+0000 front 2024-03-16T19:39:26.287612+0000 (oldest deadline 2024-03-16T19:39:42.729223+0000) 2024-03-16T19:39:43.342 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:39:43.365+0000 7ff51a8c9640 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.172:6820 osd.3 since back 2024-03-16T19:39:17.958208+0000 front 2024-03-16T19:39:26.563850+0000 (oldest deadline 2024-03-16T19:39:42.729223+0000) 2024-03-16T19:39:43.351 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:39:43.374+0000 7ff536b89640 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.172:6804 osd.0 since back 2024-03-16T19:39:19.436230+0000 front 2024-03-16T19:39:19.436086+0000 (oldest deadline 2024-03-16T19:39:41.238547+0000) 2024-03-16T19:39:43.351 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:39:43.374+0000 7ff536b89640 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.172:6828 osd.2 since back 2024-03-16T19:39:26.563770+0000 front 2024-03-16T19:39:19.436147+0000 (oldest deadline 2024-03-16T19:39:41.238547+0000) 2024-03-16T19:39:43.351 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-16T19:39:43.374+0000 7ff536b89640 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.172:6820 osd.3 since back 2024-03-16T19:39:19.436046+0000 front 2024-03-16T19:39:19.465053+0000 (oldest deadline 2024-03-16T19:39:41.238547+0000) 2024-03-16T19:39:43.742 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:39:43.765+0000 7f9f72742640 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:19.397595+0000 front 2024-03-16T19:39:19.566276+0000 (oldest deadline 2024-03-16T19:39:42.900703+0000) 2024-03-16T19:39:44.300 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:39:44.323+0000 7ff51a8c9640 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:26.563670+0000 front 2024-03-16T19:39:18.068810+0000 (oldest deadline 2024-03-16T19:39:42.729223+0000) 2024-03-16T19:39:44.737 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:39:44.760+0000 7f9f72742640 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:19.397595+0000 front 2024-03-16T19:39:19.566276+0000 (oldest deadline 2024-03-16T19:39:42.900703+0000) 2024-03-16T19:39:45.300 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:39:45.323+0000 7ff51a8c9640 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:26.563670+0000 front 2024-03-16T19:39:18.068810+0000 (oldest deadline 2024-03-16T19:39:42.729223+0000) 2024-03-16T19:39:45.771 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:39:45.794+0000 7f9f72742640 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:19.397595+0000 front 2024-03-16T19:39:19.566276+0000 (oldest deadline 2024-03-16T19:39:42.900703+0000) 2024-03-16T19:39:46.280 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-16T19:39:46.303+0000 7ff51a8c9640 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:26.563670+0000 front 2024-03-16T19:39:18.068810+0000 (oldest deadline 2024-03-16T19:39:42.729223+0000) 2024-03-16T19:39:46.569 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-16T19:39:46.592+0000 7fb3a1676640 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:25.969450+0000 front 2024-03-16T19:39:36.268911+0000 (oldest deadline 2024-03-16T19:39:45.859871+0000) 2024-03-16T19:39:46.740 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-16T19:39:46.763+0000 7f9f72742640 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:19.397595+0000 front 2024-03-16T19:39:19.566276+0000 (oldest deadline 2024-03-16T19:39:42.900703+0000) 2024-03-16T19:39:56.894 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.915+0000 7fba5340a640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:39:56.894 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.915+0000 7fba5340a640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:39:56.894 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.915+0000 7fba5340a640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:39:56.894 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.915+0000 7fba5340a640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:39:56.894 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.915+0000 7fba5340a640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:39:56.895 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.916+0000 7fba56410640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:39:56.895 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.916+0000 7fba56410640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:39:56.895 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.916+0000 7fba56410640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:39:56.895 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.916+0000 7fba56410640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:39:56.895 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-16T19:39:56.916+0000 7fba56410640 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T19:40:01.958 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:39:55.396+0000 7fda7ff29640 -1 osd.1 106 heartbeat_check: no reply from 172.21.15.172:6804 osd.0 since back 2024-03-16T19:39:19.445810+0000 front 2024-03-16T19:39:19.445844+0000 (oldest deadline 2024-03-16T19:39:44.021567+0000) 2024-03-16T19:40:09.921 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:39:57.257+0000 7fda7ff29640 -1 osd.1 106 heartbeat_check: no reply from 172.21.15.172:6828 osd.2 since back 2024-03-16T19:39:19.445795+0000 front 2024-03-16T19:39:12.741692+0000 (oldest deadline 2024-03-16T19:39:38.906098+0000) 2024-03-16T19:40:09.922 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:39:58.282+0000 7fda7ff29640 -1 osd.1 106 heartbeat_check: no reply from 172.21.15.172:6820 osd.3 since back 2024-03-16T19:39:19.464903+0000 front 2024-03-16T19:39:19.464947+0000 (oldest deadline 2024-03-16T19:39:44.021567+0000) 2024-03-16T19:40:09.922 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:39:58.417+0000 7fda7ff29640 -1 osd.1 106 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-16T19:39:19.445721+0000 front 2024-03-16T19:39:19.566347+0000 (oldest deadline 2024-03-16T19:39:44.021567+0000) 2024-03-16T19:40:09.922 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:39:58.587+0000 7fda7ff29640 -1 osd.1 106 heartbeat_check: no reply from 172.21.15.174:6822 osd.5 since back 2024-03-16T19:39:19.566379+0000 front 2024-03-16T19:39:27.740928+0000 (oldest deadline 2024-03-16T19:39:44.021567+0000) 2024-03-16T19:40:09.922 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:39:58.629+0000 7fda7ff29640 -1 osd.1 106 heartbeat_check: no reply from 172.21.15.174:6826 osd.6 since back 2024-03-16T19:39:19.566400+0000 front 2024-03-16T19:39:19.566422+0000 (oldest deadline 2024-03-16T19:39:44.021567+0000) 2024-03-16T19:40:09.922 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-16T19:39:59.130+0000 7fda7ff29640 -1 osd.1 106 heartbeat_check: no reply from 172.21.15.174:6804 osd.7 since back 2024-03-16T19:39:28.235376+0000 front 2024-03-16T19:39:19.445757+0000 (oldest deadline 2024-03-16T19:39:44.021567+0000) 2024-03-16T19:40:09.922 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:55.396+0000 7f1aa49cc640 -1 osd.2 106 heartbeat_check: no reply from 172.21.15.172:6804 osd.0 since back 2024-03-16T19:39:18.068909+0000 front 2024-03-16T19:39:26.447986+0000 (oldest deadline 2024-03-16T19:39:44.567410+0000) 2024-03-16T19:40:09.922 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:57.257+0000 7f1aa49cc640 -1 osd.2 106 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:18.102893+0000 front 2024-03-16T19:39:18.102849+0000 (oldest deadline 2024-03-16T19:39:44.567410+0000) 2024-03-16T19:40:09.923 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:58.282+0000 7f1aa49cc640 -1 osd.2 106 heartbeat_check: no reply from 172.21.15.172:6820 osd.3 since back 2024-03-16T19:39:18.069048+0000 front 2024-03-16T19:39:18.068981+0000 (oldest deadline 2024-03-16T19:39:44.567410+0000) 2024-03-16T19:40:09.923 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:58.417+0000 7f1aa49cc640 -1 osd.2 106 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-16T19:39:38.078989+0000 front 2024-03-16T19:39:30.121584+0000 (oldest deadline 2024-03-16T19:39:51.486728+0000) 2024-03-16T19:40:09.923 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:58.578+0000 7f1aa49cc640 -1 osd.2 106 heartbeat_check: no reply from 172.21.15.174:6822 osd.5 since back 2024-03-16T19:39:32.822873+0000 front 2024-03-16T19:39:19.436227+0000 (oldest deadline 2024-03-16T19:39:44.567410+0000) 2024-03-16T19:40:09.923 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:58.629+0000 7f1aa49cc640 -1 osd.2 106 heartbeat_check: no reply from 172.21.15.174:6826 osd.6 since back 2024-03-16T19:39:19.436306+0000 front 2024-03-16T19:39:33.614197+0000 (oldest deadline 2024-03-16T19:39:44.567410+0000) 2024-03-16T19:40:09.923 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:39:59.130+0000 7f1aa49cc640 -1 osd.2 106 heartbeat_check: no reply from 172.21.15.174:6804 osd.7 since back 2024-03-16T19:39:37.045664+0000 front 2024-03-16T19:39:31.663590+0000 (oldest deadline 2024-03-16T19:39:51.486728+0000) 2024-03-16T19:40:21.429 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:40:07.844+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:15.219610+0000 front 2024-03-16T19:39:15.219648+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:40:26.958 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:40:09.430+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.172:6828 osd.2 since back 2024-03-16T19:39:15.220056+0000 front 2024-03-16T19:39:15.220085+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:40:26.958 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:40:11.142+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.172:6820 osd.3 since back 2024-03-16T19:39:15.219738+0000 front 2024-03-16T19:39:15.220033+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:40:26.958 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:40:12.523+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-16T19:39:15.219242+0000 front 2024-03-16T19:39:15.219319+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:40:26.958 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:40:14.495+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.174:6822 osd.5 since back 2024-03-16T19:39:15.219420+0000 front 2024-03-16T19:39:15.219576+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:40:26.958 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:40:16.147+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.174:6826 osd.6 since back 2024-03-16T19:39:15.219499+0000 front 2024-03-16T19:39:15.219818+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:40:31.833 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:40:16.965+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.174:6804 osd.7 since back 2024-03-16T19:39:15.219703+0000 front 2024-03-16T19:39:15.219774+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:41:28.808 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:41:22.614+0000 7fbcd18e0640 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.172:6804 osd.0 since back 2024-03-16T19:39:15.219699+0000 front 2024-03-16T19:39:15.219738+0000 (oldest deadline 2024-03-16T19:39:41.928650+0000) 2024-03-16T19:41:35.233 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:41:23.957+0000 7fbcd18e0640 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:15.219720+0000 front 2024-03-16T19:39:15.219755+0000 (oldest deadline 2024-03-16T19:39:41.928650+0000) 2024-03-16T19:41:35.233 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:41:25.172+0000 7fbcd18e0640 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.172:6828 osd.2 since back 2024-03-16T19:39:15.219880+0000 front 2024-03-16T19:39:15.220043+0000 (oldest deadline 2024-03-16T19:39:41.928650+0000) 2024-03-16T19:41:35.233 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:41:25.979+0000 7fbcd18e0640 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-16T19:39:15.219944+0000 front 2024-03-16T19:39:15.219774+0000 (oldest deadline 2024-03-16T19:39:41.928650+0000) 2024-03-16T19:41:35.233 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:41:27.940+0000 7fbcd18e0640 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.174:6822 osd.5 since back 2024-03-16T19:39:15.219900+0000 front 2024-03-16T19:39:15.219848+0000 (oldest deadline 2024-03-16T19:39:41.928650+0000) 2024-03-16T19:41:35.233 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:41:29.384+0000 7fbcd18e0640 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.174:6826 osd.6 since back 2024-03-16T19:39:15.219435+0000 front 2024-03-16T19:39:15.219628+0000 (oldest deadline 2024-03-16T19:39:41.928650+0000) 2024-03-16T19:41:39.297 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-16T19:41:30.413+0000 7fbcd18e0640 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.174:6804 osd.7 since back 2024-03-16T19:39:15.219506+0000 front 2024-03-16T19:39:15.219796+0000 (oldest deadline 2024-03-16T19:39:41.928650+0000) 2024-03-16T19:42:15.326 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-16T19:42:08.180+0000 7f1aa49cc640 -1 osd.2 106 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:104 3.8 3:12420902:::1000000001b.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) 2024-03-16T19:43:18.345 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:43:15.608+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.172:6812 osd.1 since back 2024-03-16T19:39:15.219610+0000 front 2024-03-16T19:39:15.219648+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:43:18.345 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:43:15.767+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.172:6828 osd.2 since back 2024-03-16T19:39:15.220056+0000 front 2024-03-16T19:39:15.220085+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:43:18.346 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:43:15.782+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.172:6820 osd.3 since back 2024-03-16T19:39:15.219738+0000 front 2024-03-16T19:39:15.220033+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:43:18.346 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:43:15.782+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-16T19:39:15.219242+0000 front 2024-03-16T19:39:15.219319+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:43:18.346 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:43:15.782+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.174:6822 osd.5 since back 2024-03-16T19:39:15.219420+0000 front 2024-03-16T19:39:15.219576+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:43:18.346 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:43:15.782+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.174:6826 osd.6 since back 2024-03-16T19:39:15.219499+0000 front 2024-03-16T19:39:15.219818+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-16T19:43:18.346 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-16T19:43:15.782+0000 7f38f99da640 -1 osd.0 106 heartbeat_check: no reply from 172.21.15.174:6804 osd.7 since back 2024-03-16T19:39:15.219703+0000 front 2024-03-16T19:39:15.219774+0000 (oldest deadline 2024-03-16T19:39:41.928647+0000) 2024-03-17T07:19:26.299 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-17T07:19:26.322 DEBUG:teuthology.task.console_log:Killing console logger for smithi172 2024-03-17T07:19:26.323 DEBUG:teuthology.task.console_log:Killing console logger for smithi174 2024-03-17T07:19:26.323 DEBUG:teuthology.exit:Finished running handlers